{"id":3085,"date":"2026-08-03T16:06:39","date_gmt":"2026-08-03T07:06:39","guid":{"rendered":"https:\/\/aktsk.ai\/?post_type=blog&p=3085"},"modified":"2026-08-03T16:06:39","modified_gmt":"2026-08-03T07:06:39","slug":"the-slm-youre-allowed-to-use-on-premise-small-language-models-for-regulated-japan","status":"publish","type":"blog","link":"https:\/\/aktsk.ai\/en\/blog\/3085\/","title":{"rendered":"The SLM You’re Allowed to Use: On-Premise Small Language Models for Regulated Japan"},"content":{"rendered":"\r\n
\r\n

In Japanese finance, healthcare, and the public sector, the frontier model is often the one option that is off the table before a project even starts. Not because it is too weak, but because it is not allowed near the data. The winning move in these environments is rarely a bigger model. It is a Japanese-capable small model that runs where the data already lives.<\/p>\r\n

Walk into a bank, a hospital, or a municipal office with an AI proposal and you will seldom lose on model quality. You lose on a single sentence the client says early: \u201cThat data cannot leave our network.\u201d Three concerns capture where these conversations get caught:<\/p>\r\n

\r\n

\u201cIt passed the demo beautifully \u2014 but the vendor is overseas, and compliance will never sign off on the data leaving the country.\u201d<\/p>\r\n

\u201cOur patient records are special-care-required personal information. There is no version of this where the data is sent to a foreign cloud API.\u201d<\/p>\r\n

\u201cWe are bound by the client\u2019s NDA and our own risk policy. On-premise is not a preference here; it is the only option.\u201d<\/p>\r\n<\/div>\r\n

This article makes the case for the model you are actually permitted to run: a Japanese-capable small or open-weight model, deployed on-premise or in a private VPC, and tuned to one narrow task rather than to general capability. We use electronic health record summarisation as the hero example because it is the cleanest version of the constraint, then walk through how we size a model to a task, quantise it to the available hardware, evaluate its Japanese quality, and reason about the accuracy-versus-compliance trade-off with clients. This is part of our ongoing series on deploying enterprise AI under real-world constraints in Japan.<\/p>\r\n

01<\/span>The Constraint the Client Actually Has<\/h2>\r\n

The industry pitch usually starts with capability. The client\u2019s real problem is permission. And it is not caution for its own sake \u2014 three separate forces converge on the same answer:<\/p>\r\n

    \r\n
  • APPI (Act on the Protection of Personal Information).<\/strong> Medical records, and much of what finance and government hold, are special-care-required personal information. Both handling and cross-border transfer are tightly constrained.<\/li>\r\n
  • Sector guidelines.<\/strong> Healthcare providers are expected to follow the safety-management guidelines issued by Japan\u2019s Ministries of Health, Labour and Welfare; Internal Affairs and Communications; and Economy, Trade and Industry \u2014 commonly referred to as the \u201cThree Ministries, Two Guidelines.\u201d The healthcare edition is currently at Version 6.0 (2023), its most recent major revision. They require confidentiality, integrity, and availability, plus the authenticity of records, strict supervision of any subcontractor, and, in practice, that data remains in a domestic region.<\/li>\r\n
  • Contracts and internal risk policy.<\/strong> Client NDAs and internal rules frequently forbid sending data to an overseas API outright, regardless of what the law technically permits.<\/li>\r\n<\/ul>\r\n

    This is not a fringe scenario; Japan\u2019s modernisation gap makes on-premise the default posture, not the exception. METI\u2019s 2018 DX Report \u2014 the origin of the well-known \u201c2025 Digital Cliff\u201d \u2014 estimated that unaddressed legacy systems could cost the economy up to \u00a512 trillion a year, with roughly 80% of enterprises still running heavily customised legacy core systems.<\/p>\r\n

    Adoption has since caught up on the surface \u2014 by 2025 around 80% of Japanese companies reported undertaking DX activities, comparable to the US \u2014 yet only about 30% consider their DX successful, and nearly 80% say they have yet to achieve significant business outcomes. The upshot for us is simple: a great deal of the most sensitive Japanese enterprise data is not going to an overseas API any time soon.<\/p>\r\n

    Global vendors sell the frontier model. The client\u2019s real problem is a boundary the frontier model cannot cross. Solve the boundary, and you win the work the frontier vendor structurally cannot.<\/div>\r\n

    02<\/span>Size Is a Spec, Not a Scoreboard<\/h2>\r\n

    The instinct \u2014 the client\u2019s, and honestly the industry\u2019s \u2014 is that a bigger model is a better model. For an open-ended assistant that must do everything, that is roughly true. But almost none of these deployments are open-ended. They are one narrow, repetitive, high-volume task, in Japanese, on data that cannot move.<\/p>\r\n

    On that kind of task, a small model that has been continual-pretrained on Japanese and tuned to the specific job routinely matches a frontier general model \u2014 while running on a single GPU inside the client\u2019s own walls. The question stops being \u201chow capable is the model?\u201d and becomes \u201cis it capable enough at this one task, and can I run it where the data is?\u201d That second question has exactly one class of answer: Japanese-capable small and open-weight models, deployed on-premise or in a private VPC. Everything below is how we size, deploy, and evaluate them.<\/p>\r\n

    03<\/span>The Hero Case: A Doctor Re-Reading a Chart<\/h2>\r\n

    Before nearly every consultation, a physician re-reads a long patient history \u2014 previous visits, lab results, prescriptions, referral letters, free-text nursing notes \u2014 to reconstruct the story in their head. It is slow, it repeats hundreds of times a day across a hospital, and it eats into the minutes that should go to the patient in front of them.<\/p>\r\n

    An assistive model that produces a draft summary \u2014 \u201chere is this patient in six lines, with the open issues flagged\u201d \u2014 for the clinician to review is an obvious, high-value application. It is also the single cleanest illustration of our whole argument, for one reason:<\/p>\r\n

    There is no compliant path to an overseas API here. Electronic health records are special-care-required personal information under APPI and fall squarely under the Three Ministries, Two Guidelines. The \u201cjust use the biggest cloud model\u201d option does not get rejected \u2014 it never exists. Every decision is forced onto the ground where an on-premise approach is strongest.<\/div>\r\n

    It is honest about difficulty, too. Summarisation is not a trivial extract-a-field task; it is genuinely borderline for a small model, which is exactly why it is worth walking through the methodology rather than hand-waving.<\/p>\r\n

    And because summaries can inform clinical judgement, the correct framing is assistive, not autonomous: the model drafts, a clinician signs off. That posture is both the safe design and the one that keeps a human accountable for the medicine.<\/p>\r\n

    04<\/span>Architecture: The Data Never Crosses the Boundary<\/h2>\r\n

    The reference deployment is deliberately unglamorous. Every component \u2014 record store, inference server, review interface \u2014 sits inside the hospital network or a private VPC. The only thing that ever leaves is nothing.<\/p>\r\n

    \"Reference\r\n
    The reference deployment. Record store \u2192 PII handling \u2192 on-prem SLM \u2192 clinician review \u2192 signed summary written back. The overseas frontier API sits outside the boundary, permanently blocked \u2014 not by preference, but by law and contract.<\/figcaption>\r\n<\/figure>\r\n

    A few things matter more than they look:<\/p>\r\n

      \r\n
    • PII handling is a first-class stage, not a wrapper.<\/strong> Masking, redaction, and audit logging sit in the pipeline because the Three Ministries, Two Guidelines require demonstrable confidentiality and integrity, and because a clean audit trail is what lets the hospital pass its own review.<\/li>\r\n
    • The inference server is small on purpose.<\/strong> A 3\u20138B model at 4-bit quantisation serving through vLLM or llama.cpp fits a single on-prem GPU \u2014 which is what makes the whole thing procurable by a hospital IT department rather than a hyperscaler budget.<\/li>\r\n
    • Human sign-off is architectural.<\/strong> The model never writes to the record on its own. The clinician is in the loop by design, which is both the safety property and the regulatory one.<\/li>\r\n<\/ul>\r\n

      05<\/span>Sizing a Model to the Task<\/h2>\r\n

      We do not ask \u201cwhat is the best model?\u201d We ask four questions about the task, and the answers place it on a size tier. Is the task narrow and templated? High-volume? Latency-sensitive? Japanese-heavy? Each \u201cyes\u201d pulls the size down; open-ended reasoning pulls it up. Most regulated production work \u2014 the EMR case included \u2014 lands in the 3\u20138B band, where a single modest GPU does the job.<\/p>\r\n

      \"Model\r\n
      Four questions set the tier; \u201cyes\u201d pushes left. Most regulated production work lands in the 3\u20138B band, where a single modest GPU is enough. VRAM figures are 4-bit (Q4_K_M) estimates including overhead.<\/figcaption>\r\n<\/figure>\r\n

      06<\/span>Quantisation & the Hardware Footprint<\/h2>\r\n

      A model\u2019s weights do not have to run at full 16-bit precision. Quantising to 4-bit \u2014 the widely used Q4_K_M format \u2014 cuts memory roughly in four, for a typical perplexity increase of only about 1\u20133%, imperceptible on a narrow task, and lets a model that would have needed 14 GB run in around 5 GB.<\/p>\r\n

      The rule of thumb we size with: approximately 0.5 GB of VRAM per billion parameters at 4-bit, plus around 15\u201320% for the KV cache and overhead. That single fact reshapes the procurement conversation.<\/p>\r\n

      \r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
      Model size<\/th>\r\nFP16 VRAM<\/th>\r\nQ4_K_M VRAM<\/th>\r\nRuns on<\/th>\r\nOn-prem feasible<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
      ~1.5B<\/td>\r\n~3 GB<\/td>\r\n~1\u20132 GB<\/td>\r\nCPU \/ entry GPU<\/td>\r\nYes \u2014 even CPU-only<\/td>\r\n<\/tr>\r\n
      7\u20138B<\/td>\r\n~14 GB<\/td>\r\n~5\u20137 GB<\/td>\r\n1\u00d7 8 GB GPU (RTX 4060\/3060)<\/td>\r\nYes \u2014 commodity GPU<\/td>\r\n<\/tr>\r\n
      13\u201314B<\/td>\r\n~28 GB<\/td>\r\n~9\u201311 GB<\/td>\r\n1\u00d7 16 GB GPU<\/td>\r\nYes<\/td>\r\n<\/tr>\r\n
      32B<\/td>\r\n~64 GB<\/td>\r\n~22\u201324 GB<\/td>\r\n1\u00d7 24 GB GPU (RTX 4090)<\/td>\r\nYes \u2014 single card<\/td>\r\n<\/tr>\r\n
      70B<\/td>\r\n~140 GB<\/td>\r\n~38\u201340 GB<\/td>\r\nA100\/H100 80 GB, or 2\u00d7 24 GB<\/td>\r\nHeavier \u2014 justify it<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/div>\r\n

      The practical headline: an 8 GB consumer GPU serves a 7\u20138B model at 40+ tokens per second \u2014 fast enough for interactive review \u2014 and CPU-only inference of quantised small models is viable when a GPU is not available, with memory cut by around 75% versus 16-bit precision. None of this requires a cloud contract, and that is the entire point.<\/p>\r\n

      The trap to avoid:<\/strong> quantisation is nearly free at Q4 and Q5, but not below. Q2\/Q3 introduce visible quality loss, and aggressive KV-cache compression degrades long-context summaries \u2014 precisely the EMR case. Size the GPU to hold Q4_K_M comfortably rather than fighting an impossible configuration on undersized hardware.<\/div>\r\n

      07<\/span>Evaluating Japanese-Language Quality<\/h2>\r\n

      This is where overseas general models most often disappoint on Japanese enterprise work \u2014 a model fluent in English can still be quietly wrong in Japanese \u2014 and where Japanese-specialised open models earn their place. We evaluate on three layers, never on vibes:<\/p>\r\n

        \r\n
      • General Japanese capability.<\/strong> Public leaderboards such as the Swallow LLM Leaderboard v2 (JEMHopQA, MMLU-ProX, JHumanEval, M-IFEval-Ja and others) give a defensible starting shortlist rather than marketing claims.<\/li>\r\n
      • Domain capability.<\/strong> General Japanese fluency does not imply clinical or financial competence. Domain benchmarks \u2014 JMedBench for biomedical Japanese, finance-specific evaluations for banking cases \u2014 separate models that merely read Japanese from ones that understand the domain\u2019s Japanese.<\/li>\r\n
      • Task-specific evaluation on the client\u2019s own data.<\/strong> The one that actually decides it: a held-out set of the client\u2019s real, de-identified documents, scored on the metric that matters \u2014 factual accuracy of the summary, no invented findings, correct handling of keigo and clinical shorthand \u2014 with clinicians in the loop.<\/li>\r\n<\/ul>\r\n

        Encouragingly, the Japanese open-model field is strong and moving fast. Continual-pretrained small models hold their own against much larger overseas systems on Japanese tasks, which is the empirical basis for the whole \u201csmall is enough\u201d claim. A credible on-premise shortlist genuinely exists:<\/p>\r\n

        \r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
        Model family<\/th>\r\nSizes<\/th>\r\nOrigin<\/th>\r\nLicense<\/th>\r\nWhere it fits<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
        Sarashina2.2<\/td>\r\n0.5B \/ 1B \/ 3B<\/td>\r\nSB Intuitions<\/td>\r\nMIT<\/td>\r\nStrong ultra-small JP; extraction, drafting<\/td>\r\n<\/tr>\r\n
        Llama-3.1-Swallow<\/td>\r\n8B \/ 70B<\/td>\r\nSwallow (Institute of Science Tokyo)<\/td>\r\nLlama license<\/td>\r\nWorkhorse JP; 8B suits the EMR case<\/td>\r\n<\/tr>\r\n
        PLaMo 2<\/td>\r\n2B \/ 8B \/ 31B<\/td>\r\nPreferred Networks<\/td>\r\nPLaMo Community License<\/td>\r\nCompetitive small JP models<\/td>\r\n<\/tr>\r\n
        LLM-jp-3.1<\/td>\r\n13B \/ 172B; 8\u00d713B MoE<\/td>\r\nNII (national project)<\/td>\r\nApache 2.0<\/td>\r\nJapanese-native, transparent training<\/td>\r\n<\/tr>\r\n
        ELYZA (Llama-3-JP)<\/td>\r\n8B<\/td>\r\nELYZA<\/td>\r\nLlama license<\/td>\r\nEstablished JP instruction model<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/div>\r\n

        Note: Licenses and variants change frequently \u2014 we confirm terms per release before committing a client to a model. The point of the table is not a ranking; it is that a credible on-premise shortlist for Japanese genuinely exists.<\/p>\r\n

        08<\/span>The Accuracy \/ Compliance Trade-Off<\/h2>\r\n

        Plot any option on two axes \u2014 how capable it is, and how much control the client keeps over the data \u2014 and the picture explains itself. The frontier API sits bottom-right: maximal capability, minimal control, and for regulated data that means disqualified. The tuned on-premise small model sits where it needs to: enough capability to clear the task\u2019s accuracy bar, with full data control.<\/p>\r\n

        \"Accuracy\r\n
        The deciding move is not reaching the frontier model\u2019s accuracy \u2014 it is tuning a compliant small model until it clears this task\u2019s \u201cgood enough\u201d bar, then staying in the compliant zone the frontier model can never enter. Accuracy is a threshold to clear, not a score to maximise.<\/figcaption>\r\n<\/figure>\r\n

        Framed this way, the client\u2019s decision is no longer \u201csettle for a weaker model.\u201d It is \u201cmeet the requirement, keep the data, pass the audit.\u201d That is a conversation the frontier vendor cannot have, because their product is defined by the one property \u2014 data leaving the boundary \u2014 that the client cannot accept.<\/p>\r\n

        09<\/span>The Same Shape, Other Sectors<\/h2>\r\n

        The EMR case is the sharpest, but the pattern repeats wherever regulated Japanese data meets a narrow, high-volume task.<\/p>\r\n

        Finance \u2014 Insurance Claims & KYC Narratives<\/h3>\r\n

        Adjusters read claim forms, accident reports, and medical certificates by hand; KYC teams draft suspicious-activity narratives from the most sensitive data a firm holds. Both are extraction-and-drafting tasks \u2014 squarely 3\u20138B territory \u2014 on data that internal risk policy and FISC-aligned controls keep off overseas APIs. An on-premise model auto-extracts the fields and drafts the narrative for a human to confirm.<\/p>\r\n

        Public Sector \u2014 Resident Inquiry & Application Processing<\/h3>\r\n

        Counter and call-centre staff answer the same procedural questions all day and process welfare, permit, and subsidy applications by hand \u2014 work saturated with My Number data and PII that cannot leave municipal systems. A small model over the municipality\u2019s own documents, using RAG, drafts answers and pre-fills forms, entirely inside the network.<\/p>\r\n

        In every case the winning system is not the most capable model in the world. It is the most capable model the client is allowed to run \u2014 sized to the task, quantised to the hardware, and proven on their own data.<\/div>\r\n

        10<\/span>Summary<\/h2>\r\n
          \r\n
        • In regulated Japanese finance, healthcare, and public-sector work, the frontier cloud model is frequently disqualified before capability is even discussed, because APPI, the Three Ministries, Two Guidelines, and client contracts forbid the data leaving the network.<\/li>\r\n
        • Most of these deployments are one narrow, high-volume, Japanese-language task \u2014 not an open-ended assistant \u2014 so a Japanese-capable small or open-weight model tuned to the task is enough.<\/li>\r\n
        • EMR summarisation is the clearest case: there is no compliant overseas-API path at all, so the entire decision is forced onto on-premise ground, with a human clinician signing off every draft.<\/li>\r\n
        • Method: size the model from the task, quantise to Q4_K_M so it fits a single commodity GPU, and evaluate Japanese quality on general, domain, and client-specific data.<\/li>\r\n
        • The accuracy-versus-compliance trade-off is a threshold, not a maximisation: clear the task\u2019s \u201cgood enough\u201d bar while keeping the data inside the boundary \u2014 the one thing the frontier vendor cannot offer.<\/li>\r\n<\/ul>\r\n

          This is part of our ongoing series on deploying enterprise AI under real-world constraints in Japan. Earlier and upcoming articles cover securing the retrieval layer, monitoring for drift after go-live, and the governance practices these systems depend on.<\/p>\r\n

          References<\/h2>\r\n
            \r\n
          • Ministry of Economy, Trade and Industry (METI), DX Report \u2014 Overcoming the \u201c2025 Digital Cliff\u201d<\/em> (2018)<\/li>\r\n
          • Ministry of Health, Labour and Welfare \/ Ministry of Internal Affairs and Communications \/ Ministry of Economy, Trade and Industry, Guidelines for the Security Management of Medical Information Systems<\/em> and related provider guidelines, Version 6.0 (2023)<\/li>\r\n
          • Personal Information Protection Commission, Act on the Protection of Personal Information (APPI) \u2014 provisions on special-care-required personal information<\/li>\r\n
          • Swallow LLM Team (Institute of Science Tokyo), Swallow LLM Leaderboard v2<\/em><\/li>\r\n
          • llm-jp, Awesome Japanese LLM<\/em><\/li>\r\n
          • Jiang et al., JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models<\/em><\/li>\r\n
          • SB Intuitions, Sarashina2.2<\/em>; Preferred Networks, PLaMo 2<\/em>; National Institute of Informatics, LLM-jp-3.1<\/em>; ELYZA, Llama-3-ELYZA-JP-8B<\/em><\/li>\r\n
          • Published llama.cpp \/ vLLM quantisation and VRAM benchmarks for Q4_K_M (GGUF), 2026<\/li>\r\n<\/ul>\r\n<\/div>\r\n","protected":false},"featured_media":3089,"template":"","blog-cat":[24],"class_list":["post-3085","blog","type-blog","status-publish","has-post-thumbnail","hentry","blog-cat-ai","en-US"],"acf":[],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t