The SLM You’re Allowed to Use: On-Premise Small Language Models for Regulated Japan

In Japanese finance, healthcare, and the public sector, the frontier model is often the one option that is off the table before a project even starts. Not because it is too weak, but because it is not allowed near the data. The winning move in these environments is rarely a bigger model. It is a Japanese-capable small model that runs where the data already lives.

Walk into a bank, a hospital, or a municipal office with an AI proposal and you will seldom lose on model quality. You lose on a single sentence the client says early: “That data cannot leave our network.” Three concerns capture where these conversations get caught:

This article makes the case for the model you are actually permitted to run: a Japanese-capable small or open-weight model, deployed on-premise or in a private VPC, and tuned to one narrow task rather than to general capability. We use electronic health record summarisation as the hero example because it is the cleanest version of the constraint, then walk through how we size a model to a task, quantise it to the available hardware, evaluate its Japanese quality, and reason about the accuracy-versus-compliance trade-off with clients. This is part of our ongoing series on deploying enterprise AI under real-world constraints in Japan.

01The Constraint the Client Actually Has

The industry pitch usually starts with capability. The client’s real problem is permission. And it is not caution for its own sake — three separate forces converge on the same answer:

• APPI (Act on the Protection of Personal Information). Medical records, and much of what finance and government hold, are special-care-required personal information. Both handling and cross-border transfer are tightly constrained.
• Sector guidelines. Healthcare providers are expected to follow the safety-management guidelines issued by Japan’s Ministries of Health, Labour and Welfare; Internal Affairs and Communications; and Economy, Trade and Industry — commonly referred to as the “Three Ministries, Two Guidelines.” The healthcare edition is currently at Version 6.0 (2023), its most recent major revision. They require confidentiality, integrity, and availability, plus the authenticity of records, strict supervision of any subcontractor, and, in practice, that data remains in a domestic region.
• Contracts and internal risk policy. Client NDAs and internal rules frequently forbid sending data to an overseas API outright, regardless of what the law technically permits.

This is not a fringe scenario; Japan’s modernisation gap makes on-premise the default posture, not the exception. METI’s 2018 DX Report — the origin of the well-known “2025 Digital Cliff” — estimated that unaddressed legacy systems could cost the economy up to ¥12 trillion a year, with roughly 80% of enterprises still running heavily customised legacy core systems.

Adoption has since caught up on the surface — by 2025 around 80% of Japanese companies reported undertaking DX activities, comparable to the US — yet only about 30% consider their DX successful, and nearly 80% say they have yet to achieve significant business outcomes. The upshot for us is simple: a great deal of the most sensitive Japanese enterprise data is not going to an overseas API any time soon.

02Size Is a Spec, Not a Scoreboard

The instinct — the client’s, and honestly the industry’s — is that a bigger model is a better model. For an open-ended assistant that must do everything, that is roughly true. But almost none of these deployments are open-ended. They are one narrow, repetitive, high-volume task, in Japanese, on data that cannot move.

On that kind of task, a small model that has been continual-pretrained on Japanese and tuned to the specific job routinely matches a frontier general model — while running on a single GPU inside the client’s own walls. The question stops being “how capable is the model?” and becomes “is it capable enough at this one task, and can I run it where the data is?” That second question has exactly one class of answer: Japanese-capable small and open-weight models, deployed on-premise or in a private VPC. Everything below is how we size, deploy, and evaluate them.

03The Hero Case: A Doctor Re-Reading a Chart

Before nearly every consultation, a physician re-reads a long patient history — previous visits, lab results, prescriptions, referral letters, free-text nursing notes — to reconstruct the story in their head. It is slow, it repeats hundreds of times a day across a hospital, and it eats into the minutes that should go to the patient in front of them.

An assistive model that produces a draft summary — “here is this patient in six lines, with the open issues flagged” — for the clinician to review is an obvious, high-value application. It is also the single cleanest illustration of our whole argument, for one reason:

It is honest about difficulty, too. Summarisation is not a trivial extract-a-field task; it is genuinely borderline for a small model, which is exactly why it is worth walking through the methodology rather than hand-waving.

And because summaries can inform clinical judgement, the correct framing is assistive, not autonomous: the model drafts, a clinician signs off. That posture is both the safe design and the one that keeps a human accountable for the medicine.

04Architecture: The Data Never Crosses the Boundary

The reference deployment is deliberately unglamorous. Every component — record store, inference server, review interface — sits inside the hospital network or a private VPC. The only thing that ever leaves is nothing.

A few things matter more than they look:

• PII handling is a first-class stage, not a wrapper. Masking, redaction, and audit logging sit in the pipeline because the Three Ministries, Two Guidelines require demonstrable confidentiality and integrity, and because a clean audit trail is what lets the hospital pass its own review.
• The inference server is small on purpose. A 3–8B model at 4-bit quantisation serving through vLLM or llama.cpp fits a single on-prem GPU — which is what makes the whole thing procurable by a hospital IT department rather than a hyperscaler budget.
• Human sign-off is architectural. The model never writes to the record on its own. The clinician is in the loop by design, which is both the safety property and the regulatory one.

05Sizing a Model to the Task

We do not ask “what is the best model?” We ask four questions about the task, and the answers place it on a size tier. Is the task narrow and templated? High-volume? Latency-sensitive? Japanese-heavy? Each “yes” pulls the size down; open-ended reasoning pulls it up. Most regulated production work — the EMR case included — lands in the 3–8B band, where a single modest GPU does the job.

06Quantisation & the Hardware Footprint

A model’s weights do not have to run at full 16-bit precision. Quantising to 4-bit — the widely used Q4_K_M format — cuts memory roughly in four, for a typical perplexity increase of only about 1–3%, imperceptible on a narrow task, and lets a model that would have needed 14 GB run in around 5 GB.

The rule of thumb we size with: approximately 0.5 GB of VRAM per billion parameters at 4-bit, plus around 15–20% for the KV cache and overhead. That single fact reshapes the procurement conversation.

The practical headline: an 8 GB consumer GPU serves a 7–8B model at 40+ tokens per second — fast enough for interactive review — and CPU-only inference of quantised small models is viable when a GPU is not available, with memory cut by around 75% versus 16-bit precision. None of this requires a cloud contract, and that is the entire point.

07Evaluating Japanese-Language Quality

This is where overseas general models most often disappoint on Japanese enterprise work — a model fluent in English can still be quietly wrong in Japanese — and where Japanese-specialised open models earn their place. We evaluate on three layers, never on vibes:

• General Japanese capability. Public leaderboards such as the Swallow LLM Leaderboard v2 (JEMHopQA, MMLU-ProX, JHumanEval, M-IFEval-Ja and others) give a defensible starting shortlist rather than marketing claims.
• Domain capability. General Japanese fluency does not imply clinical or financial competence. Domain benchmarks — JMedBench for biomedical Japanese, finance-specific evaluations for banking cases — separate models that merely read Japanese from ones that understand the domain’s Japanese.
• Task-specific evaluation on the client’s own data. The one that actually decides it: a held-out set of the client’s real, de-identified documents, scored on the metric that matters — factual accuracy of the summary, no invented findings, correct handling of keigo and clinical shorthand — with clinicians in the loop.

Encouragingly, the Japanese open-model field is strong and moving fast. Continual-pretrained small models hold their own against much larger overseas systems on Japanese tasks, which is the empirical basis for the whole “small is enough” claim. A credible on-premise shortlist genuinely exists:

Note: Licenses and variants change frequently — we confirm terms per release before committing a client to a model. The point of the table is not a ranking; it is that a credible on-premise shortlist for Japanese genuinely exists.

08The Accuracy / Compliance Trade-Off

Plot any option on two axes — how capable it is, and how much control the client keeps over the data — and the picture explains itself. The frontier API sits bottom-right: maximal capability, minimal control, and for regulated data that means disqualified. The tuned on-premise small model sits where it needs to: enough capability to clear the task’s accuracy bar, with full data control.

Framed this way, the client’s decision is no longer “settle for a weaker model.” It is “meet the requirement, keep the data, pass the audit.” That is a conversation the frontier vendor cannot have, because their product is defined by the one property — data leaving the boundary — that the client cannot accept.

09The Same Shape, Other Sectors

The EMR case is the sharpest, but the pattern repeats wherever regulated Japanese data meets a narrow, high-volume task.

Finance — Insurance Claims & KYC Narratives

Adjusters read claim forms, accident reports, and medical certificates by hand; KYC teams draft suspicious-activity narratives from the most sensitive data a firm holds. Both are extraction-and-drafting tasks — squarely 3–8B territory — on data that internal risk policy and FISC-aligned controls keep off overseas APIs. An on-premise model auto-extracts the fields and drafts the narrative for a human to confirm.

Public Sector — Resident Inquiry & Application Processing

Counter and call-centre staff answer the same procedural questions all day and process welfare, permit, and subsidy applications by hand — work saturated with My Number data and PII that cannot leave municipal systems. A small model over the municipality’s own documents, using RAG, drafts answers and pre-fills forms, entirely inside the network.

10Summary

• In regulated Japanese finance, healthcare, and public-sector work, the frontier cloud model is frequently disqualified before capability is even discussed, because APPI, the Three Ministries, Two Guidelines, and client contracts forbid the data leaving the network.
• Most of these deployments are one narrow, high-volume, Japanese-language task — not an open-ended assistant — so a Japanese-capable small or open-weight model tuned to the task is enough.
• EMR summarisation is the clearest case: there is no compliant overseas-API path at all, so the entire decision is forced onto on-premise ground, with a human clinician signing off every draft.
• Method: size the model from the task, quantise to Q4_K_M so it fits a single commodity GPU, and evaluate Japanese quality on general, domain, and client-specific data.
• The accuracy-versus-compliance trade-off is a threshold, not a maximisation: clear the task’s “good enough” bar while keeping the data inside the boundary — the one thing the frontier vendor cannot offer.

This is part of our ongoing series on deploying enterprise AI under real-world constraints in Japan. Earlier and upcoming articles cover securing the retrieval layer, monitoring for drift after go-live, and the governance practices these systems depend on.

References

• Ministry of Economy, Trade and Industry (METI), DX Report — Overcoming the “2025 Digital Cliff” (2018)
• Ministry of Health, Labour and Welfare / Ministry of Internal Affairs and Communications / Ministry of Economy, Trade and Industry, Guidelines for the Security Management of Medical Information Systems and related provider guidelines, Version 6.0 (2023)
• Personal Information Protection Commission, Act on the Protection of Personal Information (APPI) — provisions on special-care-required personal information
• Swallow LLM Team (Institute of Science Tokyo), Swallow LLM Leaderboard v2
• llm-jp, Awesome Japanese LLM
• Jiang et al., JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models
• SB Intuitions, Sarashina2.2; Preferred Networks, PLaMo 2; National Institute of Informatics, LLM-jp-3.1; ELYZA, Llama-3-ELYZA-JP-8B
• Published llama.cpp / vLLM quantisation and VRAM benchmarks for Q4_K_M (GGUF), 2026