{"id":3117,"date":"2026-08-24T14:55:25","date_gmt":"2026-08-24T05:55:25","guid":{"rendered":"https:\/\/aktsk.ai\/?post_type=blog&p=3117"},"modified":"2026-08-24T14:56:46","modified_gmt":"2026-08-24T05:56:46","slug":"%e8%aa%b0%e3%82%82%e4%b8%ad%e8%ba%ab%e3%82%92%e7%9f%a5%e3%82%89%e3%81%aa%e3%81%84%e3%83%ac%e3%82%ac%e3%82%b7%e3%83%bc%e3%82%b7%e3%82%b9%e3%83%86%e3%83%a0%e3%82%92%e3%80%81llm%e3%81%a7%e8%aa%ad","status":"publish","type":"blog","link":"https:\/\/aktsk.ai\/en\/blog\/3117\/","title":{"rendered":"Decoding Legacy Systems No One Understands with LLMs: A New Approach to the \u201cBlack Box\u201d Problem"},"content":{"rendered":"
\n

Every large Japanese enterprise has at least one system like this. It runs the business. It has run the business for thirty years. It works. And there is no longer a single person in the building who can tell you, with confidence, what it does.<\/p>\n

The documentation describes a version replaced in 2009. The engineers who wrote it have retired. The SIer who maintained it has rotated its staff four times. What remains is behaviour \u2014 observable, load-bearing, unexplained.<\/p>\n

Three sentences we hear in the first meeting \u2014 composite paraphrases of a recurring conversation, not quotations from any single client:<\/p>\n

“We know what it outputs. We don’t know why.”<\/div>\n
“The one person who understood the settlement batch retired this spring.”<\/div>\n
“We were quoted a multi-year engagement just to find out what’s in there.”<\/div>\n

That last one is the real problem. Before a modernisation project can be scoped, someone has to read the system \u2014 and reading it has historically meant paying a team of engineers a year to do archaeology. This article is about why that discovery phase is now a different kind of problem, what an LLM can and cannot do to it, and the methodology that separates a useful documentation exercise from a confidently wrong one.<\/p>\n

This is part of our ongoing series on deploying enterprise AI under real-world constraints in Japan. The previous post<\/a> argued for small on-premise models where data cannot move. This is the opposite case: a task where you want the largest model you can legally point at the problem.<\/p>\n

\n

01<\/span>\u00a0\u00a0The Constraint Is Comprehension, Not Code<\/h2>\n

The framing that has dominated this conversation for eight years is METI’s 2018 DX Report and its \u300c2025\u5e74\u306e\u5d16\u300d \u2014 the warning that unaddressed legacy systems could cost the Japanese economy up to \u00a512 trillion a year from 2025 onward. The figure is widely quoted and, at this point, widely stale.<\/p>\n

The more useful question is what happened after the deadline passed, and there is a current answer. METI convened a \u30ec\u30ac\u30b7\u30fc\u30b7\u30b9\u30c6\u30e0\u30e2\u30c0\u30f3\u5316\u59d4\u54e1\u4f1a (Legacy System Modernisation Committee), whose \u7dcf\u62ec\u30ec\u30dd\u30fc\u30c8 was published on 28 May 2025. It carries a market survey of roughly 4,000 user and vendor companies \u2014 799 responses, fielded 17 December 2024 to 14 February 2025 \u2014 and reports that 61% of user companies still hold legacy systems, rising to 74% among large enterprises.<\/strong> The picture is not a cliff anyone fell off; it is a plateau nobody has climbed.<\/p>\n

Two details in that report matter more than the headline percentage.<\/p>\n

First, its definition of a legacy system does not turn on age at all. It names five factors, and one of them is \u30d6\u30e9\u30c3\u30af\u30dc\u30c3\u30af\u30b9\u5316 \u2014 specifications and design documents not maintained, migration and rebuild obstructed, maintenance become \u5c5e\u4eba\u7684 (dependent on specific individuals). The problem this article is about is not an unfortunate side effect of old systems; in the government’s own taxonomy it is one of the defining properties of the category.<\/p>\n

Second, the same report explicitly calls on vendors to develop \u751f\u6210AI\u306b\u3088\u308b\u30ec\u30ac\u30b7\u30fc\u30b3\u30fc\u30c9\u89e3\u6790 \u2014 generative-AI legacy code analysis \u2014 on the express grounds that labour-intensive migration approaches will soon reach their limit as the IT workforce shrinks. The approach this article describes is not a vendor’s clever idea; it is the direction the policy documents are pointing.<\/p>\n

Underneath the percentages sits a knowledge-transfer problem with a hard clock on it. The engineers fluent in COBOL, PL\/I, and the in-house job control that glues these systems together are reaching retirement age now. The difficulty is not a shortage of capable engineers \u2014 plenty of skilled people work productively in these technologies, and the languages themselves are learnable. The difficulty is that what these systems demand is largely undocumented and organisation-specific<\/em>: forty years of accumulated decisions, exceptions, and workarounds that exist in no manual and are transferable mainly by working alongside the people who hold them. Neither hiring nor training shortcuts that, because there is no external curriculum for one company’s undocumented batch schedule. METI’s report names the same mechanism, listing the ageing and departure of engineers alongside the absence of design documentation. Every retirement that happens without a structured handover takes comprehension with it.<\/p>\n

So the binding constraint is not that the code is old. Old code that is understood is merely a maintenance cost. The binding constraint is that comprehension is decaying faster than the code is being replaced<\/strong>, and every year of delay makes the next attempt more expensive than the last.<\/p>\n

02<\/span>\u00a0\u00a0Why This Is Genuinely an LLM Problem<\/h2>\n

We are on record arguing against reaching for an LLM by default. This is the case where it is the right instrument, for a specific reason.<\/p>\n

Legacy comprehension is not a code-generation task. It is a reading comprehension task over a large, messy, low-signal corpus in a dead dialect<\/strong> \u2014 200,000 lines of COBOL, forty years of stored procedures, JCL, batch schedules, and comments in Japanese written by people who assumed the reader shared their context. Static analysis handles the mechanical part well: it can build a call graph and trace a variable. What it cannot do is answer the question the client actually has \u2014 “what business rule is this implementing, and why?<\/em>“<\/em><\/p>\n

That needs a model that can hold a large amount of unfamiliar context at once, tolerate inconsistency, and read Japanese comments alongside English identifiers. It is precisely the shape of task frontier LLMs became good at, which is why the economics changed here before the tooling did.<\/p>\n

The second shift is agentic. Earlier attempts required the codebase be cleaned and normalised before analysis \u2014 its own multi-year project, and the reason most of these efforts died at the proposal stage. An agentic system navigates a messy repository the way an engineer does: follow a call, open the referenced copybook, check the job schedule, come back. It works on the code as it actually exists.<\/p>\n

One caveat governs everything below.<\/strong> In finance and the public sector, this code usually cannot be sent to an overseas API \u2014 but it is worth being precise about why, because the reasons are not interchangeable and clients notice when a vendor blurs them.<\/p>\n

Source code is not personal information, and APPI does not restrict it as such. The binding constraints on the code itself are almost always contractual and policy-based<\/strong>: client NDAs, subcontracting and re-delegation terms, sector security guidelines, and internal rules that prohibit source leaving the corporate network. APPI becomes relevant at a different point \u2014 when personal data is actually present in what you transmit, which in practice it often is: personal data embedded in configuration, test fixtures, log samples, seed data, or production-derived extracts travelling alongside the code. Those are separate obligations, with separate owners inside the client, and treating them as one thing tends to produce a review that fails on the detail you glossed over. Either way, the architecture must support private deployment (VPC-hosted frontier models, or on-premise open-weight models) as the default, not the exception.<\/p>\n

03<\/span>\u00a0\u00a0The Hero Case: The Settlement Batch Nobody Will Touch<\/h2>\n

The example that follows is an illustrative composite \u2014 the recurring shape of these systems, assembled to make the methodology concrete. It is not a description of a specific client engagement, and the quantities are indicative rather than measured.<\/em><\/p>\n

Take a nightly batch job of a few tens of thousands of lines that reconciles transactions and writes to the ledger. It has run without incident for two decades. It may encode several hundred business rules, of which only a fraction are documented. Some implement regulation that has since changed; some implement a workaround for a hardware limitation that no longer exists. Nobody can tell which is which.<\/p>\n

Nobody will touch it either, because the downside of a wrong change is a regulatory incident and the upside is a slightly cleaner codebase. So it sits there and the organisation routes around it \u2014 which is exactly how a black box becomes load-bearing.<\/p>\n

The goal is not to migrate it on day one. It is to make it legible<\/strong>: to produce, in weeks rather than quarters, a documented inventory of what it does, what depends on it, and which rules are still live \u2014 so the migration decision can be made on evidence instead of fear.<\/p>\n

04<\/span>\u00a0\u00a0Architecture: Extraction, Then Verification<\/h2>\n

The pipeline is deliberately unglamorous, and it has one non-negotiable property: nothing generated by the model is treated as fact until something other than the model has confirmed it.<\/strong><\/p>\n

\n

EXTRACTION \u00a0\u2014\u00a0 PRODUCES A DRAFT, NOT A FACT<\/p>\n

\n
Source repo<\/strong>COBOL \/ JCL \/ procs<\/span><\/div>\n
\u2192<\/div>\n
Static analysis<\/strong>call graph, deps<\/span><\/div>\n
\u2192<\/div>\n
LLM comprehension<\/strong>reads the code<\/span><\/div>\n
\u2192<\/div>\n
Rule inventory<\/strong>draft<\/span> \u2014 unverified<\/span><\/div>\n<\/div>\n

VERIFICATION \u00a0\u2014\u00a0 TWO GATES, NOT TWO STEPS<\/p>\n

\n
Rule inventory<\/strong>draft<\/span><\/div>\n
\u2192<\/div>\n
Expert sign-off<\/strong>gate 1<\/span><\/div>\n
\u2192<\/div>\n
Behaviour diff<\/strong>vs production \u2014 gate 2<\/span><\/div>\n
\u2192<\/div>\n
Strangler-fig<\/strong>migration<\/span><\/div>\n<\/div>\n

The same rule inventory, before and after the gates. Everything the LLM produces enters as a draft; only the expert sign-off and the behaviour diff turn it into something a migration may rely on. No harness, no migration.<\/strong><\/p>\n<\/div>\n

Four things matter more than they look:<\/p>\n

Static analysis runs first, not instead.<\/strong> The call graph, dependency map, and dead-code detection come from deterministic tooling. That is the skeleton the model reasons over, and also the check on it \u2014 if the LLM describes a data flow the call graph does not contain, that is a caught hallucination rather than a shipped one.<\/p>\n

Output is structured, not prose.<\/strong> The model emits a rule inventory with fields \u2014 rule ID, source location, plain-language description, inputs, outputs, confidence, open questions \u2014 not a narrative document. Structured output can be diffed, reviewed row by row, and tracked to closure. A forty-page generated document cannot be verified by anyone, which is why it never is.<\/p>\n

Confidence is a first-class field.<\/strong> Low-confidence rules route to human review first. This inverts the usual failure mode, where reviewer attention is spread evenly across output that is 95% trivially correct and 5% quietly wrong.<\/p>\n

The human checkpoint is architectural.<\/strong> A domain expert \u2014 often a semi-retired engineer brought back for exactly this \u2014 signs off on the inventory. It is the highest-value use of a retiring expert’s remaining time: not writing documentation from scratch, which is slow and which they will not do, but correcting a draft<\/em>, which is fast and which they will.<\/p>\n

05<\/span>\u00a0\u00a0Methodology: How We Reverse-Engineer Safely<\/h2>\n

Scope by blast radius, not by size.<\/strong> Start with the subsystem whose failure would be most expensive, because that is where comprehension is worth the most \u2014 not with the easiest module, which is the instinct and which produces a successful pilot that proves nothing.<\/p>\n

Behaviour-diff against production behaviour is the only proof that counts.<\/strong> Generated documentation is a hypothesis. The test is to run the legacy system and the candidate replacement against the same inputs and diff the outputs field by field.<\/p>\n

Those inputs have to be production-derived<\/em> rather than synthetic, because synthetic cases only exercise the behaviour someone already knew to write down \u2014 which is precisely not where the risk is. In a regulated environment, production-derived does not mean production data copied into a test rig. It means an extract handled under the client’s own data-handling rules: masked or anonymised wherever personal data is present, scoped to the fields the diff actually needs, access-restricted to the migration team, retained for a defined window, logged, and used under explicit authorisation from whoever owns that data internally. Where no extract can be approved at all, the harness runs inside<\/em> the production boundary \u2014 the candidate implementation shadows live traffic in place, and only the diff results leave. We do not migrate anything for which we cannot build this harness under those controls.<\/strong> That rule has ended engagements, and it should.<\/p>\n

Strangler fig, never big bang.<\/strong> Route one function at a time to the new implementation with the legacy path still live and comparing in shadow mode. The old system stays authoritative until the new one has matched it on real traffic for a meaningful period. Slower to demo, dramatically cheaper to be wrong in.<\/p>\n

\n

STRANGLER FIG \u00a0\u2014\u00a0 THE LEGACY PATH STAYS AUTHORITATIVE<\/p>\n

\n
Phase 1<\/strong>Legacy writes.<\/span>
\nNew impl shadows + compares.<\/span><\/div>\n
\u2192<\/div>\n
Phase 2<\/strong>One function cut over.<\/span>
\nLegacy still comparing.<\/span><\/div>\n
\u2192<\/div>\n
Phase 3<\/strong>Legacy retired \u2014<\/span>
\nonly after sustained match.<\/span><\/div>\n<\/div>\n

Never a big-bang cutover. The new implementation runs in shadow against real traffic and is compared, not trusted, until it has matched the legacy path for a meaningful period.<\/p>\n<\/div>\n

Documentation is the deliverable, even if migration never happens.<\/strong> A verified rule inventory has standalone value: it makes the system maintainable, de-risks the next audit, and means the next retirement is not another permanent loss.<\/p>\n

06<\/span>\u00a0\u00a0Honest Limits \u2014 Where This Does Not Apply<\/h2>\n

The failure mode of this technology is specific, and it is worth naming precisely: an LLM reading unfamiliar code produces fluent, plausible, confidently wrong explanations.<\/strong> It will describe what code of that shape usually does. When a routine contains a twenty-year-old undocumented workaround \u2014 which is the exact thing you are paying to discover \u2014 the model’s prior actively works against you. It is at its least reliable precisely where it is most needed.<\/p>\n

Concretely, this approach does not carry its weight when:<\/p>\n

\u25aa<\/span>\u00a0You cannot build a behaviour-diff harness.<\/strong> No production traffic to replay, no test environment, no way to compare. Without a verification path you have generated plausible fiction about a system that runs your business. Stop.<\/p>\n

\u25aa<\/span>\u00a0The system is small enough to read.<\/strong> If a maintainer is still employed and a small team could read the thing end to end in a matter of weeks, doing exactly that is cheaper and more reliable than building a pipeline around it. Where that threshold falls depends on the language, the density of the code, and how much of the original team remains \u2014 we size it per system rather than against a line count.<\/p>\n

\u25aa<\/span>\u00a0The real dependency is a person, not the code.<\/strong> If the critical knowledge is operational \u2014 which<\/em> exception is safe to ignore at month-end \u2014 it is not in the source, and no amount of code comprehension recovers it. That is a tacit-knowledge capture<\/a> problem, and a different engagement.<\/p>\n

\u25aa<\/span>\u00a0The code cannot leave the network and the on-prem hardware is not there.<\/strong> Comprehension quality is more model-dependent here than in most tasks. A small local model produces a worse inventory, and pretending otherwise is how a compliance constraint becomes a quality problem nobody admits to.<\/p>\n

\u25aa<\/span>\u00a0Regulatory sign-off requires human-authored evidence.<\/strong> In some contexts AI-generated documentation cannot be the artefact of record regardless of its accuracy. It can still accelerate the humans who write it \u2014 but scope it as acceleration, not as the deliverable.<\/p>\n

07<\/span>\u00a0\u00a0Summary<\/h2>\n

\u25aa<\/span>\u00a0Japan’s legacy problem has moved past its headline deadline without being solved: METI’s May 2025 committee report finds 61% of user companies still holding legacy systems, and 74% of large enterprises.<\/p>\n

\u25aa<\/span>\u00a0The binding constraint is comprehension, not code age \u2014 \u30d6\u30e9\u30c3\u30af\u30dc\u30c3\u30af\u30b9\u5316 is one of the defining factors in METI’s own definition of a legacy system, and the same report calls for generative-AI legacy code analysis by name.<\/p>\n

\u25aa<\/span>\u00a0The knowledge at risk is undocumented and organisation-specific, so retirement without a structured handover removes it permanently. This is a handover problem, not a shortage of capable engineers.<\/p>\n

\u25aa<\/span>\u00a0Legacy comprehension is a genuine LLM task \u2014 large-context reading over a messy corpus in a dead dialect \u2014 and agentic systems can now navigate the code as it exists rather than requiring it be cleaned first.<\/p>\n

\u25aa<\/span>\u00a0The architecture that works puts static analysis first, emits a structured rule inventory rather than prose, treats confidence as a first-class field, and puts a domain expert’s sign-off in the path.<\/p>\n

\u25aa<\/span>\u00a0Nothing migrates without a behaviour-diff harness built on production-derived inputs under the client’s data-handling controls, and migration proceeds strangler-fig with the legacy path live in shadow.<\/p>\n

\u25aa<\/span>\u00a0The honest limit: an LLM reading unfamiliar code is confidently wrong exactly where the undocumented workarounds are. Verification is not a phase of this method \u2014 it is<\/em> the method.<\/p>\n

\n

References<\/p>\n

\u25aa<\/span>\u00a0Ministry of Economy, Trade and Industry (METI), DX Report \u2014 Overcoming the “2025 Digital Cliff” (\u300c2025\u5e74\u306e\u5d16\u300d)<\/em> (2018) \u2014 meti.go.jp<\/a><\/p>\n

\u25aa<\/span>\u00a0\u7d4c\u6e08\u7523\u696d\u7701 \u5546\u52d9\u60c5\u5831\u653f\u7b56\u5c40 \u60c5\u5831\u7523\u696d\u8ab2 \u60c5\u5831\u51e6\u7406\u57fa\u76e4\u7523\u696d\u5ba4, DX\u306e\u73fe\u5728\u5730\u3068\u30ec\u30ac\u30b7\u30fc\u30b7\u30b9\u30c6\u30e0\u8131\u5374\u306b\u5411\u3051\u3066 \u2014 \u30ec\u30ac\u30b7\u30fc\u30b7\u30b9\u30c6\u30e0\u30e2\u30c0\u30f3\u5316\u59d4\u54e1\u4f1a \u7dcf\u62ec\u30ec\u30dd\u30fc\u30c8<\/em>, 2025\u5e745\u670828\u65e5 \u2014 legacy-system definition p.7; survey method p.15 (c. 4,000 companies surveyed, 799 responses, fielded 2024.12.17\u20132025.02.14); 61% \/ 74% retention figures p.16; generative-AI code analysis recommendation p.36 \u2014 ipa.go.jp<\/a><\/p>\n

\u25aa<\/span>\u00a0\u72ec\u7acb\u884c\u653f\u6cd5\u4eba\u60c5\u5831\u51e6\u7406\u63a8\u9032\u6a5f\u69cb\uff08IPA\uff09, 2024\u5e74\u5ea6\u30bd\u30d5\u30c8\u30a6\u30a7\u30a2\u52d5\u5411\u8abf\u67fb<\/em> \u2014 underlying response data for the above \u2014 ipa.go.jp<\/a><\/p>\n

\u25aa<\/span>\u00a0IPA \u72ec\u7acb\u884c\u653f\u6cd5\u4eba\u60c5\u5831\u51e6\u7406\u63a8\u9032\u6a5f\u69cb, DX\u52d5\u54112025 \u2014 \u65e5\u7c73\u72ec\u6bd4\u8f03\u3067\u63a2\u308b\u6210\u679c\u5275\u51fa\u306e\u65b9\u5411\u6027<\/em> \u2014 ipa.go.jp<\/a><\/p>\n

\u25aa<\/span>\u00a0\u7d4c\u6e08\u7523\u696d\u7701, \u30c7\u30b8\u30bf\u30eb\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30e1\u30fc\u30b7\u30e7\u30f3\u8abf\u67fb\uff08DX\u8abf\u67fb\uff092026<\/em>, 2025\u5e7411\u6708 \u2014 meti.go.jp<\/a><\/p>\n<\/div>\n<\/div>\n","protected":false},"featured_media":3118,"template":"","blog-cat":[22],"class_list":["post-3117","blog","type-blog","status-publish","has-post-thumbnail","hentry","blog-cat-architecture","en-US"],"acf":[],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t