2026.06.29
AIセキュリティ
Your AI assistant launched six months ago. It passed the security review. The retrieval layer enforced access control. The answers were good, the demo went well, and the business signed off.
Nothing has crashed since. No alerts have fired. And yet, quietly, the system is no longer the system you shipped. The knowledge base it searches has absorbed thousands of new documents and lost others. The questions users ask have shifted toward topics that barely existed at launch. Some answers it gave confidently last week are now simply out of date, because the underlying facts changed. And if you rely on a hosted model, the model itself may have been updated by your vendor without notice.
This is drift: the slow divergence between the system you validated and the system that is actually running. It is the defining risk of Day 2, everything after go-live, and it is dangerous precisely because it is invisible. A crashed service pages an engineer. A degrading service just gives slightly worse answers to people who have no way of knowing they should have gotten better ones. Three concerns capture where teams get caught:
“It passed every review at launch, but we cannot prove it still performs the way it did.”
“The knowledge base has changed completely since go-live, and nobody re-validated retrieval.”
“Our model provider pushed an update. We found out from a user complaint, not a dashboard.”
This article covers the Day-2 problem: why production AI degrades even when the model never changes, how that degradation shows up, how to detect it before users do, and how to respond. As in earlier posts in this series, we close by mapping these practices to the obligations that now make them mandatory rather than optional, in particular Article 72 of the EU AI Act, whose high-risk-system obligations (Articles 9, 12–15, 26, 72, 73) become applicable from 2 August 2026, and which requires providers of high-risk systems to monitor performance throughout a system’s lifetime. That deadline falls just one month after this article’s publication, so the shift from good practice to legal requirement is imminent rather than hypothetical.
Machine-learning systems have always degraded over time. The research literature calls it concept drift and has studied it for decades. The classic picture is simple: a fixed model meets a slowly changing world, and predictions tuned for yesterday’s data get worse on today’s.
LLM systems break that picture in an important way. In most enterprise deployments the model’s weights are frozen, because you call a pinned API model or run a fixed checkpoint. So “the model” is not drifting in the classical sense. Instead, five other things move, often independently, and the most damaging ones are exactly those that traditional monitoring was never built to catch.
The five axes of drift. The three with a purple bar — corpus, world, vendor — never disturb the incoming question stream, so an input-only monitor cannot see them.
The takeaway is structural. Classical drift is mostly one axis, data versus a fixed model. An LLM system has five, and three of them (corpus, world, vendor) never disturb the incoming question stream. A monitor that only watches inputs is blind to the most common causes of silent enterprise-AI decay.
Drift is abstract; the failures it causes are concrete, and they share one trait. They pass the demo and break in production without throwing an error. The monitoring literature’s founding observation is that ML systems tend to fail silently, so you have to deliberately engineer them to fail loudly.
The core asymmetry: a crash is loud and rare, but degradation is silent and continuous. The whole job of Day-2 monitoring is to turn a slow, invisible quality loss into a discrete alert, before your users absorb the cost.
There is no single drift detector. A robust setup combines three layers, from cheap-and-noisy to expensive-and-authoritative. All three depend on one foundation: instrumenting every inference, logging the query, the retrieved context and its scores, the model and its version, the answer, and any user feedback. There is now an open standard for this telemetry, the OpenTelemetry GenAI conventions, worth adopting so your data is portable.
The three-layer detection stack over a shared telemetry bus: cheap tripwires up front, a quality verdict in the middle, a stable ground-truth anchor at the back.
Watch the distribution of incoming questions, and of the chunks your retriever pulls, and test whether this week looks different from a reference period. In practice you compare the embeddings (the numeric vectors that represent text) using a statistical two-sample test. Common, well-understood options include a kernel test (MMD), a per-dimension distribution test (KS) with a correction for testing many dimensions, or the simple binned Population Stability Index (PSI), where a value above roughly 0.25 is a strong signal that something shifted.
Shift is not the same as harm. A Layer-1 alarm tells you the inputs moved, not that quality dropped. Treat it as a tripwire that tells you where to look. Quality gets decided in Layer 2.
Sample a slice of live traffic and have an LLM “judge” score each answer against a rubric. The RAG-evaluation research gives you the right dimensions to score: faithfulness (what fraction of the answer’s claims are actually supported by the retrieved context, which directly targets the fluent-and-wrong failure), answer relevance (does the answer address the question that was asked), and context precision and recall (did the retriever pull the right passages, and the ones it needed). The last pair separates retrieval failures from generation failures, which is essential when you are chasing root cause.
Track each as a number over time, and alert when it trends down. The point of using an LLM judge is that you can score live answers without writing a gold answer for every query, which is what makes continuous scoring affordable. Two honest caveats keep this credible. First, the judge is itself an LLM, so it can be biased and can drift too; pin the judge model and its prompt, and re-check it periodically against a small set of human-labelled examples so your ruler is not silently changing length. Second, sample enough that the trend is statistically real rather than noise.
Maintain a curated, versioned set of representative questions with known-good expectations, and run it on a schedule and in CI before any change ships. This is the stable baseline the noisy live signals are measured against, and the gate that catches the “small change broke three things” failure. When the world genuinely changes, you update the golden set deliberately, and that update is itself an auditable event.
Alongside these, watch the cheap operational indicators you already have: retrieval scores, refusal and fallback rates, citations per answer, tool-call errors, latency, and thumbs-down feedback. None is a verdict on its own, but together they form an early-warning surface. And rather than alarm on every dip, smooth the stream and alert only when the smoothed value crosses a statistically derived limit, the same control-chart idea (EWMA or CUSUM) that factories have used for decades to catch a process slowly going out of spec.
Detecting drift only helps if you can act without making things worse. Borrow the discipline mature software teams already use, adapted for AI.
Shadow runs a candidate on real traffic with zero user impact; canary exposes a small slice under watch, then results are compared before promotion or rollback.
Day-2 monitoring quietly reopens the data-protection problem the retrieval-security post in this series worked to close. The moment you start logging prompts, retrieved context and outputs, and sampling them through an LLM judge, you have created a second copy of sensitive data, often under weaker controls than the source system; and if the judge is a third-party model, that data may leave your boundary. Two of your launch-day controls also drift: permissions go stale as people change roles, and redaction rules miss new document formats as they arrive. After go-live, model quality is not the only thing that decays, so does your access control. Monitor all three.
These practices are no longer just good hygiene. Continuing the mapping from earlier posts, they line up with specific obligations under the EU AI Act and recommendations in Japan’s AI Business Operator Guidelines v1.2. The anchor is Article 72, which requires providers of high-risk systems to run a documented post-market monitoring system that collects and analyses performance data across the system’s lifetime.
| Control | EU AI Act (high-risk systems) | AI Guidelines for Business v1.2 |
|---|---|---|
| Continuous post-deployment monitoring | Art. 72 (Post-market monitoring) | Recommended ongoing-review practices for Provider and User roles |
| Maintaining accuracy & robustness over time | Art. 15 (Accuracy, robustness, cybersecurity) | Recommended technical-robustness practices for Provider role |
| Logging inferences to enable monitoring | Art. 12 (Record-keeping); Art. 26 (Deployer logs, min. six months) | Recommended transparency and accountability practices across all three roles |
| Detecting and reporting serious failures | Art. 73 (Serious-incident reporting) | Recommended incident-response practices for Provider role |
| Corrective action on detection | Art. 20 (Corrective actions by providers) | Recommended governance practices across Developer and Provider roles |
Note: The EU AI Act is a legally binding regulation. Japan’s AI Guidelines for Business is a voluntary, non-binding guideline that businesses are encouraged, but not legally required, to follow.
The mapping is illustrative, not exhaustive, but the direction is unambiguous. A system you can monitor, in which you can detect degradation and act on it, has moved from good practice to a baseline expectation for passing compliance review. This article is provided for general information and does not constitute legal advice; consult qualified counsel to confirm current obligations and applicable deadlines for your jurisdiction and system classification.
None of this needs exotic technology. It applies the operational discipline organizations already use for other production systems to the parts of the AI pipeline where it has usually been skipped.
This is the fourth article in our series on building AI systems ready for enterprise deployment. Earlier articles covered the five failure patterns and securing the retrieval layer; the next will continue with the production defenses these systems depend on.
