Setting up AI agent evaluation in production looks like solving a problem. The agent shipped, the eval layer followed, and the dashboard started returning green scores every morning.
That dashboard is probably lying to you. And you have no mechanism to detect it.
This is the third article in this series. The first named what to automate; the second built the harness and named its instruments. This one asks what keeps those instruments honest after six months of live traffic. Every serious practitioner guide in this space tells you how to build an evaluation system. None of them ask what happens to that system afterward.
There are two distinct engineering problems here, and most teams only name one of them. The first: is the agent producing correct outputs right now? That problem has a literature. The second: is the evaluation mechanism itself still valid? That problem runs silently, accumulates without detection, and surfaces downstream before anyone looks at the harness that was supposed to catch it.
Completion is not correctness
A system processing thousands of tasks per day generates log and trace volumes no team reads end to end. The default response is sampling - and sampling catches visible failures, not gradual degradation, not rare but consequential errors, not failures that appear only under specific input combinations. 72% of enterprise AI teams strongly believe comprehensive testing drives reliability; only 15% achieve elite eval coverage. The gap between confidence and actual coverage is not exceptional. It is the baseline condition of AI agent verification at scale.
Completion is not correctness.
This is the precise version of the problem. A flight-booking agent returns a success signal. The workflow finishes. The dashboard stays green. Whether a reservation actually exists in the target database is a separate question - one that the completion signal does not answer. Anthropic's engineering team names this the transcript-versus-outcome gap: what the agent says happened and what happened in the environment are two different things.
That gap is not narrow. Label Studio, citing Carnegie Mellon and Salesforce research, found that multi-step AI agents produce incorrect results roughly 70% of the time even while returning completion signals that look like success. The figure is secondhand - the primary study was not independently verified - but the direction is consistent with what production logs show.
Testing before ship doesn't close this. AI agent monitoring production is a different engineering problem: it begins the moment the agent goes live and runs for as long as it does. Resolve every edge case in staging and the verification problem still starts fresh at launch. This is where the two-layer structure comes in.
Two verification problems, not one
The first problem is widely recognized: is the agent producing correct outputs right now? This is what automated AI agent evaluation is designed to solve. Anthropic's engineering team has published the canonical practitioner account of this - three grader types, the mechanics of building continuous eval pipelines, the pre-ship vs. post-ship distinction. Nearly all practitioner literature lives in this space.
The second problem is not widely recognized: is the automated evaluation mechanism itself still reliable? InfoQ's five-pillar evaluation framework is the most thorough practitioner guide on building evaluation systems available. It covers intelligence and accuracy, reliability and resilience, user experience. It never asks what happens to that system over time. Longitudinal validity, criteria drift, and judge miscalibration are absent from the entire treatment.
The field data confirms this gap. A systematic review of 84 papers (2023-2025) found 83% of production deployments tracked capability metrics; only 30% tracked human-centred evaluation axes. The monitoring of the monitoring system is nearly absent in the field.
Both problems require separate engineering investment and separate operational practices. Most teams are working on the first. Almost none have built the second. The next section describes what solving the first problem actually looks like in production - and where its limits begin.
First-order: AI agent evaluation in production
Most teams building at scale arrive at the same architecture. Anthropic's engineering team names three grader types:
- Code-based graders catch schema violations, length violations, and prohibited strings - fast and objective, but brittle to valid output variations and blind to semantic correctness.
- Model-based graders, the LLM-as-judge approach, evaluate intent alignment and reasoning quality at volume; this is the dominant mechanism when semantic evaluation must scale without human reviewers reading every output.
- Human graders set the quality standard but are too slow and expensive for continuous production use. They are reserved for calibration.
That layered stack produces numbers. The operational discipline is what those numbers mean. Across 100+ enterprise AI agent deployments, the consistent principle is: track trends and deltas, not absolute scores. A score dropping from 0.85 to 0.78 over a week surfaces emerging problems that a static threshold alert at 0.75 misses entirely.
The architecture has a blind spot. Standard metrics fail to detect four of seven named production failure modes outright: silent tool degradation, distribution collapse, explanation-decision decoupling, and latency-driven correctness erosion. All four look locally correct. All four corrupt downstream processes. Six additional failure classes slip past final-output checks because the output itself appears fine:
- tool-entropy collapse, when the distinct-tool-argument ratio drops below 0.4
- role drift
- execution gap
- illegal transition
- excessive loop
- dead end
Automated AI agent evaluation is necessary. It is not sufficient. The part it does not catch is exactly where the second-order problem begins.
LLM-as-judge: the mechanism and its structural risks
LLM-as-judge is the only mechanism that scales to semantic evaluation without human reviewers reading every output. That load-bearing role is exactly why its failure modes matter.
Five bias types carry empirically measured magnitudes:
- Position bias swings pairwise evaluations 10-15 points - structural, autoregressive, and prompt-resistant.
- Verbosity bias inflates preference for longer responses by 15-30 points.
- Self-preference rates the judge's own model family 10-25% higher.
- Format bias moves scores 5-15 points for rubric-format matches.
- Calibration drift shifts the mean 3-8 points on a judge model version update alone.
When multiple judges run on the same evaluation set, divergence surfaces: one judge showing notably lower alignment than peers on identical examples. Invisible when watching any single judge in isolation.
The most dangerous production failure is not any single bias. Deepchecks names it:
"Ranking improves while real quality degrades."
The judge returns better scores. Human reviewers observe worse outputs. The dashboard and the ground truth have decoupled - and the dashboard stays green.
The cardinal mistake compounds this: using the same model family as both agent and judge. Self-preference inflation then masks performance gaps uniformly across every dataset. The system looks clean across all evaluation families and dirty in none - until external cross-family evaluation exposes it.
These are not arguments against LLM-as-judge. They are arguments for LLM-as-judge calibration and for treating LLM evaluation drift as an engineering problem with a lifecycle. That lifecycle has a name. The next section calls it evaluation rot.
Evaluation rot: the second-order problem
Evaluation rot is the process by which an evaluation layer loses validity while continuing to return numbers. Not a configuration bug. Not a testing gap. A structural property of any evaluation system that is not actively maintained.
Four failure modes define how it unfolds. Each has a distinct signature. Each goes undetected for the same reason: the eval keeps running, the scores keep arriving, and from the outside the system looks fine.
Criteria drift. The rubric goes stale. From over 100 enterprise deployments, query intent shift - users asking fundamentally different questions than the evaluation sets assumed at design - causes recall and precision to decline without triggering any threshold alert. The underlying mechanism: ground truth shifts as reference content changes constantly. The eval ends up checking for the right answer to a question whose correct answer has already changed.
Test case distribution shift. The golden test sets stop representing real inputs. TrustEvals puts it directly:
"a frozen golden dataset becomes a paper tiger inside six to nine months as the model, the user population, and the edge cases shift."
Quarterly minimum refresh, with event-triggered runs on model swaps, vendor patches, prompt edits, or production incidents. Three patterns drive test case distribution shift: index drift, query intent shift, and chunking strategy mismatch. Teams overestimate their actual coverage by 20-40 percentage points - invisible without external validation against live failure types.
Judge miscalibration. The LLM-as-judge loses alignment with human ground truth. A 3-8 point mean shift on a judge model version change is the expected magnitude - same rubric, different model priors, different scoring distribution. This is LLM evaluation drift in its measurable form. GovTech Singapore's MetaEvaluator revealed the same detection problem: divergence between judges is invisible in any single judge's scores and surfaces only through cross-judge comparison. Uncalibrated LLM judges "produce confident but incorrect evaluations" - stable, self-consistent scores that have quietly decoupled from human judgment.
Coverage gaps. New failure modes emerge that the harness was not built to catch. Seven production-specific failure modes exist; standard metrics miss four entirely - failures that look locally correct while corrupting downstream processes. "Eval saturation" is the disguised version: a 100% pass rate means the eval "tracks regressions but provides no signal for improvement." New failure modes accumulate uncovered. Six agent failure classes slip past final-output checks because the final output looks correct: tool-entropy collapse, role drift, execution gap, illegal transition, excessive loop, dead end. Post-incident analysis returns one explanation: "We thought this was safe." Coverage gaps concentrate in exactly those workflows.
This is eval rot. Not a failure of initial setup. A failure of maintenance. The meta-evaluation loop is the structural response.
Closing the meta-evaluation loop
The meta-evaluation loop is not a separate system. It is a discipline applied to the eval stack that already exists: periodically measuring whether the evaluator is still aligned with human judgment. This is the missing half of production AI agent observability - the part that keeps the instruments honest.
GovTech Singapore's MetaEvaluator gives the practical template. Three steps:
- Run multiple LLM judges on the same evaluation set
- Gather human annotations as ground truth
- Compute alignment metrics - accuracy, Cohen's Kappa, and semantic similarity between judge outputs and human judgments
The signal is divergence from human annotation. A judge that was well-aligned at deployment and is now diverging has lost calibration. That divergence surfaces before a production incident does.
Judge contract pinning makes drift traceable. Express the judge as the tuple: judge_model_id, rubric_version, prompt_template_hash. Recalibrate against human labels on every contract change. Treat model upgrades as deliberate migrations with an explicit calibration step - not automatic configuration changes. An unverified model bump changes the scoring system without anyone noticing.
Versioning discipline handles the slower drift. Track evaluation prompts with explicit version strings - "eval-criteria-v2-2026-06" - so a drifting judge surfaces as falling agreement between the current judge version and the human reference panel. The string makes the drift observable.
Refresh cadence: quarterly minimum for the golden test sets, with event-triggered runs on model swaps, vendor patches, prompt edits, or production incidents. Between events, the dataset decays; the quarterly floor stops the slow drift from compounding unnoticed.
83% of production systems track capability metrics; only 30% track human-centred evaluation axes. Closing the loop means building into that 30%. Seven observable signals confirm when the loop has slipped.
Signals your AI agent evaluation in production is losing integrity
Any one of these seven signals means AI agent evaluation in production is no longer trustworthy. All are visible before a production incident reveals the problem with AI agent output quality.
Scores improve; outcomes don't. Eval scores trend up while human spot-checks or downstream metrics don't follow. The gap between those trend lines is the signal, not either in isolation.
Inter-judge divergence rises. Multiple judges run on the same sample disagree above a meaningful threshold. Stable agreement is a health indicator; divergence is a warning.
Score distribution narrows without pass-rate change. FM-3 distribution collapse narrows output patterns 6.5x without triggering threshold alerts. Variance drops while the mean holds flat.
Pass rate hits 100%. Saturation means the eval "tracks regressions but provides no signal for improvement." Treat 100% as a warning that coverage has run out.
Judge model upgraded without recalibration. A 3-8pt mean shift is the expected magnitude. Scores after an unverified bump should not be trusted until recalibration is done.
Human spot-check agreement rate declines. A falling Cohen's Kappa indicates judge drift. Static monitoring detects goal-drift scenarios at a median of 12.3 seconds; adaptive monitoring cuts this to 5.6.
Coverage appears stable while the task distribution has shifted. Teams overestimate actual coverage by 20-40 percentage points. A quarterly audit comparing live failure types against the test suite reveals the gap.
The harness does not stay honest by itself
The prior article in this series built the evaluation harness and named its instruments: guides that shape agent behavior, sensors that observe it. This article has made the harder argument.
Sensors are only as honest as the discipline that maintains them. Evaluation rot is not a setup failure - it is what happens when a correctly-built evaluation layer is treated as done rather than as a running system that needs the same ongoing maintenance as the agent it evaluates. The layer keeps running. Numbers keep arriving. The dashboard stays green. None of that means the evaluation harness is still telling you the truth.
Most teams deploying agents have an eval by now. That is no longer the useful question. The right question is: when did you last verify that it is still telling you the truth?