When a model changes on Tuesday
October 2, 2024. No deployment. No code push. No config change on the customer's side. Teams calling the gpt-4o alias woke to production failures they couldn't explain.
OpenAI had silently switched the alias from gpt-4o-2024-05-13 to gpt-4o-2024-08-06. Function calls stopped firing. Responses that should have returned a structured tool invocation came back as plain text, with a stop finish reason instead of function_call.
"It often fails to call the function calls when it should," the original poster wrote in the OpenAI developer community forum.
Teams discovered the change through user complaints. Not monitoring. Not alerts. User complaints.
This LLM model migration pattern was not vendor-specific. Seventeen months later, three overlapping silent changes at Anthropic caused a month-long quality regression across Claude Code and the Agent SDK - "the aggregate effect looked like broad, inconsistent degradation." The LLM model migration in production problem has an older precedent: GPT-4's code execution success rate had already fallen from 52% to 10% within three months, with no API versioning change and no changelog.
The output distribution changed on their infrastructure. The failure happened in yours.
The instinct in every one of these incidents: go fix the prompts. That is the wrong frame.
The version number is marketing. The behavior is what you depend on.
Traditional software versioning is a compatibility contract. A 1.2.3 to 1.2.4 patch promises behavioral continuity; violating that promise is a bug. LLM versions carry no such contract.
A minor bump - the kind vendors position as improvements - can silently alter:
- tokenizer behavior
- default temperature
- system versus user role precedence
- refusal verbosity
- JSON mode strictness
- code fence emission around structured output
- calibration on uncertainty expressions
None of these changes appear in a changelog. Any of them can break a production system tuned against the old behavior.
LLM version drift is invisible in aggregate metrics. Testing across Claude model versions revealed JSON formatting failures and inconsistent output types that aggregate performance scores did not surface. A deployment scoring green on standard benchmarks was producing structurally malformed output in the specific workload categories that mattered. The system looked fine. It was not fine.
Apple ML Research's MUSCLE work names this as instance regression: correct predictions becoming incorrect after a model update, even when aggregate accuracy improves. Behavioral drift in LLMs is not random noise. Vendors optimize for population averages; your workload is not the population average.
The output distribution shift is what you are defending against. A harness that tests and gates on the distribution survives a model swap. A harness that tracks version numbers does not.
Three ways a version bump breaks production
Not all behavioral drift is the same failure. Conflating the three types produces the wrong diagnostic and the wrong fix.
Format shifts. JSON mode strictness changes between minor versions - a model that emitted valid JSON in one release may insert prose, code fences, or whitespace variations in the next. Output length distribution drifts. Code fence emission around structured output changes. These are parsing failures: your downstream code assumes a format contract the model no longer honors.
Reasoning regressions. Verbosity changes, instruction-following priority shifts, and calibration on uncertainty expressions alter the substance of responses without breaking the format. In April 2026, a system-prompt verbosity cap at Anthropic produced a measured 3% intelligence drop across Opus 4.6 and Opus 4.7 evaluations. The capability regressed; the format was intact. Standard prompt regression testing would not have caught this - you need behavioral evals that measure what the output means, not just its structure.
Tool-call behavior changes. Function calls stop firing. Responses return as plain text with the wrong finish reason. The model still receives the tool schema. It chooses not to call it. The failure surfaces in production behavior, not in any benchmark score.
Behavioral drift affects LLMs across all three modes, but the stakes become clear in the numbers. GPT-4's code execution success rate fell from 52% to 10% within three months of a silent vendor-side update - a 42-point regression on a single capability, with no API versioning change. The ICLR 2024 paper on prompt-format sensitivity documented swings of up to 76 accuracy points in LLaMA-2-13B, driven solely by formatting differences in the prompt.
A 76-point swing is not a distribution shift you manage with prompt tweaks. It is a harness problem.
The five-stage harness for LLM model migration: sensors, steering, and recalibration
The production LLM harness is not a testing checklist. It is a system with sensors that establish behavioral baselines and detect distribution shifts, a steering loop that routes traffic and gates progression, and guides that recalibrate the system after the swap completes. A model migration is a harness validation event - not a prompt-fixing sprint.
Stage 1: Pinned evals (sensors)
Before migration begins, run the candidate model against a fixed eval suite built against the current model's outputs. The goal is not to find out whether the new model is better in aggregate - it probably is, on someone's benchmark. The goal is to find out whether it breaks the behavioral contracts your system depends on.
Model pinning in production starts here: you pin the eval criteria, not just the model version. In the MIPRO v2 production case study (5.3 million interactions), the team calibrated against approximately 150 human-judged examples to generate confusion matrices - TPR 0.905 on the core correctness metric. Critically, baseline prompt transferability should be tested first: "The baseline prompt demonstrated surprising transferability across models." Automated adaptation via Amazon Bedrock and MIPROv2 via DSPy both failed to produce prompts clearly better than the original. Test transferability before investing in model-specific optimization.
Stage 2: Shadow traffic (sensors)
Shadow traffic in LLM production systems runs the candidate model in parallel on live requests. Users see only the incumbent's responses; the candidate responds but its output is scored, never served.
At 10% mirror rate on one million requests per day, you accumulate 100,000 scored pairs within 24 hours - sufficient statistical power for most rubric components. Four rubric thresholds to gate on:
- Groundedness (0.85)
- ContextAdherence (0.90)
- TaskCompletion (0.80)
- LLMFunctionCalling (0.92)
The anti-pattern: storing shadow responses without scoring them. In one documented case, six weeks of un-scored shadow traffic produced a 40% cost increase with no evaluation signal. Shadow without evaluators is storage bloat with a price tag. One practical approach is to start at 10% mirror rate; expect shadow mode to roughly double inference spend for the duration.
Stage 3: LLM canary deployment (steering loop)
Canary routing exposes a live slice of production traffic to the new model under monitoring. One practical approach: start at 1% and scale in stages - 5%, 20%, 50%, 100% - gating each increment on rubric scores from the prior stage. One infrastructure requirement teams frequently miss: consistent user assignment within sessions. Random per-request routing breaks session coherence and corrupts the evaluation signal.
Stage 4: Rollback path (steering loop)
The rollback gate LLM teams most commonly skip is the pre-commitment. You define the trigger condition and the execution path before migration begins - not after damage has accumulated.
Three verified rollback triggers from production deployments:
- Guardrail trip rate at 1.5x baseline over a 15-minute window
- Rubric regression of -0.5 points (p<0.05, Welch's t-test) over one hour
- p99 latency at 1.3x baseline sustained 10 minutes
Median rollback latency at the gateway: approximately 35 seconds.
"Gating on dashboard vibes rather than deterministic triggers" is the failure mode - the decision becomes discretionary when it should be automatic.
Stage 5: Recalibration (guides)
Once migration completes, update prompts, system context, and output parsers to the new model's distribution. This is the guides stage: absorbing the new behavioral reality into the system's configuration.
The MIPRO v2 case study result is instructive: recalibration effort was lower than expected. The baseline prompt was competitive with all adapted versions; the highest-signal change was capitalizing key instructions in the schema. The practical implication: run the eval set regression from Stage 1 against the new model post-migration. Let the signal tell you what actually changed - don't assume prompt surgery is required.
One LLM model migration, start to finish
Take a representative migration from a document-processing pipeline - a system handling contract extraction and classification at volume, migrating from one reasoning model to a newer, cheaper replacement.
Week one: Pinned evals. The team built an eval suite from approximately 150 human-judged examples, following the calibration methodology used in the MIPRO v2 production case study. Baseline TPR on the core correctness metric: 0.905. The candidate model scored 0.891 - within range on the primary metric. LLMFunctionCalling came in at 0.88, below the 0.92 threshold. That flag went to the recalibration queue.
Weeks two and three: Shadow traffic and LLM canary deployment. At 10% mirror rate, 100,000 scored pairs accumulated within 24 hours. Groundedness and ContextAdherence held above threshold. TaskCompletion dipped to 0.78 on the billing subclass - two points below the floor - then stabilized at 0.81 after three days. The team opened the canary at 1% of live traffic and scaled to 5% after 48 hours. Rubric scores held. Session assignment was fixed at the user level.
Wednesday, 10am: Full cutover. By noon, function-call accuracy on the billing extraction agent had dropped nine points. By 2pm, the support classification agent's refusal rate had increased seven points.
The rollback gate LLM teams need to pre-register was absent here. No trigger criteria had been written down before migration began. The on-call engineer watched the dashboard and made the call manually - two hours after the trigger should have fired. Rollback executed from the gateway dashboard in under a minute.
Recalibration. Post-rollback, the team registered explicit thresholds and re-staged the migration the following week. Eval set regression analysis pointed to the function-calling schema format as the divergence point. Capitalizing the key instruction fields was the highest-signal change found - no prompt surgery required. End-to-end migration: approximately six weeks total.
The lesson is not in the cutover failure. It is in the missing pre-commitment. The harness existed. The rollback path had not been registered before migration began.
The migration will happen. The question is who controls the timing.
Every major LLM provider operates a formal deprecation program with hard retirement dates. API calls to retired models return errors. The LLM model deprecation clock is not optional. The question is whether you control the timing or the vendor does.
OpenAI provides a minimum six months' notice for generally available models and three months for specialized variants including chat and codex models. Preview models can receive as little as two weeks. The model from the cold-open incident - gpt-4o-2024-05-13 - is now retired. GPT-4 variants and gpt-3.5-turbo were deprecated April 22, 2026; shutdown is October 23, 2026.
Anthropic's four-stage lifecycle - Active, Legacy, Deprecated, Retired - delivers a minimum 60-day notice before retirement. Recent retirements have run 60-115 days. Active models carry stated retirement floors: claude-opus-4-8 not before May 28, 2027; claude-sonnet-4-6 not before February 17, 2027. Anthropic has also committed to preserving model weights for at minimum the lifetime of the company.
Google communicates exact shutdown dates with advance notice; versioned IDs such as gemini-2.0-flash-001 are safer than unversioned aliases, which point silently to updated versions. Gemini 2.0 Flash shut down June 1, 2026.
This is the alias trap. Model pinning in production buys you stability - at the cost of a managed migration you now have to plan and execute. Vendor lock-in in AI is not just provider dependency; it is a deprecation calendar. Engineering against that calendar on your terms, not the vendor's, is exactly what the harness is for.
Three things this framework does not solve
A working harness absorbs model swaps with discipline. It does not eliminate residual risk. Three problems survive even when the five stages execute cleanly - and conflating them with "migration is hard" misses each one's specific cause.
No behavioral compatibility guarantee. Apple ML Research's MUSCLE work reduces negative flips - correct predictions becoming incorrect after a model update - by up to 40% when updating from Llama 1 to Llama 2. This requires deliberate design decisions made by the vendor during model training and update rollout. Application teams cannot purchase or configure behavioral drift compatibility. Behavioral drift in LLMs follows from how vendors optimize: for population-average gains, not behavioral continuity in your specific workload. Even within a vendor's minor release series, the next version can produce targeted regressions in the capabilities you depend on. The vendor's aggregate benchmark won't show it.
Eval-set decay mid-migration. An eval suite calibrated against the old model's output style becomes a stale instrument as migration proceeds. The new model may produce outputs that are behaviorally better by any human judge yet score worse against criteria tuned to the old distribution. This is the same phenomenon the eval-rot article in this series describes in testing infrastructure: when the judge model itself receives a minor version bump, mean rubric scores shift 3 to 8 points - an eval set regression driven not by the application model but by the measuring instrument. The same logic applies during migration. Jiang et al.'s "When Benchmarks Age" (EACL 2026) addresses this at the static benchmark level; production eval sets face the same decay mechanics.
Model-agnostic abstraction layers don't normalize distributions. Model-agnostic LLM architecture - LiteLLM-style routing, provider-neutral SDKs - normalizes provider APIs. It does not normalize output distributions. A model-agnostic interface ensures you can route to any provider; it does not ensure the routed model produces outputs your downstream parsers can handle. In the MIPRO v2 production migration, switching from Claude 3 Haiku to Qwen3-32B required full behavioral validation regardless of what interface layer was in place. The latency overhead and operational complexity are real costs. So is the optionality. For teams managing migrations across multiple providers, the reduction in vendor lock-in in AI infrastructure may justify the overhead. For teams managing single-vendor version bumps, it often does not.
We build harnesses that outlive the model
A team that treats LLM model migration in production as a harness validation event can absorb a model swap as scheduled maintenance. The five stages run, the output distribution shift is detected before it reaches users, the rollback path fires in seconds if needed, and the recalibration step closes the loop. The model changed. The production LLM harness did not break.
Teams that treat migration as a prompt-fixing sprint will keep discovering the swap through user complaints. That gap is not a testing gap. It is an architecture gap.
Every model you depend on today will be retired. The vendor has already decided that. The question is whether your system is built to absorb the transition when it comes - on your terms, not theirs.