TL;DR
When to use a small language model comes down to one task and four axes: latency, cost, control, and deployability. The frontier model behind your narrowest, highest-volume call is probably still the one the prototype shipped with. Score that one call and you get a defensible yes or no in an afternoon. Sometimes the answer is that the big model stays.
The model you never chose
Somewhere in your stack there is a call that fires a few hundred thousand times a month. It takes a support ticket, or a search string, or one sentence a user typed, and it returns one of nine labels. It has worked since the week you shipped it. It runs on the strongest model your vendor sold at the time.
Ask your team who chose that model. You will get a shrug, or a pull request from the prototype.
Every team I ask can name the model. Almost none can name the meeting where it was chosen.
That is how the decision got made almost everywhere. The frontier model was right for the prototype. Nobody knows what good output looks like yet, and the strongest model is the fastest way to find out.
Then the feature shipped. The label set froze, the volume grew tenfold, and the model stayed.
The model behind your narrowest, highest-volume task is a default, not a decision.
The vendors are ahead of their customers here. Anthropic's model-selection guidance puts high-volume, straightforward tasks and tight latency requirements on the smaller-model side of its own lineup. The frontier tier is kept for complex reasoning.
Framed as small language model vs LLM, it sounds like a question about model classes. That framing is why it never gets answered. It is a question about one task, and four axes settle it: latency, cost, control, and deployability.
An audit costs one afternoon: one task, four numbers, and a decision you can defend at the next architecture review.
Sometimes the answer comes back no.
Deciding when to use a small language model starts with the task
A default is only worth auditing if the task can survive the audit. Small language models for narrow tasks earn their place on three properties, and a task needs all three at once:
- Constrained output. The task returns a label, a field, a score, or a short bounded string. If the output is a paragraph a human reads and judges, stop here.
- Measurable correctness. You can score a run against a held-out set with no human in the loop. A task whose quality you can only feel is a task you cannot move, because you will never know whether you broke it.
- Bounded inputs. The input domain is yours: your ticket taxonomy, your product catalogue, your document types. Not the open internet, and not whatever a user decides to type.
Miss one of the three and your task is not a candidate. The three are a qualifying round, and plenty of tasks never get past them.
Most tasks fail on the second property, and they fail quietly. Nobody built the scored set, so nobody can tell whether a change helped.
SLM vs LLM is the wrong first question anyway. The frontier model has a job on a movable task, and it comes first: it is where you define what good enough means, on real inputs, against a scored set, before anything gets fine-tuned or distilled down to size. Both of those are work you pay for once and then maintain forever.
You cannot shrink a target you have not drawn.
Axis one: latency, where a voice agent decides it for you
Take a task that clears all three properties: a voice agent that answers inbound calls and books appointments. Bounded domain, constrained output, scoreable against a transcript set. Latency decides this one, and cost never gets a vote.
Human conversation has a floor. Turn-taking gaps average 239 milliseconds in English, and the pattern holds across the ten languages measured by Stivers and colleagues in PNAS in 2009. That is a fact about people. It is what your caller's ear is calibrated to before your model has done anything at all.
By WebRTC.ventures' accounting, the teams who build these pipelines work to a round trip of roughly 800 milliseconds before the interaction feels slow. That budget covers eight stages, microphone to speaker. Under about 500 milliseconds reads as natural.
AssemblyAI puts its own layer's share at around 300 milliseconds for transcription alone. Past 500 milliseconds the delay reads as hesitation, and past a second people start repeating themselves. Treat every number here as a rule of thumb from someone who ships this, not a measured threshold, useful enough to do the arithmetic on.
So do the arithmetic on your own pipeline. Instrument every stage and take the p95, not the average: voice activity detection, transcription, retrieval or tool calls, time to first token, speech synthesis, network at both ends.
Add up everything that is not the model. Subtract that from your budget.
What is left is the entire allowance for one thing: your model's time to first token.
Now measure it. Not the vendor's published number, and not your laptop at midnight. Your p95 time to first token, at your real concurrency, in your busiest hour.
Take the p99 too if your callers are the kind who hang up. The tail is the experience people complain about.
If your allowance is smaller than that number, the axis has decided, and cost never enters the conversation.
If it fits with room to spare, latency abstains and the other three axes get their say. Most non-voice tasks land there. A voice agent rarely does.
Nothing in a prompt buys back 300 milliseconds. Streaming tokens hides some of the wait once speech starts, and it does nothing for the gap before the first sound. Caching helps on repeats and this is not a repeat.
A smaller model, served closer to the caller, is the only lever with that much travel in it. That is why a voice workload is usually the first task in a stack that should move.
Axis two: cost, and what the switch actually costs
Most tasks clear the latency bar, so most audits land on cost. It is also where the audit is most often done wrong: someone opens two price pages, sees a large multiple, and calls it a saving.
Cost is a multiplication, not a comparison. Your monthly token throughput times the gap in inference cost per token. Count input and output separately, because the frontier prices are not symmetric.
Pull the throughput from your billing export rather than an estimate. Input and output tokens per month, for that one call, at last month's volume.
The gap is real. Artificial Analysis lists GPT-4o at $2.50 per million input tokens and $10.00 per million output tokens. Together AI prices Llama 3.1 8B Instruct Lite at $0.14 per million tokens in both directions.
On an output-heavy classifier running at volume, that gap changes the invoice by an order of magnitude.
Then there is the other side of the ledger, and no price page carries it.
- An evaluation harness. It did not exist before, because a task you cannot score is a task you cannot move.
- Fine-tuning and distillation. Work that closes the quality gap, once, and again every time the label set changes.
- Serving. Capacity, autoscaling, and an on-call rota for an endpoint that used to be somebody else's problem.
None of those has a list price, which is exactly why they get left out of the business case.
So put twelve months of the token gap against them. If a year of savings does not clear the one-off build plus the ongoing operation, the frontier model is the cheaper one. The price pages do not settle it.
Price the quality gap into the same sum. A cheaper answer that is wrong more often costs you somewhere else, usually in a queue with people in it.
At low volume the multiplication returns a rounding error, and the audit should stop at this axis.
Axis three: control, where the weights and the data sit
A payback calculation assumes the task stays where it is. Control decides whether it can. Three separate questions, and any one of them flips the decision on its own.
Can the vendor change the model under you? They can. GPT-4o, the flagship half of that price comparison, is itself marked deprecated on Artificial Analysis in favour of a newer release. Your prompts were tuned against weights that retire on someone else's calendar.
Your regression suite is the only thing between that and a silent change in output quality. Weights you host retire when you retire them.
Where do the weights run? A hosted endpoint is a dependency with an availability record and a rate limit. The queue at four in the afternoon is not yours to clear.
Where does the data go? This is the one your counsel asks about first.
Take the ticket classifier. Customers put names, addresses, order histories, and sometimes health details into support tickets. Every call to a hosted endpoint moves that text across a boundary somebody drew for a reason.
Inside the EU the question runs through GDPR and the EU AI Act; outside it there may be no cross-cutting AI statute at all, and the answer comes from data protection law and a sector regulator instead.
Answer the three questions for your own task in one line each: who owns the weights, where they execute, and what class of data crosses the wire. One answer of "not ours" is the flip.
The axis does not tell you what you owe. Your own obligations do that. It tells you what a data residency constraint does to the decision.
The effect is blunt. When the text cannot leave, a hosted frontier endpoint is off the shortlist before anyone prices it. The cost arithmetic never runs.
Axis four: deployability, what runs on hardware you own
Once the data is not allowed to leave, the question is what will actually run on the hardware you have. This axis is a yes or a no about one named device.
Name the device first.
A rack in your own data centre. A box on a customer's premises. A handheld scanner on a warehouse floor. A van with no signal for an hour at a stretch.
Then check whether a model of the size you need is documented to run on it.
For a lot of hardware, it now is. Google documents its Gemma 4 edge models running fully offline on phones, a Raspberry Pi, and an NVIDIA Jetson Orin Nano, at a 128K context window. The work was done with Qualcomm, MediaTek, and Google's own Pixel silicon teams.
Microsoft's Phi-4-mini-instruct, a 3.8B model released in February 2025, ships quantized builds through llama.cpp, Ollama, LM Studio, and Jan. Quantization is what turns a size claim into something that fits. It is a build step.
Offline claims in a vendor announcement describe the good case. Confirm it on your device, at your context length, with your battery and thermal budget.
A hosted frontier endpoint scores zero here and always will. On-premise language model deployment is not a feature it can grow, and neither is edge and on-device deployment. If the workload lives on hardware you own, the axis has answered.
The narrow logic case: high volume, bounded domain
Four axes, each taken on its own so far. On one shape of task they all point the same way: bounded logic, high volume, a correctness bar you can score.
In 2024, Checkr moved its background-check classification off a direct GPT-4 integration and onto a fine-tuned Llama-3-8B model. Computerworld's Lucas Mearian reported the result that September.
The task ran 1.5 million checks a month, and monthly infrastructure cost fell from roughly $12,000 to roughly $800. Per-request latency fell from roughly 15 seconds to roughly 0.15 seconds, and overall accuracy rose to 97%. The ZenML LLMOps database recorded the same migration and the same broad figures.
Read that as four axes agreeing at once. Latency moved by two orders of magnitude on a task where nobody had called latency the problem. Cost fell because high volume multiplies a per-token gap.
Control and deployability follow the model's size. Weights small enough to fine-tune are weights you can also place.
Two years on, both the prices and the models in that story have moved. The shape of the result is what carries.
The same shape turns up at a much larger scale, with one limit worth holding onto. On The Diary of a CEO in February 2026, Uber's CEO Dara Khosrowshahi said:
"Built the entire company on small AI models that have been trained on local problems and then get stitched together."
Per-market logic, stitched together, instead of one general model asked to know every city on earth, and a per-city pricing model is not necessarily a transformer reading text.
Take the line as evidence that narrow beats general at volume, which is what it says, not as a language-model migration story, because it is not one.
The pattern under both cases is the same. Bounded logic scales on small models. Open-ended reasoning does not.
Where the frontier model stays the right answer
Bounded logic at volume is one shape of task. Most of what your team builds is not that shape, and for the rest the frontier model is the right call.
Open-ended reasoning belongs to it. So does low-volume work, where a migration costs more engineering time than a year of the saving is worth. So does any feature whose requirements are still moving, where the output schema changes every sprint and today's fine-tune is next quarter's rework.
And so does any feature whose actual product value is broad world knowledge. A small model trained on your domain does not have it, and cannot be given it cheaply.
Anthropic's own guidance draws the line in the same place from the other side. Complex reasoning, scientific and mathematical work, and tasks needing nuanced understanding start on its capability-first tier. The top of its range is reserved for agent sessions that run for hours, multistep deep research, and analysis carried through to a finished document.
A small model is not an upgrade. It is a fit, and it fits fewer tasks than it misses.
Route, measure, escalate - then audit one call this week
The end state is rarely a switch. It is a route: the narrow task on the small model, the low-confidence cases escalated to the frontier one, every call measured either way. Model routing and escalation is the shape a real migration takes, and it has numbers on it. Anthropic reports an orchestrator pattern cutting 90th-percentile cost from $33 to $12 on one workload, with average cost roughly halved.
The same documentation carries the condition that matters more than the saving. A multi-model setup has to beat the single stronger model's whole cost and accuracy curve before it earns its complexity.
So the baseline comes first, on the model you already run: accuracy on your scored set, p95 latency, cost per thousand calls. You cannot route your way out of a number you never took, and the same scored set has to grade both sides.
Take the single highest-volume AI call in your stack and score it on the four axes. If it comes out small-model-shaped, book a one-session architecture review with Athru IT to cost the switch before you build it.