Somewhere in your agent workflow, a model with frontier-level reasoning is being asked to decide whether an email is a complaint or an invoice. That is a very expensive coin flip.
Model routing is the least glamorous optimisation available and usually the largest. Here is how to think about it without turning quality into a casualty.
What the Comparisons Actually Show
Mid-2026 evaluations show a consistent seven to eight point lead for frontier models across both easier and harder benchmarks. That lead is real and it does not disappear on the tasks that matter most.
But the cost side is dramatic. One analysis reported infrastructure costs dropping from around $3,000 to $127 a month for standard enterprise workloads after moving high-volume tasks off frontier APIs onto purpose-built smaller models. Small models are noticeably weaker at reasoning, coding and instruction following. They are also fast, cheap and perfectly capable of the work most agents spend most of their time doing.
Split Your Workflow by Task Type, Not by Product
| Task | Good fit | Why |
|---|---|---|
| Classification and routing | Small model | Narrow output space, easy to evaluate |
| Extraction from structured docs | Small model | Pattern matching, not reasoning |
| Summarising a known format | Small model | Low ambiguity |
| Multi-step planning | Frontier | Errors compound quickly |
| Code generation and review | Frontier | The quality gap is widest here |
| Ambiguous customer intent | Frontier | Judgement, not pattern |
Look at your traces and count. In most production agents, the majority of calls fall into the top three rows, and the majority of spend does not.
The Case for Running Your Own
Local and self-hosted small models win on latency, privacy, cost and control. Frontier APIs win on depth, context length and open-ended reasoning. That is a genuine trade rather than a ranking.
Self-hosting stops making sense the moment your volume is low. A GPU sitting idle is more expensive than a per-token bill, and somebody has to patch it. Run the numbers at your actual traffic, not your hoped-for traffic.
How to Migrate Without Breaking Things
- Pick one high-volume, low-risk step. Classification is almost always the right first move.
- Shadow it. Run the small model alongside the current one and compare outputs on live traffic without acting on them.
- Set a quality floor. Decide the acceptable accuracy drop in advance, in writing, so the decision is not made emotionally later.
- Add a confidence escalation. When the small model is unsure, hand the case up rather than guessing.
- Re-measure cost per successful outcome. Not cost per call. A cheaper model that fails more can cost more.
The Trap to Avoid
Do not standardise on one model for everything, in either direction. Teams that go all-frontier burn budget on classification. Teams that go all-small ship an agent that is confidently wrong on the ten percent of cases that carry ninety percent of the consequence.
Routing is the answer, and routing needs evaluation to stay honest. Without a golden set you will not notice quality sliding until a customer does.
Conclusion
Treat model choice as a per-step decision. Send narrow, high-volume, easily verified work to small models. Keep frontier models for planning, code and genuine ambiguity. Escalate on low confidence, and measure cost per successful outcome rather than per call. Done properly this is the rare optimisation that cuts the bill by an order of magnitude and improves latency at the same time.
Frequently Asked Questions
How small is small?
Usually a few billion parameters, optimised for speed on focused tasks. The label matters less than whether it clears your accuracy floor on your own test set.
Can I fine-tune a small model instead of routing?
Yes, and for a stable, narrow, high-volume task it often wins outright. It becomes a liability when the task definition keeps changing, because retraining is slower than editing a prompt.
What about privacy-sensitive data?
That is one of the strongest arguments for a local small model. If data cannot leave your environment, the routing decision is made for you regardless of benchmark scores.