Every few months a team decides to bring their models in-house, and roughly half of them regret it. The other half wonder why they waited. The difference is rarely technical skill. It is whether the workload actually fit.
Here is the honest arithmetic, without the ideology that usually accompanies this debate.
What Self-Hosting Genuinely Wins
Local and self-hosted models win on four things: speed, privacy, cost at volume and control. Frontier APIs win on depth, context length, open-ended reasoning and the fact that somebody else is awake at 3am.
Those are real trade-offs rather than a ranking, and which side matters depends entirely on what your application does most often.
The Four Cases Where It Clearly Makes Sense
- Data cannot leave your environment. Regulatory or contractual restrictions settle the argument before cost enters it. If this is you, the decision is already made.
- Very high volume of narrow tasks. Classification, extraction and routing at large scale. One analysis reported infrastructure costs falling from roughly $3,000 to $127 a month after moving high-volume work off frontier APIs to purpose-built smaller models.
- Latency is the product. When a round trip to a provider is a meaningful share of your response budget, local inference changes what you can build.
- You need behaviour that does not change. A pinned local model behaves the same next quarter. A hosted one may not.
The Costs That Get Missed
| Cost | Why it surprises people |
|---|---|
| Idle capacity | You pay for the GPU whether or not traffic arrives |
| Ops ownership | Someone must patch, monitor and be on call |
| Model upkeep | Open models improve fast, and staying current is work |
| Evaluation | You now own quality, with no vendor to blame |
| Peak headroom | Sizing for your busiest hour means overpaying for every other hour |
Idle capacity is the one that kills projects. A per-token bill scales to zero on a quiet weekend. A GPU does not.
The Hybrid Everyone Ends Up With
The arrangement that actually survives is not either-or. Run small models locally for high-volume narrow work, and call a frontier API for planning, ambiguity and code. Route by task difficulty, with confidence-based escalation from the local model to the hosted one.
You capture most of the cost saving without inheriting the quality ceiling. This is also the architecture that lets you switch providers later without rewriting anything, which has its own value.
Run the Numbers Properly
Before committing, calculate cost per successful outcome on both paths at your real traffic, not your hoped-for traffic. Include the ops hours honestly, at a loaded rate.
If self-hosting wins only at ten times your current volume, you are not making a decision, you are making a bet on growth. Sometimes that bet is right. Make it consciously.
Conclusion
Self-host when data residency demands it, when volume is high and tasks are narrow, or when latency is the product. Use hosted frontier models for reasoning, code and ambiguity. Most teams should run the hybrid, and almost nobody should move everything in-house in one step. Start with your highest-volume, lowest-risk task, measure it properly, and expand only where the numbers hold.
Frequently Asked Questions
What hardware do we need to start?
Less than people assume for small models, and considerably more for anything approaching frontier capability. Size against your actual peak concurrency, then check what the idle hours cost you.
Are open models good enough now?
For classification, extraction and summarisation of known formats, comfortably. For hard reasoning and code, frontier models retain a clear measured lead, and mid-2026 comparisons still show it.
Can we start hosted and move later?
Yes, and that is the sensible order. Build behind an abstraction, prove the workload, then move the parts where the economics are obvious.