You did not deploy anything on Thursday. Your provider did. Quality shifted, a prompt that worked for six months started behaving differently, and your team spent the afternoon proving they had changed nothing.
That is the core problem with AI features. Part of your production system updates on someone else timetable. Feature flags are the cheapest way to get the steering wheel back.
Flag Four Things, Not One
Most teams put a single on-off flag around the AI feature and stop. That gives you exactly one move in an incident, and it is the nuclear one.
- The feature itself. The kill switch. Necessary, blunt, rarely the right first response.
- The model. Switch provider or version without a deploy. This is the flag that saves your Thursday.
- The prompt version. Roll back a prompt change independently of code. Prompts change more often than code and break things just as effectively.
- The autonomy level. The most underused flag of all. Drop from acting automatically to proposing for approval, keeping the feature alive while you investigate.
That fourth one deserves emphasis. In an incident, the choice is usually framed as leave it running or turn it off. A degrade-to-suggestions mode gives you a middle option that keeps value flowing while removing the risk.
Roll Out By Blast Radius
| Stage | Audience | Gate to advance |
|---|---|---|
| Internal | Your own team | No obvious failures in a week |
| Design partners | Customers who agreed to it | Quality scores hold on real data |
| 5% of traffic | Random sample | Error and refusal rates stay flat |
| 25% | Broader sample | Cost per outcome within budget |
| 100% | Everyone | Rollback tested at least once |
That last row is not decoration. Test the rollback before you need it. A flag nobody has ever flipped in production is a hypothesis, not a control.
Compare Before You Cut Over
Shadow mode is the highest-value pattern here. Run the new model or prompt alongside the current one on live traffic, log both outputs, act on neither until you have compared them.
You get real-traffic evidence without customer risk. It costs double tokens for the sampled period, which is trivially cheaper than a rollback conducted in public.
Pin Your Model Versions
If your provider offers pinned versions, use them. Floating to the latest is convenient right up to the day the behaviour shifts under you.
Pin, then upgrade deliberately through the same flagged rollout you would use for your own code. A provider improvement is still a change to your production system, and it deserves the same process.
Conclusion
Separate flags for feature, model, prompt and autonomy level, pinned model versions, shadow mode before cutover, and a rollback you have actually tested. None of this is novel release engineering. It is the discipline you already apply to code, applied to the parts of your system that change without your involvement.
Frequently Asked Questions
Is shadow mode too expensive?
Sample it. Running 5% of traffic through both paths for a week gives you a defensible comparison for a rounding error on your bill.
How long should flags live?
Delete feature flags once fully rolled out, but keep model and prompt version flags permanently. Those are operational controls, not temporary scaffolding.
What about prompt version control?
Treat prompts as code. Version them, review changes, and tie each version to an evaluation run. A prompt edit made directly in a console is an unreviewed production deploy.