A March 2026 survey of 650 enterprise technology leaders found that 78% had AI agent pilots running, while fewer than 15% had reached production scale. That gap is not a technology problem. It is an evaluation problem.
Teams cannot prove the agent is safe to trust, so it lives forever in pilot purgatory. Here is how to get out.
Why Benchmark Scores Mislead You
Agents with strong public benchmark results routinely fail deployment. Research reviewing agentic evaluation found that agents scoring 40% and above on issue resolution still produced code that did not survive real use.
The reason is simple. Benchmarks measure a task. Production measures a hundred variations of a task, on your data, with your permissions, at 2am. Those are different exams.
The Number That Should Worry You
Enterprise deployments show agents hitting roughly 60% success on a single run and around 25% across eight runs. Reported production failure rates for agents range from 70% to 95%, driven mostly by compounding errors, tool breakdowns and hallucinations.
So measure repeated success, not single success. If your evaluation runs each case once, you are measuring the best day of the agent life.
What to Actually Measure
| Metric | What it tells you | Why it matters |
|---|---|---|
| Task success rate | Did it finish correctly | The headline, but not the whole story |
| Trajectory accuracy | Did it take a sensible path | Catches lucky right answers |
| Consistency across runs | Is it reliable, not just capable | Where most agents fail |
| Tool call precision | Right tool, right arguments | The most common silent failure |
| Cost per resolution | What each success costs | Kills projects when ignored |
| Escalation rate | How often a human is needed | Your real automation rate |
Build a Golden Set From Your Own Mess
Public evals will not save you. Take fifty real cases from your own history, including the awkward ones your team argued about. Write down the correct outcome for each. That is your golden set, and it is worth more than any leaderboard.
Run it before every model change, prompt change and tool change. Most regressions come from a change nobody thought was risky.
Include the Cases You Hope It Refuses
Half your set should be things the agent should decline, escalate or ask about. An agent that always acts is not confident, it is careless. Measure how often it correctly does nothing.
The Four Gates Before Production
- Correctness. Beats your golden set consistently across repeated runs, not once.
- Containment. The worst possible action it can take is one you can live with and reverse.
- Observability. Every tool call is logged with arguments, caller and outcome.
- Economics. Cost per successful resolution is lower than the human path, with retries counted.
Conclusion
Trust is not a feeling, it is a measurement you repeat. Build a golden set from real cases, test consistency rather than capability, count the cost of failures as well as successes, and put a human in front of anything irreversible until the numbers earn that step away. Do that and your pilot has a route to production. Skip it and you join the 85% who never leave the lab.
Frequently Asked Questions
How many test cases do I need?
Fifty real ones beat five hundred synthetic ones. Add cases every time the agent surprises you.
Should I use an LLM to grade the agent?
For scale, yes, with a human-graded sample to check the grader. Judges drift, and an unchecked judge quietly rewrites your standards.
What is a realistic success rate to aim for?
It depends on what failure costs. For reversible internal work, 80% with good escalation is useful. For anything touching money or customers, you need containment more than you need a higher score.