Your LLM application went down. Not down in the sense of returning 500s, down in the sense of returning confident nonsense to customers for six hours while every dashboard stayed green.
That is the observability gap in one paragraph. Traditional monitoring answers whether the service responded. It has nothing to say about whether the response was any good.
The Good News: There Is a Standard Now
OpenTelemetry GenAI semantic conventions have become the reference point for instrumenting LLM workloads. That matters more than it sounds, because it means your traces are portable. You can change observability vendors without re-instrumenting your application, which was the quiet tax on this whole category.
The conventions cover the parts that matter: prompts, completions, token usage, model identity and latency, structured so tooling can read them consistently.
What to Instrument on Day One
- The full trace, not just the call. One user request may fan out into retrieval, three model calls and two tool invocations. If you only trace the outer request, every incident investigation starts blind.
- Token counts by direction. Input and output separately. They have different costs and different causes when they spike.
- Model and version on every span. When quality shifts after a provider update, this is the field that tells you why.
- Tool call arguments and outcomes. The most common silent failure is the right tool called with the wrong arguments.
- A stable request identifier through the whole chain. Without it you cannot reconstruct what happened, only what happened somewhere.
The Metrics That Predict Trouble
| Metric | What a spike means |
|---|---|
| Tokens per request | Context is growing unchecked, cost follows |
| Retry rate | Something downstream is failing quietly |
| Tool error rate | An integration changed or arguments drifted |
| Time to first token | Provider degradation, before users complain |
| Fallback rate | Your primary path is unhealthy |
| Refusal or empty-answer rate | Prompt or retrieval regression |
Watch tokens per request most closely. It is the earliest warning for both cost blowouts and quality decline, because a bloated context degrades reasoning before it breaks the budget.
Quality Needs Its Own Pipeline
Traces tell you what happened. They cannot tell you whether the answer was correct, and no amount of instrumentation will change that.
Run a sampled evaluation on live traffic. Score a small percentage of production responses against your golden set criteria, using a model judge with a human-graded sample to keep the judge honest. Alert on the score trend rather than individual results, because single bad answers are noise and a sliding average is a signal.
Control Your Tracing Costs
Full prompt and completion capture on every request produces an enormous volume of data, and the observability bill can quietly rival the model bill.
Sample aggressively for normal traffic and capture fully on errors, on slow requests and on flagged conversations. Redact before storage rather than after, because prompts contain customer data and your trace store is now in scope for the same privacy rules as your database.
Conclusion
Instrument with OpenTelemetry GenAI conventions so your traces stay portable, trace the whole chain rather than the outer call, and treat tokens per request as your leading indicator. Add a sampled quality evaluation, because uptime and correctness are different questions. The teams that catch a silent quality regression in an hour rather than a week are not smarter. They just instrumented the middle of the chain.
Frequently Asked Questions
Do we need a specialist LLM observability tool?
Not necessarily. If you already run OpenTelemetry, you can get a long way with your existing backend plus GenAI conventions. Specialist tools earn their place for evaluation workflows and prompt versioning.
Should we log full prompts and completions?
Sample them, redact them, and set a short retention. Full capture on every request is expensive and turns your trace store into a copy of your customer data.
How do we alert on quality?
On trends, not events. A sampled score that drops several points across an hour is worth waking someone. A single low-scoring answer is Tuesday.