What Is LLM Observability? Traces, Evals, and What to Actually Measure

LLM observability is the practice of instrumenting AI applications so you can see what happened inside a request: which prompt was sent, which model answered, which tools were called, what they returned, how many tokens it consumed, what it cost, and whether the output was any good.

It exists as a separate discipline because the dominant failure mode is new. A conventional service fails loudly, with an error code or a timeout. An LLM system fails quietly, returning a fluent, plausible, wrong answer in the normal amount of time. Every infrastructure dashboard shows green.

What does traditional monitoring miss?

Latency, error rate and throughput remain necessary and stop being sufficient.

Correctness is invisible. A hallucinated answer is a 200. Nothing in the response envelope distinguishes it from a correct one.

Cost is per-request and variable. Two requests to the same endpoint can differ by orders of magnitude in cost depending on input size and output length. Averages hide this completely, and a single runaway agent loop can dominate a day's spend.

Behaviour is non-deterministic. The same input can produce different outputs. Reproducing a reported failure requires having captured the exact inputs at the time, because you cannot reliably recreate it afterwards.

The failure is often several steps back. An agent returns a wrong answer because a tool call three steps earlier returned an empty result and the model carried on regardless. The visible symptom and the actual fault are far apart.

What does LLM observability capture?

Traces and spans

The core abstraction, borrowed from distributed tracing. A trace is one complete request, start to finish. Spans are the nested operations inside it: a model call, a retrieval, a tool invocation, a sub-agent run.

For an agent this matters more than for a conventional service, because a single user request can fan out into dozens of model calls and tool invocations. Without a trace you have a black box with a bad answer coming out of it. With one you can see that step four returned nothing and step five hallucinated around the gap.

The OpenTelemetry project provides the general model, and its generative AI semantic conventions define standard attribute names for model calls specifically. Using those conventions rather than inventing your own is worth the small upfront cost, because it keeps you portable across tooling.

Prompts and completions

The actual text sent and received, including the fully rendered prompt after templating. Teams frequently discover their prompt is not what they thought it was: a variable did not interpolate, retrieved context was truncated, a system message was dropped.

This is also where the privacy question bites, and it needs deciding before you instrument rather than after. Full prompt capture in an enterprise means storing whatever users typed and whatever was retrieved on their behalf, with all the permissions and retention implications that carries.

Tokens and cost

Input tokens, output tokens, and cost per request, attributed to a user, a feature and a model. Aggregate spend tells you the bill. Per-request attribution tells you which feature caused it, which is the only version that supports a decision. This overlaps with what an LLM gateway provides, and the two are often deployed together.

Watch the distribution rather than the mean. The p99 request is where runaway loops and unbounded retrieval show up.

Tool calls

Which tools were offered, which were chosen, what arguments were passed, what came back, and whether the call errored. The tool-use documentation describes the request and response shape that should be captured, and Anthropic's guidance on building effective agents is worth reading on where agent designs tend to go wrong. In agent systems this is usually the highest-value signal, because most agent failures are tool failures the model then papered over. If you instrument one thing beyond the model call, instrument this.

Retrieval

What was retrieved, in what order, and whether it was used. A retrieval step returning plausible but wrong passages is one of the most common root causes of a bad answer, and it is invisible unless you log what came back.

Observability versus evaluation

These get conflated and they answer different questions.

You need both, and they connect: production traces are the raw material for building realistic evaluation sets. The best test cases are the real failures you captured.

How do you evaluate a non-deterministic output?

Three approaches, usually combined.

Deterministic checks. Did it return valid JSON, call the required tool, stay within a length limit, avoid a banned term. Cheap, fast, and catches more than people expect. Where a tool defines its arguments with JSON Schema, validating against that schema is close to free.

Model-graded evaluation. Another model scores the output against criteria. Scales well, and it is not free of judgement error, so the grader itself needs occasional checking against human labels. Research on multi-agent conversation frameworks covers some of the patterns for having one model assess another's work.

Human review. Expensive, slow, and the only real ground truth. The practical pattern is to sample thinly and continuously rather than in big periodic batches.

What should you actually measure?

A short list that covers most needs, ordered by how often it earns its place.

1. Tool error rate. The most common upstream cause of a bad answer in an agent system.
2. Cost per request, by feature. Segmented, with the distribution not just the mean.
3. Latency by span, so you can tell a slow model call from a slow tool.
4. Retrieval hit quality, even if measured crudely at first.
5. Task completion rate for agents, which is the outcome metric users actually care about.
6. Token consumption per request, watching the tail.
7. Output validity against whatever deterministic checks apply.
8. User feedback signals, thumbs, corrections, retries, abandonment. Blunt but honest.

A note on retries. A high retry rate is one of the most useful early signals available and one of the least instrumented. Users silently rephrasing and asking again is a failure the system reports as two successful requests.

Common mistakes

Logging only failures. Failures are 200s here. If you log on error you have logged almost nothing.

Capturing traces nobody reads. Storage is cheap and attention is not. Instrument toward specific questions rather than collecting everything and hoping.

Averaging cost. The mean is dominated by the many cheap requests and hides the few expensive ones that make up the bill.

Treating the model as the suspect. In agent systems the fault is usually a tool, a retrieval, or a prompt-assembly bug. Swapping models before reading traces is a common and expensive detour.

Deferring the privacy decision. Deciding what to capture, how long to keep it, and who can read it after you have started capturing is a much worse position than deciding first.

Where Coworker AI fits

Agents that act across real systems need this more than chat interfaces do, because a wrong action has consequences a wrong sentence does not.

Coworker AI connects to 50+ tools, maintains organizational memory across them, and runs agents against that context, with execution history for what each run did. It exposes the same context over MCP for the AI tools your team already uses. Pro is $29.99 per user per month, Max is $149.99, and Enterprise pricing is on request.

Book a demo to see it against your own stack.

Frequently asked questions

What is LLM observability?

The practice of instrumenting AI applications so you can see what happened inside a request: the rendered prompt, the model, tool calls and their results, retrieval, token usage, cost and output quality. It exists separately from conventional monitoring because LLM systems typically fail by returning confident wrong answers rather than errors.

How is it different from normal application monitoring?

Normal monitoring tracks availability, latency and error rate, all of which can look perfect while an AI system returns wrong answers. LLM observability adds the content of the interaction, per-request cost, and some measure of output quality, because none of those are visible in infrastructure metrics.

What is a trace in LLM observability?

One complete request from start to finish, made up of nested spans for each operation inside it: model calls, retrievals, tool invocations, sub-agent runs. For agents this is essential, since a single user request can produce dozens of internal steps and the fault is often several steps before the visible symptom.

What is the difference between observability and evaluation?

Observability tells you what happened on a specific request in production. Evaluation tells you whether the system performs well against criteria, usually on test sets or sampled traffic. They connect, because real production failures make the best evaluation cases.

How do you measure quality when output is non-deterministic?

Combine deterministic checks such as valid JSON or required tool calls, model-graded evaluation where another model scores against criteria, and thin continuous human review as ground truth. No single method is sufficient, and the grader model needs its own periodic checking.

What is the most useful thing to instrument first?

Tool calls, in an agent system. Most agent failures are tool failures the model then continued past, so tool arguments, results and error rates surface root causes faster than anything else. Per-request cost segmented by feature is a close second.

Related reading

• What Is an LLM Gateway?
• What Is a Context Window?
• What Is AI Agent Memory?
• What Is MCP (Model Context Protocol)?
• LLM Agent Architecture
• AI Agent Orchestration Platforms
• 25 Agent Performance Metrics You Need to Track
• Agent Workflows Explained
• Enterprise AI Agents

If answers worsen as inputs grow, you are probably looking at context rot.

What you measure here feeds context engineering.