LLM Observability and Tracing: The Basics Before You Need Them

Traditional application monitoring was built around requests, response times, and error codes — a mental model that maps cleanly onto a REST API but breaks down for an LLM call. A single AI request can spawn multiple model calls, tool invocations, and retries, and the thing most worth knowing when it fails isn't just "it errored" — it's what the model was actually asked to do, and what it decided to do about it.

What observability means for an LLM app

For a traditional service, observability usually means three signals: logs, metrics, and traces. The same three apply to an AI app, but each one needs an LLM-specific layer on top. A log needs to capture the actual prompt and response, not just a status code. A metric needs to track tokens and cost alongside latency. A trace needs to follow a request through every model call and tool invocation it triggers, not just a single hop.

Traces, spans, and why the vocabulary matters

A trace is the full journey of one user-facing request, start to finish. A span is one step inside that journey — a single model call, a single tool invocation, a single retrieval query. For a simple chatbot, a trace might contain exactly one span. For an agent, a trace can contain a dozen spans: a planning call, several tool calls, a verification call, and a final response — each with its own latency, token count, and cost. Without this structure, an expensive or slow request just looks like one big number with no way to tell which step caused it.

The three things worth capturing on every span

• The actual prompt and response — not a summary or a truncated preview. When something goes wrong, the exact input is usually the fastest way to understand why.
• Token counts and cost, per span — so a slow or expensive trace can be attributed to the specific step responsible, not just the request as a whole.
• Tool calls and their results — for agentic flows, knowing which tool was called, with what arguments, and what came back is often more useful for debugging than the model's final text output.

What to look for when something goes wrong

Most production AI issues fall into one of a few shapes: a response that's wrong but not an error (the model followed instructions but gave a bad answer), a response that's technically correct but unexpectedly expensive (too many tool calls, too much context), or a request that hangs (an agent stuck in a loop). Each of these needs a different signal to catch — quality issues need the actual output visible for review, cost issues need per-span token tracking, and hangs need a step count or duration ceiling that flags a trace as abnormal.

Cost belongs in the same view as latency and errors

It's common to bolt cost tracking on separately from the rest of observability — a billing dashboard in one tab, error monitoring in another. That split makes it hard to answer the most useful question: is this trace slow, expensive, or both, and why? A trace that's slow because it made twelve tool calls is also the trace that's expensive because it made twelve tool calls — the same root cause, visible from two different angles, and easiest to diagnose when both angles are in the same view.

Observability for AI apps doesn't need to be complicated to be useful. The minimum viable version is: capture the full prompt and response per span, track tokens and cost alongside latency, and structure multi-step requests as traces instead of flattening them into a single log line. That's enough to turn "something felt off with this feature last week" into a specific trace you can actually open and read.