Observability for AI Systems: Traces, Prompts, Tokens, and the SLOs That Actually Matter

Classic APM (Datadog APM, New Relic, Sentry) tells you a request took 3 seconds and returned 200. That's completely useless for an LLM app: the request may have taken 3 seconds AND cost $0.40 AND cited three chunks that don't exist AND been rated 1/5 by the user. None of that is visible without an AI-specific observability layer on top.

This article is the concrete set of signals, spans, and dashboards I put on every serious AI system in 2026. It pairs with evaluating LLMs and classical ML (the offline side), RAG that actually works (what to trace in retrieval), and MLOps on a shoestring (the deploy/rollback plumbing).

Distributed tracing (OpenTelemetry) is the backbone. Every user request creates a root span. Each downstream unit of work — LLM call, vector search, tool invocation, DB query — is a child span with typed attributes:

from opentelemetry import trace

tracer = trace.get_tracer("ai-app")

async def answer(query: str, user_id: str):
    with tracer.start_as_current_span("answer") as root:
        root.set_attribute("user.id", user_id)
        root.set_attribute("query.length", len(query))

        with tracer.start_as_current_span("retrieve") as s:
            hits = await hybrid_search(query, k=20)
            s.set_attribute("hits.count", len(hits))
            s.set_attribute("hits.top_score", hits[0].score if hits else 0.0)

        with tracer.start_as_current_span("rerank") as s:
            top = await rerank(query, hits, k=5)
            s.set_attribute("reranker.model", "cohere-rerank-3")

        with tracer.start_as_current_span("llm.chat") as s:
            resp = await llm.chat(messages=build_prompt(query, top))
            s.set_attribute("llm.model", resp.model)
            s.set_attribute("llm.tokens.prompt", resp.usage.prompt_tokens)
            s.set_attribute("llm.tokens.completion", resp.usage.completion_tokens)
            s.set_attribute("llm.cost.usd", cost_of(resp))
            s.set_attribute("llm.finish_reason", resp.choices[0].finish_reason)

        return resp

Attribute names matter. Use the OTel gen_ai semantic conventions where they exist (gen_ai.request.model, gen_ai.usage.input_tokens) — dashboards and alerting rules become portable across tools.

Without the exact prompt and response bytes, debugging a single bad answer is impossible. But raw prompt logs are also the fastest way to leak PII into every downstream log store. The rule: capture, but scrub, and gate by classification.

• Always log: request id, user id, model, temperature, seed, prompt template id + variables (not the rendered prompt), token counts, cost, finish_reason, latency.
• Log scrubbed: rendered prompt and full response, with email/phone/credit-card regex-scrubbed and named-entity redaction for names and addresses.
• Never log: raw payload from a user marked 'sensitive' (health, financial, legal) — hash and reference.
• Retain narrowly: 30 days for full traces, 1 year for aggregates. Longer retention = larger blast radius on a breach.

For sensitive systems, consider a replay id pattern: store the full prompt in an encrypted bucket keyed by request id, and log only the id in traces. Debuggers with the right role can decrypt on demand.

Classic SRE golden signals are latency, traffic, errors, saturation. For AI systems the useful reframing is:

'Traffic' and 'saturation' still matter — they're just the same as any other service. The four above are the ones you don't get for free.

Every LLM call span carries cost.usd, user.id, feature, and route. That's enough to slice cost by any dimension in your warehouse. The dashboards I ship on every project:

• Cost per active user per day — the number that decides pricing.
• Cost per feature per day — which feature is running away? Usually one specific agent.
• Cost per model — surfaces the 'we forgot to switch off Claude 4 Opus after the experiment' bug.
• Top 100 most expensive requests today — one-off runaway generations show up here first.

Move the raw trace data into your warehouse nightly and query with SQL. See modern data stack 2026 for the pipes, and BigQuery cost control that scales for the FinOps discipline.

Three quality signals in production, in order of leading vs lagging:

1. Groundedness (leading) — for RAG, does the answer's claims appear in the cited chunks? Compute with a small model on a sample of requests; alert when it drops.
2. Golden-set score (leading) — a 30–200 example golden set replayed on every deploy. Non-negotiable. Full pattern in evaluating LLMs and classical ML.
3. User feedback (lagging) — thumbs-up/down, star ratings, task completion. Trailing indicator but the only true one.

Route online-eval scores back into the trace as attributes (quality.groundedness=0.92) so a single query can slice by feature × time × score.

The nastiest AI regressions don't crash — they degrade. A model update, a prompt tweak, a data change, and suddenly answers are 5% worse in a way no unit test catches. Drift detection catches this by comparing the distribution of production outputs to a reference distribution:

• Embedding drift: embed responses, compute cosine distance from a reference set weekly; alert on 2σ moves.
• Length drift: response length p50/p95 per route; a 30% jump usually means the model started rambling.
• Refusal drift: sudden increase in 'I cannot help with that' means either a jailbreak wave OR an over-cautious model swap.
• Topic drift: cluster embeddings weekly; new clusters = new user behaviors (opportunity) or hallucinated topics (bug).

None of these tell you what broke. They tell you something broke and where to look, days before user feedback would.

Agents make the trace tree explode: one user turn can become 8 LLM calls + 12 tool calls + 3 retrievals. Instrument every tool call as a child span with input, output, latency, and error. Track two extra metrics per agent turn:

• Steps per turn — p50 and p95. A p95 of 15 means your agent is looping.
• Cost per turn — the number that decides whether the feature is viable at scale.
• Tool error rate — per tool. A single flaky tool poisons the whole loop.

Cap step count in code (see building production AI agents with n8n). Alert when steps_per_turn.p95 exceeds the cap × 0.8 — that's the leading indicator of runaway loops.

Tracing every request in a chat app at 100 QPS produces terabytes of span data per month. Two tactics keep the bill sane:

• Tail-based sampling: keep 100% of errored + slow + expensive traces, sample 5–10% of the rest.
• Attribute hygiene: never put unbounded cardinality (user ids, full URLs, prompt text) as metric labels — only as span attributes. A metric with a million label values will bankrupt any TSDB.

Prompt text belongs on spans (queryable, sampled), not on Prometheus metrics (aggregated, cardinality-bounded). Learn the difference or your first monthly Datadog bill will.

• Traces: OpenTelemetry SDK → any backend (Tempo, Datadog, Honeycomb). Vendor-neutral is worth the small setup cost.
• LLM-specific: Langfuse (open source, hostable), Helicone, Arize Phoenix. All speak OTel now.
• Metrics: Prometheus + Grafana for the boring golden signals.
• Logs: whatever your platform gives you; structured JSON with a request id is the whole game.
• Evals: Promptfoo or a small in-house harness — see the evals article.

The one non-negotiable is that all four layers (traces, metrics, logs, evals) share the same request_id. When you can pivot from a Slack alert → trace → prompt → eval score in three clicks, you've won.

Realistic SLOs I ship on production AI systems:

99.9% is usually not achievable when your critical dependency is a third-party LLM provider with its own uptime story. Set SLOs you can actually hit; burning error budget every month means the SLO is wrong.

When the pager fires:

1. Open the trace for the alerting request. Which span is red? Which is slow?
2. Check the provider's status page. 40% of AI incidents are upstream.
3. Diff the golden-set eval from this deploy vs last. Regression there = code/prompt problem.
4. Check cost dashboard for a step-change. A cost spike with normal latency = usually a prompt-injection or a loop.
5. Check drift dashboards. Slow degradation vs sudden = different remediations.
6. If unclear, roll back the last deploy. You'll debug it in the morning; users won't wait.

• Only logging errors — you need the successful traces too, or you can't compare good vs bad.
• No cost attribution — 'we spent $80k on OpenAI last month' with no breakdown = paralyzed team.
• Per-request eval on 100% of traffic — costs more than the app itself; sample.
• Prompt text in metric labels — kills the metrics store.
• Ignoring drift because 'evals pass' — evals are a fixed set; drift catches the unknown-unknowns.
• Alerting on absolute cost — noisy; alert on rate of change instead.
• One dashboard for everything — separate ops, cost, and quality dashboards, each with one owner.