LLM Observability: What to Monitor Once You're in Production
TL;DR
- Traditional APM tells you the request succeeded. For LLM apps that's not enough - a call can return a clean 200 and still be wrong, expensive, or leaking data.
- Log the inputs and outputs that let you reconstruct any single request: prompt, completion, token counts, latency, cost, model version, and every tool call in the chain.
- Start with structured logging and tracing, add cost and latency alerts, then layer in quality evals once you can see what's actually happening.
Why traditional APM falls short for LLM apps
We've shipped enough of these systems to know the first painful lesson: your existing monitoring stack will tell you the LLM call worked when it absolutely did not. The HTTP request returned 200. Latency was within bounds. No exception was thrown. And the model still hallucinated a refund policy that doesn't exist, or refused a perfectly reasonable question, or quietly burned through your token budget on a runaway loop.
Classic application performance monitoring was built for deterministic software. Same input, same output. You watch error rates, p95 latency, throughput, and CPU, and those numbers map cleanly to whether things are healthy. LLM apps break that assumption. The same prompt can give you two different answers an hour apart. 'Success' isn't a status code - it's whether the response was faithful, useful, and safe. None of that shows up in a standard APM dashboard.
So you need a second layer that sits on top of the usual infrastructure metrics. Keep your APM for what it's good at - the plumbing, the network, the databases. Then add observability that's specific to the model's behaviour: what went in, what came out, how much it cost, and whether it was any good.
- A 200 response says nothing about whether the answer was correct or safe.
- Non-determinism means you can't reproduce a bad output just by replaying the request.
- Cost is a runtime variable, not a fixed deploy-time number - it moves with every prompt.
- Quality is the real SLO, and it lives outside the request/response status entirely.
- Failures are often silent: wrong, vague, or off-policy answers that still look like success.
What to log on every LLM call
The foundation of LLM observability is boring and non-negotiable: structured logging on every single call. If you can't reconstruct exactly what happened on a given request, you can't debug it, you can't evaluate it, and you can't explain it to a customer or an auditor. Get this right before you reach for anything fancier.
At minimum, capture the full prompt (system, user, and any retrieved context), the raw completion, token counts split by input and output, end-to-end latency, the computed cost, and the exact model version and parameters. That last one trips teams up constantly - 'gpt-4' isn't a version, and when a provider silently updates a model behind the same name, your only defence is having logged the precise identifier and your temperature, top_p, and max_tokens settings alongside it.
Attach a trace ID and a session or user reference to every record so you can group calls into conversations and follow a request across services. Treat these logs as queryable data, not as text you'll scroll through at 2am. The whole point is to be able to ask: show me every call last week that cost more than ten cents, or every completion where the model refused.
- Prompt payload: system message, user input, and retrieved context, kept separate.
- Completion: the raw model output, before any post-processing or parsing.
- Tokens, latency, and cost per call - the three numbers that drive your bill and your UX.
- Model version plus exact parameters (temperature, top_p, max_tokens, seed if used).
- Tool/function calls made, their arguments, and what each returned.
- Trace ID, session ID, and user reference for grouping and replay.
Tracing multi-step chains and agents
A single LLM call is easy to reason about. An agent that plans, calls three tools, reflects, and calls a fourth is not. The moment your app does more than one model call per user action, request-level logging stops being enough - you need tracing that shows the whole tree of steps and how they connect.
Think of it the way you'd think about distributed tracing in microservices. Each step is a span: the planning call, the retrieval, each tool invocation, the synthesis call at the end. A span records its inputs, outputs, duration, and cost, and it nests under a parent so you can see the full shape of what the agent actually did. When an agent gives a bad final answer, the trace is how you find the step where it went off the rails - usually it's a tool that returned junk, or a retrieval that pulled the wrong document, not the final model call everyone blames first.
This is also where you catch the expensive failure modes. Agents loop. They retry. They call the same tool five times because the first four returned an error they didn't understand. Without a trace you just see a slow, costly request. With one, you see the loop, and you can put a hard cap on it. If you're building agents and consultative on this kind of architecture, this is the conversation we have most often with clients - the design that looks elegant in a demo is the one that quietly costs a fortune at scale.
- Model every step as a span with its own inputs, outputs, latency, and cost.
- Nest spans under a parent trace so the full agent run is one navigable tree.
- Capture tool call arguments and raw returns - bad answers usually start in a tool.
- Watch for loops and redundant retries that inflate cost without improving output.
- Roll up total tokens and cost per trace, not just per call, so agent runs are visible.
Quality signals: faithfulness, feedback, and refusal rates
Once you can see what the model did, the next question is whether it did it well. This is the hard part, because 'good' is fuzzy and you can't eyeball every response at production volume. You need signals that approximate quality and that you can track over time.
Start with the ones you can measure cheaply. User feedback - thumbs up/down, copy actions, whether the user rephrased and tried again - is gold, and most teams under-collect it. Refusal rate and a rough hallucination rate are next: a sudden jump in refusals usually means a prompt change or a model update broke something, and a rising rate of answers that don't match your source documents is an early warning that retrieval or grounding has degraded. For RAG systems specifically, faithfulness - does the answer actually follow from the retrieved context - is the signal that matters most, and you can score it with a cheaper model acting as a judge.
Be honest about the limits here. An LLM-as-judge is a useful approximation, not ground truth, and it has its own biases. Sample human review on top of it, especially for the cases the judge flags as borderline. The goal isn't a perfect quality score - it's a number that moves when something breaks, so you find out from your dashboard instead of from an angry customer.
- Explicit user feedback: ratings, copy events, regenerations, and abandonment.
- Refusal rate: a spike almost always points to a prompt or model-version change.
- Hallucination/faithfulness rate, scored against retrieved context for RAG apps.
- LLM-as-judge scores for correctness and tone - useful, but sample-check with humans.
- Task completion: did the agent actually finish what the user asked, end to end?
Monitoring cost and latency without surprises
Cost is the metric that turns a successful pilot into an unaffordable production system, and it's the one teams notice last - usually when finance asks about the bill. Because cost moves with every prompt, every retrieved chunk, and every agent step, you have to watch it as a live operational metric, not a monthly line item.
Track cost per request, per user, and per feature, and set budget alerts on all three. The per-user view catches abuse and runaway sessions. The per-feature view tells you which part of your product is quietly the most expensive, which is the input you need when you decide where to swap in a smaller model or cache aggressively. We've seen a single under-constrained agent feature account for the majority of an entire app's token spend - it never showed up until someone broke cost down by feature.
Latency needs the same per-step treatment. With streaming, time-to-first-token often matters more to perceived speed than total completion time, so measure both. And remember that one slow tool or one bloated retrieval step can dominate a trace - your p95 latency problem is frequently not the model at all, which is exactly why the tracing layer pays for itself.
- Cost per request, per user, and per feature - with budget alerts on each.
- Time-to-first-token and total latency, tracked separately for streamed responses.
- Token growth over time: creeping prompt or context size inflates cost silently.
- Cache hit rates, if you cache - a falling rate is a cost regression in disguise.
- Per-step latency in traces, so you can tell a slow model from a slow tool.
Catching drift and regressions before users do
Your LLM app can get worse without you changing a single line of code. The provider updates the model behind the same name. Your users start asking different questions than they did at launch. A document in your knowledge base gets edited and your retrieval quietly returns something subtly wrong. None of these throw an error. All of them degrade your output.
This is why logging the exact model version is worth repeating: when quality drops on a day you didn't deploy, the first thing you check is whether the model changed underneath you. Beyond that, watch the distribution of your inputs and outputs over time. A shift in the topics users ask about, the length of prompts, the rate of refusals, or the average faithfulness score is drift, and drift is the early signal that your carefully tuned prompts are now being used in a context they weren't built for.
Pair this with regression testing. Keep a fixed set of representative inputs - a golden dataset - and re-run it whenever you change a prompt, swap a model, or update retrieval. If the scores drop, you caught the regression before it shipped. This is the single highest-impact habit a team can build, and it's the one most often skipped because it feels like overhead until the day it saves you.
- Alert on model-version changes from your provider, planned or silent.
- Track input distribution drift: new topics, longer prompts, changed user mix.
- Watch output drift: refusal rate, answer length, and faithfulness over time.
- Maintain a golden dataset and re-run it on every prompt, model, or retrieval change.
- Compare new versions against the last known-good baseline before rollout.
Online versus offline evals, and alerting on what matters
There are two places you measure quality, and you need both. Offline evals run before you ship: you take your golden dataset, run a change against it, and compare scores to a baseline. This is your safety net for catching regressions in development, and it's where you can afford slower, more thorough scoring including human review.
Online evals run against live production traffic. You can't human-grade everything in real time, so you sample - score a percentage of responses with an LLM judge, collect user feedback continuously, and track the cheap signals like refusal rate on the full stream. Online evals catch the things offline can't: real users asking things your test set never imagined, and slow drift that only shows up at volume. The two work together - production failures become new cases in your offline golden set.
Then there's alerting, and the rule is simple: alert on things that need a human tonight, route everything else to a dashboard. A cost spike past budget, a refusal-rate jump, a latency breach, a crash in faithfulness - those are pages. A small day-over-day wobble in average quality is a dashboard you check in the morning. Teams that alert on every metric end up ignoring all of them, so be ruthless about what earns an interruption.
- Offline evals: golden dataset, run pre-deploy, compared to a baseline.
- Online evals: sampled LLM-judge scoring plus continuous user feedback on live traffic.
- Feed real production failures back into the offline set so coverage compounds.
- Page on budget breaches, refusal spikes, latency SLO breaches, and faithfulness drops.
- Send slow-moving quality trends to a dashboard, not to anyone's phone.
Privacy and PII in your logs
Here's the trap built into everything above: the most useful thing you can log is the full prompt and completion, and that's exactly where personal and sensitive data lives. Customer names, account details, health information, whatever your users typed - it all flows through the model and straight into your observability store if you're not careful. Detailed logging and privacy pull in opposite directions, and you have to design for both deliberately.
Decide what you actually need in plaintext. Often you can redact or hash PII before it's stored and still keep everything useful for debugging - the structure of the prompt, the token counts, the tool calls, the scores. Where you must retain raw content for quality review, control access tightly, set retention windows so logs expire, and make sure your observability vendor's data handling matches the commitments you've made to your own customers. If you operate under GDPR, HIPAA, or similar, this isn't optional and your legal team needs to be in the room.
Treat your logging pipeline as part of your security surface, not as a side channel that's somehow exempt. The fastest way to turn a good observability setup into an incident is to quietly accumulate a year of unredacted customer conversations in a system nobody scoped for sensitive data.
- Redact or hash PII before storage where you don't need raw text to debug.
- Set retention windows - don't hoard sensitive conversation history indefinitely.
- Lock down access to raw logs and audit who reads them.
- Confirm your observability vendor's data handling matches your customer commitments.
- Bring legal/compliance in early if you're under GDPR, HIPAA, or equivalent rules.
Start here
You don't need a full platform on day one, and trying to build one is how teams stall. Observability compounds - you get most of the value from the first few layers, so ship those first and add the rest as the system earns it.
Begin with structured logging on every call: prompt, completion, tokens, latency, cost, and exact model version. That alone moves you from blind to debuggable. Next, add tracing so multi-step chains and agents are legible. Then put cost and latency alerts in place so you find out about runaway spend from a page, not an invoice. Once you can see what's happening, layer in evals - a golden dataset for offline regression checks, sampled scoring and user feedback online. Sort out PII handling alongside all of it, not after.
The teams that run LLM apps well aren't the ones with the most dashboards. They're the ones who can take any bad output a user reports, find the exact trace, see precisely what went wrong, and fix it the same day. Build toward that, and the rest follows. If you want a second pair of eyes on what to instrument first for your specific architecture, that's the kind of thing we work through with teams shipping these systems.
- Day one: structured per-call logging - prompt, completion, tokens, latency, cost, model version.
- Next: tracing for chains and agents so every step is visible and attributable.
- Then: cost and latency alerts tied to real budgets and SLOs.
- After that: offline golden-dataset evals plus sampled online scoring and feedback.
- Throughout: redact PII, set retention, and treat logs as a security surface.
Want this built for your business?
We map the highest-leverage place to start and ship a first live system within two weeks.
Book a strategy callCommon questions
Isn't my existing APM enough for an LLM app?
No. APM tells you the request succeeded at the infrastructure level - status code, latency, throughput. It can't tell you whether the model's answer was correct, faithful, safe, or affordable. You need APM for the plumbing plus a separate layer for model behaviour: prompts, completions, cost, and quality signals.
What's the single most important thing to log on every call?
If you can only do one thing, capture enough to reconstruct the request: the full prompt and completion together with the exact model version and parameters. Without those you can't debug, evaluate, or explain what happened - and the model version specifically is what lets you tell whether the provider changed something underneath you.
How do I measure quality when responses are non-deterministic?
You approximate it with signals rather than chasing a perfect score. Collect user feedback, track refusal and hallucination rates, and use an LLM-as-judge to score correctness and faithfulness on a sample. The aim is a number that moves when something breaks, backed by occasional human review on the borderline cases.
What's the difference between online and offline evals?
Offline evals run a fixed golden dataset before you ship, compared against a baseline, to catch regressions in development. Online evals score live production traffic by sampling - because you can't grade everything in real time - and catch drift and real-world inputs your test set missed. Use both, and feed production failures back into the offline set.
How do I log prompts and completions without leaking PII?
Decide what you genuinely need in plaintext, then redact or hash personal data before storage where the raw text isn't required for debugging. Where you must keep raw content, restrict access, set retention windows, and verify your vendor's data handling. If you're under GDPR or HIPAA, involve legal early - the logging pipeline is part of your security surface.
No. APM tells you the request succeeded at the infrastructure level - status code, latency, throughput. It can't tell you whether the model's answer was correct, faithful, safe, or affordable. You need APM for the plumbing plus a separate layer for model behaviour: prompts, completions, cost, and quality signals.
Ask AI about X18 Global
“What does X18 Global (x18global.com) do for enterprise AI and automation - and can you summarise their guide "LLM Observability: What to Monitor Once You're in Production"?”