AI Observability Is More Than Token Counts

Most AI observability dashboards are billing dashboards with a latency chart attached. They show requests, tokens, response time and model errors. Those numbers are useful for operating an API. They are almost useless when a user says, “The assistant gave me the wrong answer,” or worse, when an agent updates the wrong record without anybody noticing.

The unit of observation in an enterprise AI system is not the model call. It is the decision or workflow that the model call participated in.

Trace the whole decision

An answer is shaped before the prompt reaches the model. Identity determines which sources can be retrieved. Search configuration determines what evidence is available. Policy may select a model, remove content or prohibit a tool. Application code may validate, transform or discard the output. A downstream system decides whether an action succeeds.

A useful trace connects those stages:

• the user, service identity and relevant permission context;
• the task and application version;
• identifiers and versions for retrieved evidence;
• model, prompt, tool and policy versions;
• structured decisions, validation results and tool calls;
• the final user-visible or business outcome.

That does not mean logging every prompt and document forever. In many organisations, doing so creates a second, poorly governed repository of sensitive data. Store identifiers, hashes, classifications, decision metadata and short-lived diagnostic content according to the risk of the workflow. The team should be able to reconstruct what the system knew and did without retaining an indiscriminate transcript of everything users supplied.

Build failure categories that lead somewhere

“Bad answer” is not an operational category. It could mean that the authoritative document was missing, retrieval ranked an obsolete source first, the model invented a claim, a validator accepted malformed output, or the interface presented a draft as a completed action.

I use failure categories that point to an owner and a possible intervention:

Failure	Likely owner	Useful response
Evidence absent or inaccessible	Content or data owner	Fix publication, permissions or source coverage
Relevant evidence ranked poorly	Search or retrieval team	Change indexing, metadata or ranking
Claim unsupported by evidence	AI product team	Tighten task design, evaluation or refusal behaviour
Tool request invalid	Application team	Improve schema, validation or error handling
Action prohibited	Policy owner	Confirm the control or improve the user route
Downstream operation failed	Service owner	Retry safely, recover or escalate

This taxonomy matters because model upgrades are frequently proposed for failures that have nothing to do with the model. Without a trace, every problem looks probabilistic and every fix becomes another prompt change.

Observe outcomes, not just responses

For an assistant, a thumbs-up is weak evidence. People approve plausible text and ignore useful text for reasons unrelated to quality. For an agent, a technically successful tool call may still produce the wrong business result.

Connect traces to meaningful signals: whether a case was reopened, whether a recommendation was overridden, whether a generated document required substantial correction, whether an action was reversed, and whether the workflow actually completed. Review outcome quality alongside cost and latency. A cheaper system that creates more manual repair is not cheaper.

Observability is part of governance

Governance cannot rely on what a system was designed to do. It needs evidence of what the system is doing now, for whom, using which data and under which controls. That evidence should support incident investigation, model and prompt changes, access reviews and periodic evaluation without exposing more user data than the task requires.

When a consequential output is challenged, the team should be able to explain the evidence, versions, controls and actions involved, identify who was affected and decide what must change. If the investigation ends at a token count and a green latency graph, the AI system is not observable.