What AI Observability Covers in Agentic Systems

The five telemetry layers, the OTel gaps, the MCP credential sprawl: none of that is speculative. It's already logged, measured, and in most enterprise stacks, ignored. What's missing is the discipline to wire it into something an auditor can read.

What AI observability in agentic systems means

Enterprise agents write code, move money, and touch customer records without a human checking most of it. Watching whether the server is up tells you nothing about whether the agent made a defensible decision. That gap, between infrastructure health and decision quality, is what AI observability in agentic systems is built to close. Gartner predicts over 40% of agentic AI projects get canceled by 2027, and the stated reasons are cost overruns and monitoring gaps, not model failure. Somebody built the plane. Nobody built the black box.

Traditional application monitoring answers a narrow question: is the system up? Response times, error rates, CPU load, HTTP status codes. That question made sense when software did the same thing every time it ran. Agents don't work that way. Feed the same prompt into the same agent twice and you can get two different reasoning paths, because the model is non-deterministic by design. Yesterday's green dashboard says nothing about what the agent decides to do this afternoon.

IBM's framework for this states that agents need "additional data points unique to generative AI systems, such as token usage, tool interactions and agent decision paths." None of that is visible in a CPU graph. A single user request to a single agent might trigger three LLM calls, two tool calls, and a vector database lookup, and each one is a separate place things can go sideways. One error rate metric can't tell you which step broke, or why.

The failure mode that should worry engineering leads most is quiet. Teams without real observability don't have fewer incidents than teams with it. They have the same number. They just don't see them, because nothing announces itself. Those incidents pile up into compliance violations and eroded customer trust, and by the time someone notices, there's no paper trail to produce.

Observability, here, means the ability to see what an enterprise agent did, understand why it did it, and explain that decision well enough to debug it, enforce a guardrail against it, or prove to an auditor that the guardrail held. Arthur AI frames it as a control plane: the layer that turns autonomous behavior into something measurable and auditable, so leaders can tie agent performance to business metrics, survive an audit, and scale deployment without crossing their fingers.

Three types of telemetry make this possible, all three borrowed from traditional software observability and stretched to fit a system that reasons instead of just executing. Traces follow the causal chain of a single request, from what the user asked for through every retrieval, every tool call, every model step, to the final output. Metrics track operational signals continuously: latency, cost, success rate, token usage, hallucination rate. Logs are the structured record of what happened, and they only become useful for governance once tied back to who owns the data, what policy applied, and who was accountable.

Three terms get used interchangeably and shouldn't be. Monitoring tracks performance metrics. Tracing follows the full causal chain of a request. Observability combines both with evaluation, so a team can debug, govern, and improve the system instead of just watching it happen. Traditional software observability never had to solve semantic correctness. Was the agent's answer factually sound? Did it follow policy? Did it finish the task it was given? None of that gets caught by a simple assertion the way a null pointer exception does. Without a decision trace, "why did the agent do that" has no answer, and an unanswerable question is a dead end for audit, forensics, and trust alike.

The five telemetry layers that agentic observability must capture

Five layers, each catching a category of failure the other four miss.

Reasoning chains and decision traces capture the full sequence: user intent, planner decisions, sub-task breakdown, routing logic, every intermediate step, final output. Skip this layer and root-causing a 20-step workflow turns into archaeology, since an error at step 15 can cascade in directions nobody predicted. Maxim AI's tracing architecture organizes this into sessions (a full multi-turn interaction), traces (one end-to-end request), and spans (the logical units inside that trace), with generations, retrievals, and tool calls as specialized sub-span components rather than separate tiers.

Tool call monitoring tracks which tools an agent invoked, what parameters it passed, how often those calls succeeded or failed, and how the results changed the agent's next move. This is where agent behavior actually touches enterprise systems, databases, APIs, SaaS platforms, which makes it the highest-risk layer for data exposure. Once agents start collaborating, the combined tool usage of the group can look nothing like what any single agent would do alone, so tracing has to cross agent boundaries, not just track what happens inside one.

Cost and token telemetry covers per-request token counts, cost per run, cost broken out by agent type, and how spend accumulates across teams. Token spend gets away from people fast, and without visibility at the request and team level, budgets blow past their limits before finance notices. This layer is also how a team catches a token-hungry prompt before it ships, spots a latency regression forming, or finds a ReAct loop dragging around redundant context it doesn't need. The FinOps Foundation's FOCUS specification now normalizes AI spend alongside cloud and SaaS billing, and per the State of FinOps Report, 98% of respondents manage AI spend today, up from 63% in 2025. AI cost management is now the top forward-looking concern in enterprise FinOps, which says a lot about where the money anxiety has moved.

Identity and access context records which user or service account kicked off a given agent run, which identity invoked each tool, and which data assets got touched under what authorization. Leave identity out of the trace and a governance review can't answer who authorized a given action, which breaks the audit trail exactly where it matters most. A trace showing a tool call with no identity attached can't prove policy compliance, full stop.

Audit logs with governance context turn raw events into something an auditor can actually read. A log says what happened. Governance context says what it meant, tying the event to the data asset, the owner, and the policy that should have governed it. Atlan describes this as connecting observability logging to a context graph: raw event records become interpretable governance incidents. Regulated environments need the full chain, which data got touched, which tools got called, which policies actually held, and OpenTelemetry's structured logs can support that audit trail when paired with the right retention and security controls around them.

How the OpenTelemetry standard is shaping enterprise AI telemetry, with gaps that remain

OpenTelemetry's GenAI semantic conventions give the industry a shared vocabulary for recording LLM operations. These conventions capture which model got called, input and output token counts, and, when a team opts in, the full content of prompts, completions, tool calls, and their results. OpenTelemetry graduated within the CNCF in May 2026, moving it from experimental project to stable, vendor-neutral foundation. As of version 1.41, the spec defines spans for agents, workflows, tools, and models, plus recommended metrics for latency and token usage.

Every gen_ai.* attribute in the spec still carries a Development stability badge. Attribute names can change without a major version bump, so teams instrumenting on top of these conventions should treat them as still settling, not finished.

Instrument at the AI gateway. That's the call, and it's not close. The gateway is where all model traffic and MCP traffic converge, so observability built there covers every provider and every agent behind it without touching a line of application code. That's the fastest route to coverage across an entire enterprise, and every other approach means chasing instrumentation framework by framework, forever.

The standard exists to solve a fragmentation problem already visible in the wild. Different agent frameworks handle observability differently. CrewAI has OTel-based telemetry built in, but it feeds CrewAI's own analytics rather than producing traces a user can query directly, so user-facing tracing still needs an external instrumentor layered on top. Other frameworks skip built-in telemetry entirely and require external integration from day one. Without a shared convention, telemetry from one framework simply doesn't compare to telemetry from another. The GenAI observability project inside OTel is now defining both an agent application convention and a separate agent framework convention, with frameworks including IBM's Bee Stack, AutoGen, Semantic Kernel, LangGraph, and PydanticAI in scope.

Instrument with OTel first. It's vendor-neutral, it's portable across backends, and switching monitoring tools later doesn't mean re-instrumenting the whole stack. But the spec has real gaps around semantic evaluation and hallucination detection, and that's exactly where purpose-built platforms need to fill in.

What the MCP layer adds to the observability problem, requiring its own governance layer

Anthropic released the Model Context Protocol in late 2024 as an open standard for connecting AI assistants to outside data, tools, and systems, built on JSON-RPC 2.0 as its messaging layer. By December 2025, more than 10,000 active public MCP servers were running, and the protocol had logged tens of millions of monthly SDK downloads, infrastructure at scale. That's infrastructure at scale. That's infrastructure.

The adoption curve backs it up. Datadog reported MCP tool calls up 22x since the fourth quarter of 2025. Atlassian reported a 400% jump in MCP calls in a single quarter. Salesforce reported sixfold growth in agentic use of its apps through MCP and CLI calls. Growth at that speed outruns governance every time, and this case is no exception.

Without a gateway sitting in front of MCP traffic, security teams are flying blind: no inventory of which servers are running, what credentials those servers hold, what systems they can reach, or what data actually moves through them. Credential sprawl makes this concrete. Every agent tends to carry its own API keys, OAuth tokens, and service account credentials for each tool it touches, and GitGuardian found 24,008 unique secrets exposed in MCP configuration files in 2025 alone. That's a credential dump waiting for someone to trip over it.

Call it the visibility black hole. Without logging at the MCP layer, an organization can't see which tools its agents are actually using, can't track access patterns to sensitive data, can't attribute cost to the team that generated it, and can't catch tool poisoning or a rug-pull attack before it does damage. An MCP gateway closes most of that gap at once. It tracks every tool invocation, every parameter, every result, and ties each one to an identity. It produces performance analytics and error rates broken out by tool and by team. It allocates cost to whichever team is actually burning the tokens. And it catalogs approved servers while blocking the ones nobody signed off on, making sanctioned AI a matter of official approval rather than unauthorized, unofficial use.

The Enterprise-Managed Authorization extension to MCP, which went stable on June 18, 2026, matters here because it makes enterprise identity providers the authoritative source for who gets access to which MCP server. Once that's formalized at the protocol level, observability and identity context stop being two separate problems and become one.

The identity gap that breaks most enterprise observability implementations

Most LLM observability tools stop at prompts, tokens, and latency. They'll tell a team something broke. They won't tell anyone who authorized the action that broke it, or which policy should have stopped it before it ran. In a regulated environment, that gap is disqualifying. Every tool invocation and every data access event has to trace back to a specific identity, one tied precisely to the individual agent rather than a generic service account or display name.

MCP's own spec has been catching up on this. OAuth 2.1 support landed in March 2025, with further refinement in June 2025, but implementation still varies a lot from one gateway to the next and from one framework to the next. Getting this right at the enterprise level means wiring observability into the identity providers already in place, Okta, Entra ID, whatever the organization runs, through SAML, OIDC, or OAuth 2.1, so an agent's trace carries the same identity context a human employee's activity log would.

Role-based access control and observability aren't substitutes for each other. RBAC, tied to the identity provider, decides what an agent is allowed to do in the first place. Observability shows whether that permission actually got respected in practice. Neither does the job alone. The EMA extension pushes this into the protocol itself, making the enterprise identity provider the authoritative source for who gets access to which MCP servers and ensuring every tool call carries an identity from the moment it fires.

Deploy observability without identity integration and here's what happens: the trace is fine for debugging a slow response and useless for a compliance review or an incident. Forensically, it's incomplete, and nobody can reconstruct who did what. A governance program built on incomplete traces is just paperwork.

Purpose-built AI observability platforms cover what infrastructure tooling alone cannot

Maxim AI's 2025 analysis of leading agent observability platforms lands on five names: Maxim AI, Arize AI's Phoenix, LangSmith, Langfuse, and AgentOps, evaluated across distributed tracing, multi-agent workflow support, evaluation capability, and how well teams collaborate across functions using them. On the evaluation and testing side, the field includes Patronus AI, Braintrust, Galileo, and Giskard. Governance and compliance tooling runs through Credo AI, Fiddler AI, Arthur AI, Holistic AI, and Monitaur.

Raw OTel infrastructure was never built to do a handful of things these platforms add. Semantic evaluation checks output quality, factual accuracy, whether the task actually got completed, whether the response matched what the user meant, none of which a latency metric can tell you. Hallucination detection flags outputs that aren't grounded in whatever evidence got retrieved, tracked by domain and query type so patterns show up over time. Session-level tracking treats a multi-turn conversation as one coherent unit instead of a pile of disconnected requests. Simulation and replay let a team re-run an agent interaction from any point in the chain, reproduce a bug, and test a fix, without touching live traffic.

Above all of it sits what's increasingly called the AI control plane: an enterprise-wide layer tying observability data to access control, policy enforcement, identity context, and cost attribution, across every agent, every MCP server, every AI tool the organization runs. This is the layer that turns "we have traces" into "we can actually govern this at scale," a distinct and larger claim. Judge any platform on one question: are its traces connected to a governed context graph, tool calls and data access tied back to specific assets, owners, and policies, or do they just float there, disconnected from any business or governance meaning?

The reliability payoff for getting this right isn't theoretical. Galileo's research found elite teams, the ones running full evaluation and observability, hit 2.2x better reliability than non-elite teams, and reach the highest reliability tier 70% of the time versus 32% for everyone else. But there's a gap between knowing this and doing it: 72% of AI teams say comprehensive testing drives reliability, and only 15% actually reach elite evaluation coverage. Fifty-seven points sit between belief and execution, and that gap remains one of the defining challenges in enterprise AI.

The scale of ungoverned agent deployment that makes observability urgent now

None of this is a future problem waiting to arrive. MCP servers numbered over 10,000 and SDK downloads hit 97 million a month by December 2025. Tool calls at Datadog went up 22x in a single stretch. Atlassian saw a 400% jump in one quarter. GitGuardian found tens of thousands of exposed secrets sitting in MCP configs in a single year. Every one of those numbers describes deployment that outran governance months before anyone built the guardrail.

Gartner's projection that over 40% of agentic AI projects get canceled by 2027 is a warning about the technology working fine and nobody being able to prove it, budget it, or explain it when something goes wrong. The agents aren't the risk. The absence of a record of what they did is the risk, and it's the one nobody budgeted for.