Why most AI agents in production are flying blind

At 2:47am, the on-call alert fired. The voice agent was giving customers wrong information about return windows, and the calls were escalating to human agents at four times the normal rate. The support lead pulled up the monitoring dashboard.

There were LLM call logs. Prompt in, completion out, token counts, latency. About 200 of them from the last hour, each logged in isolation, none of them connected to the conversation they belonged to.

She spent two hours trying to reconstruct what happened. Which customer triggered the first bad response? What context did the agent have? Which tool call came before the wrong answer? Without traces linking LLM calls to tool calls to customer context, the logs were an archaeological dig, not a debugging tool. By morning they had a hypothesis but no proof.

That experience is more common than most teams admit. A 2026 market analysis found that 57% of organizations now have AI agents in production. Only a third of them are satisfied with their observability setup. The gap isn't a technology problem. It's a category problem: teams instrument agents like LLM wrappers when they need to instrument them like distributed systems.

Why single-call logging fails for agents

Single-call logging captures individual LLM interactions but misses the agent execution that surrounds them. This gap is wider than it looks because agents are sequential systems where each step depends on the results of the previous one.

Consider a customer calling about a billing dispute. The agent's resolution path looks like this: identify the customer from their phone number (CRM tool call), retrieve the relevant invoice (billing API call), look up the return policy for the product category (retrieval from knowledge base), generate a response (LLM call), and either resolve the issue or escalate (decision and action). That's four distinct operations before any model response reaches the customer.

LLM logging captures the final LLM call. It misses the three tool calls that set the context for that call, the retrieval that provided the policy text, and the decision logic that chose escalation over resolution. When the agent gives the wrong answer, you see the wrong output but not the wrong input that caused it.

This isn't an edge case. A typical production CX agent makes 8 to 15 LLM calls, 6 to 10 tool calls, and 2 to 3 retrieval lookups per customer interaction. Logging each call in isolation gives you 20 to 30 disconnected data points per conversation. You'd need to manually correlate timestamps and customer IDs to reconstruct a single interaction, and you'd need to do that for hundreds of conversations before a pattern would emerge.

The discipline that fixes this is agent observability: capturing the full execution as a linked hierarchy of operations, not a flat list of calls.

Traces and spans: the building blocks

Agent observability uses the same conceptual model as distributed systems observability, adapted for the specific operations agents perform.

A trace is the complete record of one agent interaction from start to finish. Every LLM call, tool invocation, retrieval query, and memory access that happens during that interaction belongs to the same trace. The trace is the unit you investigate when something goes wrong: pull a trace, replay the interaction step by step, see exactly what the agent had access to at each moment.

A span is one discrete operation within a trace. For agents, you'll create spans for LLM calls (capturing the full prompt and completion), tool calls (capturing the function name, arguments, and result), retrieval operations (capturing the query and the documents returned), and memory accesses (capturing what was read or written). Spans have start and end timestamps, status, and parent-child relationships that reveal how one operation constrained the next.

import { trace, context, SpanStatusCode } from "@opentelemetry/api";
import { SEMATTRS_GEN_AI_SYSTEM, SEMATTRS_GEN_AI_REQUEST_MODEL } from "@opentelemetry/semantic-conventions-ai";
 
const tracer = trace.getTracer("cx-agent", "1.0.0");
 
async function handleCustomerInteraction(call: IncomingCall) {
  // Root span for the entire interaction
  return tracer.startActiveSpan("agent.interaction", async (interactionSpan) => {
    interactionSpan.setAttribute("customer.id", call.customerId);
    interactionSpan.setAttribute("call.id", call.id);
    interactionSpan.setAttribute("channel", "voice");
 
    try {
      // Span for CRM lookup
      const customer = await tracer.startActiveSpan("tool.get_customer", async (span) => {
        span.setAttribute("tool.name", "get_customer");
        span.setAttribute("tool.input.identifier", call.phoneNumber);
        const result = await crmTool.getCustomer(call.phoneNumber);
        span.setAttribute("tool.output.customer_id", result.id);
        span.setAttribute("tool.output.tier", result.tier);
        span.end();
        return result;
      });
 
      // Span for retrieval
      const policy = await tracer.startActiveSpan("retrieval.policy", async (span) => {
        const query = `return policy for ${customer.lastOrder.category}`;
        span.setAttribute("retrieval.query", query);
        const docs = await knowledgeBase.search(query, { limit: 3 });
        span.setAttribute("retrieval.doc_count", docs.length);
        span.setAttribute("retrieval.top_score", docs[0]?.score ?? 0);
        span.end();
        return docs;
      });
 
      // Span for LLM call
      const response = await tracer.startActiveSpan("llm.completion", async (span) => {
        span.setAttribute(SEMATTRS_GEN_AI_SYSTEM, "anthropic");
        span.setAttribute(SEMATTRS_GEN_AI_REQUEST_MODEL, "claude-sonnet-4-6");
        span.setAttribute("llm.prompt_tokens", 0); // set after call
        const result = await model.complete(buildPrompt(customer, policy, call.transcript));
        span.setAttribute("llm.prompt_tokens", result.usage.promptTokens);
        span.setAttribute("llm.completion_tokens", result.usage.completionTokens);
        span.end();
        return result;
      });
 
      interactionSpan.setStatus({ code: SpanStatusCode.OK });
      return response;
    } catch (err) {
      interactionSpan.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
      throw err;
    } finally {
      interactionSpan.end();
    }
  });
}

The GenAI semantic conventions from the OpenTelemetry community give you a shared vocabulary for these span types, with standard attribute names for model names, token counts, tool inputs, and retrieval scores. Using the conventions means your traces are readable by any compatible backend without custom parsing.

Once you have this instrumentation in place, every customer interaction produces a trace you can open and replay. You see the CRM lookup result that set the customer context, the retrieval score that determined which policy text was included, and the full prompt that went to the model. When the output is wrong, you can trace it back to a specific input failure.

The missing layer: linking spans to outcomes

Instrumenting operations is the first step. The second, which most teams skip, is evaluating those operations against quality criteria in production.

A trace tells you what happened. An evaluator tells you whether what happened was correct.

Online evaluation attaches scoring functions to live production traces. When a customer interaction completes, a scoring pass runs against the full trace: did the agent cite the correct policy? Did it stay within its authorization scope? Did the customer issue get resolved in one call? These scores flow back into your monitoring system, where you can track quality trends across time, alert on sudden drops, and correlate quality with specific model versions, prompt changes, or tool failures.

import { chanl } from "@chanl/sdk";
 
// Register evaluators that run on every completed interaction trace
chanl.scorecards.register({
  name: "billing-interaction-quality",
  evaluators: [
    {
      name: "policy_accuracy",
      type: "llm-as-judge",
      prompt: `Given the agent's response and the policy documents retrieved, 
               rate whether the agent cited the correct policy for this customer's 
               situation. Score 0-10. Explain your reasoning.`,
      threshold: 7,
    },
    {
      name: "resolution_success",
      type: "metric",
      metric: (trace) => trace.outcome === "resolved" ? 1 : 0,
      threshold: 0.85, // alert if resolution rate drops below 85%
    },
    {
      name: "escalation_appropriate",
      type: "llm-as-judge",
      prompt: `Did the agent escalate to a human agent appropriately? 
               Review the conversation context and the escalation decision.`,
      threshold: 8,
    },
  ],
  alertOn: "any_below_threshold",
  alertChannels: ["slack:#agent-alerts", "pagerduty:cx-oncall"],
});

This is what production monitoring looks like for agents. Not just "did the server return 200" but "did the agent do its job correctly." The Chanl scorecards feature runs these evaluators automatically against every production trace, so you get quality signals without writing custom evaluation pipelines.

The complement to online evaluation is offline evaluation: running your agent against a curated dataset before you deploy a change. If you update your system prompt, change a retrieval parameter, or swap a tool implementation, your offline eval suite catches regressions before they reach customers. Online evaluation catches the edge cases your eval dataset didn't anticipate. Each fills the gap the other misses.

What your observability stack should surface

Once you have traces flowing and evaluators running, you'll have more data than you've ever had about your agent's behavior. The signal that matters depends on which failure mode you're watching for.

Context bloat shows up as rising prompt token counts across conversations. When your mean tokens per turn crosses a threshold, it usually means your retrieval is returning too much, your conversation history isn't being compressed, or a tool result is dumping more data than the agent needs. Catching this early prevents the latency spikes and accuracy drops that come from overfull context windows. We covered the mechanics of this in the context engineering piece.

Tool errors show up as elevated failure rates on specific tool spans. A CRM lookup that starts returning null for 3% of calls might mean a schema change in the upstream API. A retrieval operation with dropping scores might mean your knowledge base needs a reindex. These are backend failures that express as agent quality problems, and they're invisible without span-level tool monitoring.

Resolution drift shows up as declining first-contact resolution rates over time, even when the model hasn't changed. This is usually a knowledge base problem: the world has changed but your retrieval content hasn't caught up. Your evaluators notice it before your customers file enough complaints to trigger an alert.

Latency variance at p95 is almost always a tool call that occasionally takes 3x longer than median. The LLM call looks fine at p50. But the p95 user is waiting on a CRM that sometimes takes 800ms to respond. Without span-level latency breakdown, this is invisible. With it, you see exactly which operation is adding variance and can add timeouts or caching appropriately.

Building your observability stack

Here's the practical sequence for teams starting from zero.

Start with instrumentation. Add OpenTelemetry spans around every LLM call, tool call, and retrieval. Use the GenAI semantic conventions for attribute names. Don't try to build custom logging before you have structured traces: the structured format is what makes the data queryable.

Pick a trace backend that understands agent-specific data. Langfuse, Datadog LLM Observability, and Arize Phoenix all have agent trace support in 2026 with multi-step trace visualization, span-level filtering, and session grouping. If you're already running Datadog for infrastructure, adding LLM Observability is the path of least resistance. If you want purpose-built agent tooling, Langfuse is open-source and self-hostable.

Add online evaluators for your highest-risk scenarios first. For a billing agent, that's policy accuracy. For a scheduling agent, that's booking confirmation correctness. For any agent with escalation logic, that's escalation appropriateness. Start with two or three evaluators that correspond to the outcomes your team actually cares about, not an exhaustive list of everything that could go wrong.

Build an offline eval suite from production failures. Every time a customer complaint reveals an agent failure, add that case to your evaluation dataset. Over six months, you'll have a dataset that reflects the actual distribution of hard cases your agent faces, not idealized test scenarios. We cover the mechanics of building this dataset in the trajectory evaluation article.

Set alerts on the metrics that predict customer impact, not just the ones that are easy to measure. Latency at p95, tool error rates, resolution rates, and quality score trends are leading indicators. Customer complaint rates and escalation volume are lagging indicators that tell you something already went wrong. You want the leading ones.

The monitoring feature in Chanl connects traces, quality scores, and business metrics in a single view, so you can see a drop in resolution rate and immediately drill into the traces that explain it. That correlation is what turns a "something is wrong" alert into a "here's the specific change that caused it" diagnosis.

The cost of flying blind

57% of organizations have agents in production. The teams in that 57% that can't see what their agents are doing will find out through customer complaints, escalation spikes, and post-incident reviews that take two days to yield a hypothesis. The teams that built proper observability find out through an alert at 2:47am that tells them exactly which tool call failed, which conversations it affected, and what the quality score drop looks like.

The instrumentation is a few hundred lines of boilerplate. The evaluators are two or three scoring functions. The alert configuration takes an afternoon. The payoff is that when something goes wrong in production, and it will, you see it in traces rather than support tickets.

Total Calls

0+12%

Avg Duration

4:23-8s

Resolution

0%+3%

Live Dashboard

Active calls23

Avg wait0:04

Satisfaction98%

El briefing de Signal

Un email por semana. Cómo los equipos líderes de CS, ingresos e IA están convirtiendo conversaciones en decisiones. Benchmarks, playbooks y lo que funciona en producción.

Subscribe

500+ líderes de CS e ingresos suscritos