Tracing AI agent failures across multi-step tool chains

Your CX agent just gave a customer wrong shipping information. The final LLM response looks confident, the customer is frustrated, and you're staring at a log that shows a perfectly normal completion. Model temperature is fine. The prompt looks right. So why is it wrong?

The answer is probably six steps back.

In a multi-step agent -- one that calls your order lookup tool, checks a fulfillment API, queries a customer profile, and then generates a response -- the final output inherits everything that happened before it. A wrong tool argument at step 3 flows forward invisibly until it surfaces as a wrong answer at step 10. Standard output monitoring catches the symptom. It doesn't show you the cause.

This is the problem that trace-level debugging solves.

Why output monitoring fails for multi-step agents

Output monitoring works fine for single-call LLMs. You send a prompt, score the response, and you're done. But once your agent starts chaining tool calls -- looking up records, calling external APIs, passing intermediate results forward -- that simple model breaks down.

Each tool call introduces a potential failure point. The tool might return stale data. The agent might pass wrong arguments. A retrieval might fetch the wrong document. Any of these can corrupt the downstream context without producing an error -- just a subtly wrong intermediate result that the LLM builds on, confidently.

By the time the final response arrives, you have no visibility into how it got there. You know the destination is wrong. You don't know where the agent turned.

The fix isn't to monitor outputs more aggressively. It's to instrument every step.

What a multi-step trace actually looks like

Think of a trace as a time-ordered graph of everything the agent did. Each node in the graph is a span. Each span represents one discrete action: an LLM call, a tool invocation, a retrieval, a decision branch.

A typical CX agent trace for a "where is my order?" query might look like this:

The user asked where their order is. The agent returned a non-answer. Look at the trace: the problem is {tracking: null} at step 4, then a fulfillment API error at step 6. The LLM did exactly what you'd expect given those inputs. The bug isn't in the model. It's in the tool response at step 4 -- specifically why tracking is null when the order is marked shipped.

Without the trace, you'd spend a long time debugging your prompt.

The anatomy of a good span

Not all traces are useful. A trace that only logs "tool called, tool returned" doesn't give you enough to debug anything. Each span needs to capture five things.

Tool name and version. Tools change. If you deployed a new version of order_lookup yesterday and errors spiked today, you need to know which version each span called.

Full input arguments. This is the field developers most often skip. The same tool called with slightly different arguments can produce wildly different behavior. Log the exact object passed in, not a summary.

Full response. Not just success or failure -- the actual payload. You need to see {tracking: null}, not just status: 200.

Latency. Tool call latency often predicts data quality. A fulfillment API that normally responds in 40ms but returned in 800ms probably hit a timeout or a degraded service.

Preceding LLM context. What did the model see before it chose this tool? This is the key to understanding why the agent made the decision it made.

Here's how to instrument this in TypeScript:

import { trace, SpanStatusCode } from "@opentelemetry/api";
 
const tracer = trace.getTracer("cx-agent", "1.0.0");
 
async function callTool(
  toolName: string,
  toolVersion: string,
  args: Record<string, unknown>,
  precedingContext: string
): Promise<unknown> {
  return tracer.startActiveSpan(`tool.${toolName}`, async (span) => {
    span.setAttributes({
      "tool.name": toolName,
      "tool.version": toolVersion,
      "tool.args": JSON.stringify(args),
      "tool.preceding_context": precedingContext.slice(0, 2000),
    });
 
    try {
      const start = Date.now();
      const result = await executeToolCall(toolName, args);
 
      span.setAttributes({
        "tool.result": JSON.stringify(result),
        "tool.latency_ms": Date.now() - start,
        "tool.success": true,
      });
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.setAttributes({
        "tool.success": false,
        "tool.error": String(err),
      });
      span.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
      throw err;
    } finally {
      span.end();
    }
  });
}

Propagate the OpenTelemetry trace context through each agent step, and you'll have a complete execution graph for every run -- ready to query, diff, and score.

Scoring individual spans, not just final outputs

Once you have span-level data, you can score each step independently. This is where agent scorecards get genuinely useful -- not just grading the final response, but grading the decisions the agent made to get there.

For each tool call span, you can evaluate three things: were the arguments correct for the task, did the agent pick the right tool for this step, and did it handle the response correctly when something went wrong?

You can run these evaluations automatically against every trace. The result is a per-step quality score attached to each span, so you can see "step 4 argument quality: poor" rather than "final answer: wrong."

import Chanl from "@chanl/sdk";
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
async function scoreTrace(traceId: string) {
  const traceData = await chanl.calls.getTrace(traceId);
 
  for (const span of traceData.spans) {
    if (span.type === "tool_call") {
      const score = await chanl.scorecards.evaluate({
        spanId: span.id,
        criteria: [
          "tool_selection_appropriate",
          "arguments_well_formed",
          "error_handling_correct",
        ],
        context: {
          userIntent: traceData.metadata.userMessage,
          toolName: span.attributes["tool.name"],
          toolArgs: span.attributes["tool.args"],
          toolResult: span.attributes["tool.result"],
        },
      });
 
      if (score.overall < 0.7) {
        console.warn(
          `Low-quality span: ${span.id} scored ${score.overall} -- ${score.details}`
        );
      }
    }
  }
}

This shifts your quality signal from "the conversation went poorly" to "the agent made a bad tool choice at minute 1:42." That's a fix you can actually ship.

Total Calls

0+12%

Avg Duration

4:23-8s

Resolution

0%+3%

Live Dashboard

Active calls23

Avg wait0:04

Satisfaction98%

Finding root causes by diffing traces

The fastest debugging technique once you have traces is diffing. Find 10 failing conversations and 10 successful ones for the same intent. Compare their traces side by side.

The root cause is almost always at the first point where the two sets diverge. Three common patterns cover most cases:

Same intent, different tool chosen. Your agent calls order_lookup on successful traces but product_search on failing ones for the same user message. The LLM is making inconsistent routing decisions. Look at what's different in the preceding context -- often it's a small variation in how the user phrased the question, and your tool descriptions aren't sharp enough to survive that variation.

Same tool, different arguments. Both sets call order_lookup, but failing traces pass order_id: undefined while successful ones pass the correct value. The agent is failing to extract the order ID. Check your entity extraction step.

Same arguments, different response. Both call with the same arguments, but failing traces get a null tracking field. The bug is in the tool itself or the data it queries.

Each pattern points to a different fix. The trace diff makes the distinction visible in minutes, not hours.

Trajectory evals take this further by scoring the full decision sequence against an expected path. Combining span-level scoring with trajectory-level grades gives you both fine-grained debugging and a summary quality signal that's actually predictive of customer outcomes.

Setting up trace-based regression detection

Once you're scoring traces, you can catch regressions automatically. After any code or prompt change, compare the trace score distribution before and after. If the distribution shifts -- more errors at a specific span type, lower argument quality scores -- you've found a regression before users report it.

import Chanl from "@chanl/sdk";
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
async function checkForRegressions(
  deployVersion: string,
  baselineVersion: string
) {
  const current = await chanl.calls.getMetrics({
    version: deployVersion,
    timeWindow: "24h",
    groupBy: "span_type",
  });
 
  const baseline = await chanl.calls.getMetrics({
    version: baselineVersion,
    timeWindow: "24h",
    groupBy: "span_type",
  });
 
  for (const spanType of Object.keys(current.bySpanType)) {
    const delta =
      current.bySpanType[spanType].errorRate -
      (baseline.bySpanType[spanType]?.errorRate ?? 0);
 
    if (delta > 0.05) {
      console.warn(
        `Regression in ${spanType}: error rate up ${(delta * 100).toFixed(1)}% vs baseline`
      );
    }
  }
}

Pair this with the online vs offline evals pattern: catch regressions offline against a golden trace set before deploy, then confirm with live trace scoring after. The two signals together give you pre-deploy safety and post-deploy confidence.

What to do when you find a root cause

Finding the root cause is only useful if it maps to a fix. The three most common trace-based findings and what to do with each:

Malformed tool arguments. The agent calls the right tool with wrong or incomplete arguments. The fix is usually in the prompt -- your system prompt isn't giving the model enough context about what each argument expects. Add examples of correct calls with the exact field structure the tool requires.

Wrong tool selection. The agent picks the wrong tool for a given intent. This usually means your tool descriptions are too similar or too vague. Sharpen each description so the distinctions are clear. Sometimes the fix is consolidating two similar tools into one with a routing parameter.

Tool response mishandling. The agent receives a null or error response and continues instead of escalating. Add explicit handling in your prompt: "If order_lookup returns tracking: null, tell the customer you'll follow up with the warehouse. Do not guess at tracking information."

Each of these is a targeted, one-line fix. You'd never land on it by staring at final outputs alone.

The shift from output monitoring to path monitoring

Most teams start with output monitoring because it's fast: run a scorer on the final response, get a quality number, done. But as your agent chains more tools across more turns, output monitoring stops predicting reliability.

A high-scoring final response can mask a fragile execution path. An agent that usually lands the right answer through lucky intermediate steps is much less reliable than one that gets there correctly. The analytics view that matters most isn't "how good was the response" -- it's "which steps in the execution path are failing, and under what conditions."

The monitoring data you want isn't conversation-level pass rates. It's span-level error rates broken down by tool name, argument pattern, and version. Once you have that, debugging changes from "something is wrong somewhere" to "step 4 fails 12% of the time when the order_id is from a guest checkout." That's a fix you can ship in an afternoon.

Path monitoring also changes how you think about what to watch in production. Conversation-level metrics still matter for business reporting. But for debugging and reliability, span-level data is what tells you whether your agent is actually doing what you think it's doing -- step by step, tool by tool, at production volume.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed