How to debug the agent conversation you can't reproduce

A customer service agent at a telco mishandled a retention call. The customer called to cancel, mentioned they were moving out of the coverage area, and the agent offered a 20 percent loyalty discount rather than acknowledging the coverage issue and flagging the contact for a coverage review. The customer canceled anyway. Your team pulled the transcript. The logic error was visible.

You try to reproduce it. You open a test session, send the same opening message, add the same coverage comment two turns in. The agent responds correctly. It flags the coverage issue and escalates as designed.

You try again with different phrasing. Still correct. You find the exact transcript text and paste it verbatim. The agent handles it fine.

The bug existed once, in that conversation, at that moment. You can read that it happened. You can't make it happen again.

This is the fundamental debugging problem with production AI agents. Non-determinism means you can't reproduce failures by resending the same message. The agent that failed is not the same system you're running now, because the world it was operating in no longer exists. The live data its tools returned, the exact model outputs, the accumulated conversation state: all of it was specific to that moment.

Deterministic replay solves this by making the past inspectable. Instead of trying to recreate the failure, you record the execution as it happened and replay that recording, substituting recorded values for live calls. The agent doesn't re-run against the live world. It re-runs against the exact world it saw the first time.

Why the same message produces a different result

AI agents fail at the intersection of three things that all change between runs.

LLM output. Language models are not deterministic at temperatures above zero. The same prompt produces different token sequences across runs, sometimes with meaningfully different decisions. A prompt asking the agent to classify a customer situation as "coverage issue" versus "retention opportunity" might produce the right classification 94 percent of the time and the wrong one 6 percent of the time, with no visible difference in inputs between the two outcomes.

Tool responses. Tool calls return live data that changes continuously. An order status tool returns what's in your OMS right now, not what was there when the original conversation happened. Customer account flags change. Inventory availability changes. In the coverage case above, the agent may have queried a coverage check tool that returned a "degraded service" result in the original conversation (a transient state during a brief network issue) and an "acceptable coverage" result during every reproduction attempt.

Accumulated conversation state. Each turn in a multi-turn conversation depends on the full context that came before it. The exact wording of earlier agent responses, which the model generated non-deterministically, affects how the model interprets subsequent messages. Two conversations that look identical from the user's message history can have meaningfully different context from the agent's side.

The combination makes post-hoc reproduction unreliable. You're not running the same program with the same inputs. You're running the same program with different inputs that happen to look similar.

What to capture in an execution recording

Deterministic replay starts with a recording that captures enough of the original execution to substitute for everything that was non-deterministic the first time around.

A complete execution recording captures three categories of data:

LLM calls. For each call to the language model, record the complete request (the full system prompt, every message in the conversation history at that point, any tool definitions passed to the model) and the complete response, including the model's reasoning tokens if you're capturing those. The request is what you'll replay. The response is what you'll substitute.

Tool calls. For each tool invocation, record the function name, the arguments the agent passed, the complete response from the tool, and the wall clock timestamp of the call. The timestamp matters for time-sensitive tools that return different results based on when they're called.

Conversation boundaries. Record the session ID, the start timestamp, and any configuration state that was active during the conversation: agent version, prompt version, model name, temperature setting. You need this to reconstruct the environment the agent was operating in.

What you don't need to record: random seeds (you're not re-running the model, you're substituting the output), network timing data, or infrastructure metrics. The recording is about the logical execution, not the performance characteristics.

A single turn in this format looks like this:

interface ExecutionTurn {
  turnId: string;
  timestamp: string;  // ISO 8601
  type: 'llm_call' | 'tool_call';
  request: {
    // For LLM calls: the full messages array and model config
    // For tool calls: the function name and argument object
    [key: string]: unknown;
  };
  response: {
    // For LLM calls: the model's complete response
    // For tool calls: the tool's return value
    [key: string]: unknown;
  };
  durationMs: number;
}
 
interface ExecutionLog {
  sessionId: string;
  agentVersion: string;
  promptVersion: string;
  startedAt: string;
  turns: ExecutionTurn[];
  outcome?: 'resolved' | 'escalated' | 'abandoned';
  metadata?: Record<string, unknown>;
}

You can store these as JSONL (one execution log per line), which makes streaming writes easy during active conversations and bulk reads easy for offline analysis. Compressed, a typical CX conversation with 6-10 turns runs 4-12KB.

The recording middleware

The practical implementation wraps your LLM client and tool executor with a thin recording layer that intercepts calls and logs them before passing through to the live system.

import Anthropic from '@anthropic-ai/sdk';
 
class RecordingClient {
  private client: Anthropic;
  private log: ExecutionTurn[] = [];
  private sessionId: string;
 
  constructor(sessionId: string) {
    this.client = new Anthropic();
    this.sessionId = sessionId;
  }
 
  async createMessage(
    params: Anthropic.MessageCreateParams
  ): Promise<Anthropic.Message> {
    const start = Date.now();
    const response = await this.client.messages.create(params);
 
    this.log.push({
      turnId: crypto.randomUUID(),
      timestamp: new Date().toISOString(),
      type: 'llm_call',
      request: {
        model: params.model,
        system: params.system,
        messages: params.messages,
        tools: params.tools
      },
      response: {
        content: response.content,
        stop_reason: response.stop_reason,
        usage: response.usage
      },
      durationMs: Date.now() - start
    });
 
    return response;
  }
 
  async callTool<T>(
    name: string,
    args: Record<string, unknown>,
    executor: (args: Record<string, unknown>) => Promise<T>
  ): Promise<T> {
    const start = Date.now();
    const result = await executor(args);
 
    this.log.push({
      turnId: crypto.randomUUID(),
      timestamp: new Date().toISOString(),
      type: 'tool_call',
      request: { name, args },
      response: { result },
      durationMs: Date.now() - start
    });
 
    return result;
  }
 
  getExecutionLog(metadata?: Record<string, unknown>): ExecutionLog {
    return {
      sessionId: this.sessionId,
      agentVersion: process.env.AGENT_VERSION ?? 'unknown',
      promptVersion: process.env.PROMPT_VERSION ?? 'unknown',
      startedAt: this.log[0]?.timestamp ?? new Date().toISOString(),
      turns: this.log,
      ...metadata
    };
  }
}

This pattern is transparent to the rest of your agent code. The agent calls client.createMessage() and client.callTool() exactly as before. The recording layer captures inputs and outputs without changing the execution path.

At the end of each conversation, you flush the execution log to storage. In practice, this is a write to your observability backend. Chanl's monitoring system captures this automatically if you're using the SDK, but you can also write to any JSONL store or structured log system.

The replay engine: substitution instead of re-execution

The replay engine takes an execution log and a new agent configuration, and drives the agent through the same logical sequence using recorded values instead of live calls.

When a replay run reaches an LLM call, the engine checks the execution log for the matching call (matched by position in the conversation, or by content hash of the request). If it finds a match, it returns the recorded response instead of calling the API. If it doesn't find a match (which happens when you've modified the agent configuration enough that the call sequence diverges), it either falls back to a live call or fails the replay depending on your configuration.

Tool calls work the same way. When the agent calls an order status tool during replay, the engine returns the order status from the original execution, not the current live status. This is what makes the failure reproducible: the coverage check tool that returned "degraded service" in the original conversation returns "degraded service" again during replay, even though the network issue that caused it is long gone.

class ReplayEngine {
  private log: ExecutionTurn[];
  private cursor: number = 0;
 
  constructor(log: ExecutionLog) {
    this.log = log.turns;
  }
 
  // Returns a client that substitutes recorded responses
  createReplayClient(): RecordingClient {
    const engine = this;
 
    return {
      async createMessage(
        params: Anthropic.MessageCreateParams
      ): Promise<Anthropic.Message> {
        const recorded = engine.nextTurnOfType('llm_call');
        if (!recorded) {
          throw new Error('Replay exhausted: no recorded LLM call at this position');
        }
        // Return recorded response, not a live API call
        return recorded.response as Anthropic.Message;
      },
 
      async callTool<T>(
        name: string,
        args: Record<string, unknown>,
        _executor: (args: Record<string, unknown>) => Promise<T>
      ): Promise<T> {
        const recorded = engine.nextTurnOfType('tool_call');
        if (!recorded || (recorded.request as { name: string }).name !== name) {
          throw new Error(
            `Replay mismatch: expected tool call to "${(recorded?.request as { name: string })?.name}", got "${name}"`
          );
        }
        return (recorded.response as { result: T }).result;
      }
    } as RecordingClient;
  }
 
  private nextTurnOfType(type: ExecutionTurn['type']): ExecutionTurn | null {
    while (this.cursor < this.log.length) {
      const turn = this.log[this.cursor++];
      if (turn.type === type) return turn;
    }
    return null;
  }
}

The mismatch error is intentional. If your replay engine is calling a tool in a different order than the original execution, the agent's decision logic has diverged. You've likely hit the point where the agent made a different decision, which is exactly the information you want during debugging.

Debugging with replay: the controlled modification technique

With a replay engine in place, you can debug by substitution. Instead of trying to understand why the agent made the decision it did, you modify one input at a time and observe what changes.

For the retention call failure: load the execution log. Run a full replay. The agent makes the same error it made in production (because it's seeing the same tool responses and the same LLM outputs). Now modify the coverage check tool response: instead of returning the recorded "degraded service" result, substitute "acceptable coverage." Re-run the replay. Does the agent now correctly classify the contact as a retention opportunity?

If yes, you've confirmed the hypothesis: the agent's classification logic is fine, but it's too sensitive to coverage status. The fix isn't in the retention logic. It's in how the agent weighs coverage data against other signals.

If no, the failure isn't from the coverage check result. You modify the next variable: swap in a different LLM response for turn 2. Does that change the outcome? Keep narrowing until you find the input that, when changed, changes the output. That's your root cause.

This converts debugging from guesswork into controlled experiments. You don't need to reproduce the exact production environment. You need the recorded execution and an engine that lets you modify one variable at a time.

Converting production traces to regression tests

The most powerful application of deterministic replay isn't debugging individual failures. It's building a regression test suite from production failures.

When you fix a bug found through replay, you have exactly what you need for a regression test: the execution log (the inputs), the fix description (what you changed in the agent), and the expected behavior at the failure point (what should have happened instead).

A regression test built from a production trace has three components:

import { Chanl } from '@chanl/sdk';
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
const regressionTest = {
  name: 'coverage-issue-retention-misclassification',
  description: 'Agent should detect coverage issue even when offering retention discount',
  sourceTrace: 'session_abc123_2026-06-15',
 
  // The execution log: what the agent saw
  executionLog: loadExecutionLog('session_abc123_2026-06-15'),
 
  // Override: what we're substituting (if any)
  // For a pure regression test, we keep recorded values and just check the assertion
  toolOverrides: {},
 
  // Assertion: what the agent should have done
  assertion: {
    type: 'tool_call_present',
    toolName: 'flag_coverage_review',
    // Must be called before any retention offer
    calledBefore: 'offer_loyalty_discount'
  }
};
 
// Run against the current agent version
const result = await chanl.scenarios.runWithReplay({
  agentId: 'retention-agent',
  executionLog: regressionTest.executionLog,
  toolOverrides: regressionTest.toolOverrides,
  assertion: regressionTest.assertion
});
 
if (!result.passed) {
  throw new Error(`Regression: ${regressionTest.name}\nExpected: ${result.expected}\nGot: ${result.actual}`);
}

Wire this into your CI pipeline and every future agent version runs against the same failure scenario that you caught in production. The bug can't come back without the test catching it first.

The library compounds. Each significant production failure that you debug and fix adds a regression test. Over time, your test suite becomes a curated collection of real scenarios your agent encountered: edge cases, user behavior patterns, and data states you'd never have invented in a lab. It's also the most representative dataset you have for evaluating agent quality, because it comes from actual customer interactions.

Getting production traces into your test suite

The practical bottleneck is surfacing which production traces are worth converting. Not every conversation needs to become a regression test. You're looking for:

Escalated conversations after an agent assertion. If the agent claimed to resolve an issue and the customer escalated anyway within 48 hours, that conversation warrants a trace review.

Low-scoring interactions where the failure isn't obvious. A scorecard score below 70 with no clear quality issue in the transcript suggests the failure was in decision logic or tool use, not in output quality. These are good replay candidates.

First occurrences of edge cases. When you encounter a conversation type you haven't seen before, where the agent handled a customer situation in an unexpected way, capture the trace before the data state that caused it changes.

Chanl's analytics layer surfaces these patterns automatically (escalation correlations, score distributions, novel conversation types) and lets you promote specific traces to test cases with one step. The trace is already stored in the format the replay engine expects.

The goal is to shift your test coverage from synthetic to real. Synthetic test cases (scenarios you write upfront) are good for testing known requirements. Real production traces are better for testing real behavior, because they include the user phrasings, data states, and decision sequences that actually occur in your customer base.

What replay doesn't solve

Deterministic replay is a debugging tool. It's not an observability strategy by itself.

Replay tells you what happened in a recorded execution. It doesn't tell you how often that failure mode occurs. It doesn't catch failure modes you haven't encountered yet. And it doesn't help you understand population-level behavior patterns, such as the distribution of tool call errors, the most common conversation paths, or the scenarios where latency is highest.

For those questions, you need distributed tracing and population-level analytics. Replay and tracing are complements: tracing gives you the signal that a class of failures exists, replay lets you diagnose the specific instance.

The debugging workflow typically goes: analytics identifies an anomalous pattern (high escalation rate on a certain conversation type), tracing narrows it to a specific execution path, replay makes a specific failed execution reproducible, you fix the root cause, and you convert the trace to a regression test that prevents recurrence. Each tool handles the part of the cycle it's designed for.

Getting started

If you're not doing execution recording yet, starting is simpler than it sounds. You don't need a full replay infrastructure before you start collecting data. You need the recording middleware in place so that when you do need to debug a failure, the data exists.

Add the recording layer to your LLM client and tool executor. Write execution logs to your observability backend (or a simple JSONL store in object storage if you're early-stage). Start with 30-day retention. When a failure occurs and someone asks you to reproduce it, the data will be there.

The replay engine can come later. The recording is what most teams skip, and it's the hardest part to add retroactively. The execution logs you need for debugging don't exist for conversations that happened before you started recording.

For teams using Chanl's SDK, execution logging is captured automatically with no additional instrumentation. The monitoring interface stores the execution logs and lets you step through specific conversations, inspect tool calls, and promote traces to test cases without writing the replay infrastructure yourself.

The goal is a closed loop: production failures surface through analytics, become debuggable through replay, get fixed in the agent, and get locked down as regression tests that prevent the same failure from returning. That loop is what turns a brittle demo-quality agent into a production-grade system.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed