The trace-to-dataset loop: turning live conversations into eval cases

Here's how most teams build an eval suite: they write test cases by hand before launch, cover the happy path and a few edge cases they can think of, and ship.

Then the agent goes live. It encounters a customer who asks about the refund policy while disputing a charge, mid-way through changing their shipping address, on an account that has a pending fraud flag. The agent loops. The customer escalates. After the incident, someone adds the case manually. Six months later, the eval suite has grown to 80 cases, mostly written after things went wrong. It covers yesterday's failures. It doesn't predict tomorrow's.

The trace-to-dataset loop inverts this. Instead of chasing failures after they happen, you let production conversations continuously feed the dataset. Every week, your eval suite gets richer. Coverage grows to match what real customers actually do, not what you imagined before deployment.

Why production traces are your best eval data

Your production conversations have three properties that make them more valuable than hand-written test cases.

They're real. No one sat down and tried to construct this particular sequence of events. A customer with a disputed charge and a pending shipping change and a fraud flag actually called. The edge case exists in nature, which means it will happen again.

They're novel. They represent gaps in your current eval coverage almost by definition. If your suite already covered them, you'd have caught the failure in testing. The fact that they reached production means they're testing something you didn't test.

They're specific. A real conversation with a real customer is far more concrete than "customer asks a complicated question." The exact wording, the emotional state, the sequence of turns, the tool call that failed -- all of it is in the trace.

The question isn't whether these traces are useful. It's how to get them into your eval suite systematically rather than by hand after incidents.

What to flag

Not every production trace belongs in your eval dataset. You want the interesting ones, and interesting has a definition.

Low-scoring traces are the first category. If you run a quality metric on your production conversations -- task completion rate, intent resolution, an LLM-judge score -- the bottom decile is where failures concentrate. Pull those traces. They're the cases where your agent struggled most.

Escalation triggers are the second category. Any conversation that resulted in transfer to a human agent is a conversation the AI agent didn't resolve. That doesn't mean the AI did something wrong; some issues genuinely need humans. But it means the case was complex enough to exceed the agent's current capability, which is exactly what you want in an eval dataset.

Tool call failures are the third. A conversation where a tool returned an error, where the agent retried a call and got inconsistent results, or where a tool call produced an unexpected output is worth testing explicitly. Tool failures are often intermittent in testing but systematic in production because production data is messier than test data.

Rare paths are the fourth. Most conversations cluster around a handful of common intents. The cases in the long tail -- the 0.3% of conversations that hit an unusual combination of states -- are disproportionately likely to break. Flag any conversation that followed a path you've seen fewer than five times.

Manual flags are the fifth. Support team members who review transcripts sometimes tag conversations as notable. These are human-judgment signals that a conversation was unusual. They're worth adding to the dataset even if the automated score was fine.

Building the curation pipeline

The pipeline has three steps: evaluate, flag, and review.

Evaluate. Run a quality score on every production trace. At the scale most CX teams operate, you can't use a large frontier model as a judge for every single call -- at 8 cents a trace and 5,000 calls a day, that's $400 per day just to monitor. The good news is that smaller fine-tuned models can now run quality metrics at under 200ms and less than a tenth of a cent per trace. Galileo's Luna-2 family (3B and 8B Llama-based models released in 2026) demonstrated this concretely: 20+ quality metrics simultaneously at 97% lower cost than GPT-4o-class judges. At that cost, you can evaluate 100% of your production traffic.

Flag. Apply rules to the evaluation output to identify curation candidates. A simple starting rule set: score below 0.7, or escalated, or contained a tool error, or followed a path seen fewer than five times. More sophisticated: flag if the customer-expressed-frustration signal was high but the agent-estimated-resolution score was also high. That pattern means your agent thought it succeeded when the customer didn't, which is the hardest class of failure to catch without this signal.

Review. Add a human step before traces join the live dataset. Not every flagged trace is a good eval case. Some are flagged because of a transient infrastructure issue unrelated to agent reasoning. Some duplicate a case already in the dataset. A 30-second review per trace -- scan the transcript, read the failure reason, check for duplicates -- is enough to make the call.

Here's what this pipeline looks like in code:

import Chanl from '@chanl/sdk';
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
async function runDailyCuration() {
  // Pull yesterday's production traces
  const traces = await chanl.calls.list({
    since: new Date(Date.now() - 86_400_000),
    limit: 1000,
  });
 
  // Score each trace for quality signals
  const scored = await Promise.all(
    traces.map(async (trace) => {
      const score = await chanl.scorecards.evaluate({
        callId: trace.id,
        criteria: ['task_completion', 'intent_resolution', 'tool_reliability'],
      });
      return { trace, score };
    })
  );
 
  // Identify curation candidates
  const candidates = scored.filter(({ trace, score }) => {
    const lowQuality = score.overall < 0.7;
    const escalated = trace.outcome === 'escalated';
    const toolFailure = trace.toolErrors.length > 0;
    const rarePath = trace.pathFrequency < 0.01;
    return lowQuality || escalated || toolFailure || rarePath;
  });
 
  // Add to staging for human review before the dataset
  await chanl.datasets.addToStaging({
    datasetId: 'cx-agent-eval',
    cases: candidates.map(({ trace, score }) => ({
      traceId: trace.id,
      failureReason: score.primaryFailureMode ?? 'unknown',
      curationSignal: deriveCurationSignal(trace, score),
      suggestedExpectedOutcome: score.idealOutcome,
    })),
  });
 
  console.log(`Staged ${candidates.length} candidates from ${traces.length} traces`);
}

The staging step is where the human review happens. A team member sees the transcript, the failure reason, and a list of similar cases already in the dataset. They approve, reject, or edit the expected outcome before the case lands in the live eval set.

Closing the loop in CI

The dataset only matters if CI runs against it. Every pull request that changes a prompt, a tool, or an agent configuration should trigger an eval run against the full curated dataset. A regression on any case in the dataset blocks the merge.

This is the loop that gives the system its name:

The key property is that coverage grows automatically. The first week you might have 30 eval cases. Three months in, you have 400. And unlike a hand-built dataset, yours is weighted toward real failure modes -- not failures you imagined before you knew what the real world would throw at your agents.

Chanl's scenario testing supports running a curated eval dataset as part of CI. You define the expected outcome for each case when you approve it, and the system flags regressions when a new agent version produces a different result. The scorecards provide the quality signal that feeds curation, so the same infrastructure drives both production monitoring and eval dataset growth -- no separate tooling required.

Deduplication and dataset hygiene

Left unmanaged, the curation pipeline will add similar cases indefinitely. If your agent struggles with a particular category of question, every variant of that question will get flagged. After a month you might have 40 near-identical "customer disputes a charge" cases that all test the same thing.

Deduplication is part of the review step. Before approving a trace, the reviewer sees: "There are 7 similar cases already in the dataset." If those 7 cases have different dimensions -- different emotional states, different tool call sequences, different outcomes -- 7 cases is fine. If they're all nearly identical, one good case is enough.

The dimensions that matter for diversity in an eval dataset:

• Customer intent type (billing dispute vs. product question vs. cancellation request)
• Emotional state (neutral, frustrated, angry)
• Tool call sequence (which tools were called and in what order)
• Session length (short 2-turn vs. extended 15-turn)
• Resolution outcome (resolved, escalated, abandoned)
• Edge case type (rare path, tool failure, loop, handoff failure)

Aim for coverage across these dimensions, not volume. A 100-case dataset with good dimensional coverage catches more regressions than a 500-case dataset where 400 cases are slight variations on the same scenario.

Preventing the dataset from going stale

Every quarter, do a dataset review. Look for cases that your current agent handles perfectly in every recent eval run. These cases have served their purpose -- they caught a regression at some point in the past -- but they're no longer testing anything at the edge of your agent's capability. Removing them speeds up CI without reducing meaningful coverage.

Keep cases where:

• The case tests a failure mode you've fixed before and could regress
• The expected behavior is ambiguous enough that you want to enforce consistency
• The case probes a rare path that doesn't show up often but matters when it does

Remove cases where:

• Your current agent has passed this case for the last 20 consecutive eval runs
• The failure mode they represent no longer applies (the tool was deprecated, the policy changed)
• They're near-duplicates of stronger cases in the dataset

After your first quarterly review, you'll typically find 15 to 25% of cases can be pruned without reducing meaningful coverage. The dataset gets faster to run and the signal-to-noise ratio improves.

Building a living eval suite

The teams that catch regressions early have eval suites that reflect the real world. Not because they're better at imagining edge cases before launch, but because they've instrumented their production traffic to surface those cases automatically.

The trace-to-dataset loop isn't a sophisticated system. It's a pipeline that evaluates every production trace, flags the interesting ones, routes them through a fast human review, and adds them to the dataset that CI runs against. The sophistication is in the compounding: every week, your eval coverage gets better. Every failure caught in production becomes a test case that prevents the same failure from shipping again.

You already have the most valuable eval data you'll ever have. It's in your production traces. The loop is how you use it.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed