Why your eval scores don't predict real conversation quality

Your support agent scored 91% on your eval suite. You shipped it. Customer satisfaction dropped.

This happens often enough that there's a pattern to it. The teams most confused by the gap between eval performance and real-world quality almost always have the same underlying issue: they built an eval that measures whether each response is good in isolation, but customers experience entire conversations.

A response can be accurate, on-tone, and policy-compliant in isolation, and still contribute to a conversation that ends in frustration. The agent might solve turn 3 correctly while introducing a misunderstanding that derails turn 6. It might answer every question it's asked while missing that the customer's actual goal, stated in turn 1, never got addressed at all.

Single-turn evaluation doesn't catch any of that. Multi-turn evaluation does.

Why single-turn eval misses the real problem

Single-turn evaluation grades each response independently: given this input, was this output good? It's the dominant approach in most eval pipelines because it's easy to build. You collect prompts, define expected outputs or criteria, run your agent, grade the results. Each test case stands alone.

For an FAQ bot answering questions about return policies, this is probably fine. The questions are stateless. Each question could appear as the first message in any conversation.

For a support agent handling a return request that involves checking an order, verifying the reason, confirming the refund method, and sending a confirmation, single-turn eval is measuring the wrong thing. The quality of any individual response is almost irrelevant compared to whether the full sequence leads to a resolved return.

A study published in ACM Transactions on Intelligent Systems and Technology found that frontier models show measurable performance degradation in complex multi-turn conversations compared to equivalent single-turn tasks. The researchers called it "Lost in Conversation" degradation: the agent handles early turns well, but as the conversation accumulates, the original intent gets buried and quality declines.

The specific number they found, 39% performance drop in complex multi-turn conversations, understates the problem for CX specifically, because CX conversations are designed to be multi-turn. Your agent isn't being evaluated on individual questions with a conversation history incidentally attached. The conversation history IS the task.

What actually degrades across turns

Three failure modes account for most multi-turn quality problems in CX agents: intent drift, knowledge retention failures, and compounding small errors. Understanding each one tells you what to measure and where to look when your eval scores don't match your CSAT.

The first is intent drift. The customer states what they want in turn 1. Over the next several turns, the conversation branches into verification, troubleshooting, and edge cases. By turn 8, the agent is solving a sub-problem and has lost track of the original goal. The customer gets a technically correct response to the sub-problem, and the thing they actually came for is never addressed.

The second is knowledge retention failures. The customer provides information early in the conversation: their order number, what they've already tried, why they're calling. The agent uses it correctly for a few turns, then forgets. It asks for the order number again. It suggests the customer try resetting their password even though the customer said in turn 2 that they'd already done that. Each individual suggestion might score well in isolation, but repeated requests for information the customer already provided is one of the most reliable predictors of poor CSAT.

The third is compounding small errors. A minor misclassification in turn 3 sends the conversation down a path that turns out to be wrong. In a single-turn eval, that misclassification would score as a borderline case. In a real conversation, it costs four turns to correct and leaves the customer waiting while the agent works its way back to the actual issue.

Conversation Completeness: the metric that matters most

Conversation Completeness is the most important metric for CX agent evaluation that most teams aren't tracking. It measures whether the customer's original intent was resolved by the end of the conversation.

The evaluation works in two steps. First, extract the customer's stated or implied goals from the conversation transcript. A customer who opens with "I need to return a pair of shoes I ordered last week" has one explicit goal (initiate a return) and several implied ones (confirm it was received, understand the refund timeline, know what to do with the item). Second, check each goal against the conversation outcome. Was a return initiated? Did the agent confirm receipt and timeline? Or did the conversation end with unresolved questions?

This is different from asking "was each response correct?" It's asking "did this conversation accomplish what the customer came for?"

Research consistently finds Conversation Completeness to be the single metric most predictive of customer satisfaction. Comet's 2026 observability guide and Confident AI's multi-turn eval framework both identify it as the primary benchmark for customer-facing agent evaluation, ahead of per-turn accuracy, tone scores, or response length metrics.

You can implement a basic version with an LLM-as-judge setup:

import Anthropic from "@anthropic-ai/sdk";
 
const anthropic = new Anthropic();
 
interface EvalResult {
  score: number;
  resolvedGoals: string[];
  unresolvedGoals: string[];
  reasoning: string;
}
 
async function evaluateConversationCompleteness(
  conversation: Message[]
): Promise<EvalResult> {
  const transcript = conversation
    .map((m) => `${m.role.toUpperCase()}: ${m.content}`)
    .join("\n");
 
  const response = await anthropic.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 1024,
    messages: [
      {
        role: "user",
        content: `You are evaluating a customer service conversation for completeness.\n\nCONVERSATION:\n${transcript}\n\nTASK:\n1. Identify all goals the customer expressed (explicit requests + implied needs)\n2. For each goal, determine if it was resolved by the end of the conversation\n3. Return a completeness score from 0.0 to 1.0 (goals resolved / total goals)\n4. List resolved and unresolved goals\n\nRespond as JSON with keys: score, resolvedGoals (array), unresolvedGoals (array), reasoning`,
      },
    ],
  });
 
  const text =
    response.content[0].type === "text" ? response.content[0].text : "";
  return JSON.parse(text);
}

For a production eval pipeline, you'd run this across your full test set and track the distribution. A completeness score below 0.75 on a customer-facing support agent is usually a signal that your agent is solving the proximate question rather than the customer's underlying goal.

Knowledge Retention: catching the second failure mode

Knowledge Retention is harder to measure than Completeness because it requires comparing information across turns. But it's worth measuring because retention failures are both common and directly damaging to customer experience.

The test looks like this: identify facts the customer stated in early turns, then check whether the agent used those facts correctly in later turns or whether it ignored them, forgot them, or contradicted them.

async function evaluateKnowledgeRetention(
  conversation: Message[]
): Promise<{ score: number; failures: RetentionFailure[] }> {
  const transcript = conversation
    .map((m, i) => `Turn ${i + 1} (${m.role}): ${m.content}`)
    .join("\n");
 
  const response = await anthropic.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 1024,
    messages: [
      {
        role: "user",
        content: `Analyze this customer service conversation for knowledge retention failures.\n\nCONVERSATION:\n${transcript}\n\nA knowledge retention failure is when:\n- The agent asks for information the customer already provided\n- The agent suggests something the customer said they already tried\n- The agent contradicts or ignores facts stated by the customer earlier\n\nIdentify all retention failures with the turn numbers involved.\nScore retention from 0.0 to 1.0 (1.0 = no failures).\n\nReturn JSON: { score, failures: [{ description, customerTurn, agentTurn }] }`,
      },
    ],
  });
 
  const text =
    response.content[0].type === "text" ? response.content[0].text : "";
  return JSON.parse(text);
}

In practice, retention failures concentrate in a few patterns. Agents that use RAG for knowledge retrieval sometimes retrieve incorrect information that contradicts what the customer stated. Agents with aggressive context trimming lose information the customer provided early. Agents handling high-variation prompts get distracted by new information and drop earlier context.

Measuring retention failure rates before and after prompt changes or model updates is one of the most useful regression signals for CX agents. A prompt change that improves per-turn accuracy can simultaneously increase retention failures, and that tradeoff is invisible in a single-turn eval.

Building a multi-turn test set

A multi-turn test set looks different from a standard eval set. Instead of individual prompts and expected outputs, you need complete conversation transcripts with defined personas, intents, and resolution outcomes.

The best source for these is your real conversation history. Export a sample of production conversations, particularly ones that ended in escalations or negative CSAT ratings, alongside ones that resolved cleanly. These give you the real distribution of what your agent faces, not a hypothetical one.

For each conversation in your test set, define:

1. The customer's original intent (what they came to accomplish)
2. The resolution outcome (what actually happened at the end)
3. The ground-truth resolution (what should have happened)

Then when you run your agent through the test conversation, you're measuring against a defined standard, not just grading each response in isolation.

For automated test execution, AI-powered simulation lets you run full multi-turn conversations at scale without needing human testers to play the customer role. Chanl's scenarios use a separate AI to simulate customer behavior according to the persona you define, then run the conversation through your agent and grade the result:

import Chanl from "@chanl/sdk";
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
// Define a test scenario: returns customer who had a bad experience
const scenario = await chanl.scenarios.run({
  name: "frustrated-return-customer",
  customerPersona: {
    description: "Customer who ordered shoes 3 days ago, wrong size, wants return",
    emotionalState: "mildly frustrated",
    communicationStyle: "direct, brief messages",
  },
  customerGoal: "Initiate a return and confirm refund timeline",
  agentEndpoint: "https://your-agent.example.com/chat",
  maxTurns: 12,
  grading: {
    rubric: "cx-support-quality-v1",
    criteria: [
      "conversation_completeness",
      "knowledge_retention",
      "task_completion",
      "tone_appropriateness",
    ],
  },
});
 
console.log("Conversation completeness:", scenario.scores.conversation_completeness);
console.log("Knowledge retention:", scenario.scores.knowledge_retention);
console.log("Turns to resolution:", scenario.turnsToResolution);
console.log("Escalated:", scenario.escalated);

Running 50 such scenarios takes a few minutes and catches the failure modes that would take weeks to surface through production monitoring.

You can also build regression coverage with Chanl's scorecards set up to grade full conversation transcripts rather than individual turns. See the criteria-based evaluation guide for how to structure rubrics that grade conversation-level quality.

What the production picture looks like

Development eval catches structural problems before they reach production. But conversation quality can drift after launch as real customer messages diverge from your test set. You need the same conversation-level metrics on live traffic.

The signals to watch in production that single-turn monitoring misses:

Customers repeating themselves is the most reliable leading indicator of retention failures. If you parse conversation transcripts and find customers restating information from earlier in the conversation, the agent is failing to use what it was told. This usually surfaces before CSAT data because customers don't leave a survey, they just repeat themselves and wait.

Average turns to resolution is a useful efficiency metric. An agent that resolves issues in 4-5 turns is using context efficiently. One that consistently takes 9-12 turns on issues that should resolve in 4 is probably looping, asking redundant questions, or working through the wrong path. Turn counts are easy to calculate from any conversation log.

The gap between resolution rate (conversation ended with the issue resolved) and completion rate (customer explicitly confirmed satisfaction) tells you whether your agent is technically resolving issues while missing something in the customer's experience. A narrow gap is good. A wide gap often points to tone or communication style problems that don't show up in task completion metrics.

Analytics and monitoring on these conversation-level signals give you an early warning system for the kinds of problems that don't show up until CSAT data arrives three days later.

Starting with what you have

If you're running single-turn evals today, you don't need to throw them out. Single-turn eval is still useful for catching obvious regressions in per-response quality. You want both.

The migration to multi-turn eval starts with a small set of end-to-end test conversations, ideally exported from your real conversation history. 20 to 30 conversations with a mix of clean resolutions and escalations is enough to start measuring Conversation Completeness and Knowledge Retention as additional metrics alongside your existing per-turn scores.

The teams that find their eval suite most useful are almost all measuring at the conversation level. They can tell you their resolution rate, their average turns to resolution, and their retention failure rate. The teams that get surprised by production failures are measuring per-turn accuracy and wondering why it doesn't translate to customer satisfaction.

The 91% pass rate on your eval suite tells you individual responses look good. Conversation Completeness tells you whether that translates to resolved customers.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed