A quality control plane for your production agent fleet

You have five agents in production. Billing, returns, account management, upsell, and tech support. Each one was tested before launch. Each one passed.

Three weeks after the billing agent's last prompt update, a product manager pulls the weekly metrics and notices something off. Resolution rate is down 25%. Not a cliff, not a crash. Just a quiet, steady slide from 81% down to 61% over about sixteen days. The logs show nothing. No errors. No timeouts. No tool failures. The agent has been answering every call, completing every conversation, and silently getting worse the whole time.

Nobody noticed because nobody was watching the right thing. The dashboards showed call volume and handle time. The logs showed HTTP 200s. What they didn't show was that the agent had started misunderstanding the difference between "billing dispute" and "billing question" after the prompt was adjusted, and was routing dispute-intent conversations into a resolution path that couldn't actually fix the problem.

This is the quality drift problem. And it's the default outcome when you run agents in production without a quality control plane.

What a quality control plane actually is

A quality control plane is a three-component automated system: a scoring pipeline that assigns quality dimensions to every completed conversation, a policy engine that enforces thresholds and blocks deployments when metrics regress, and a feedback loop that routes failures back into the improvement process. Together they turn quality from a periodic manual review into a continuous, measurable property of your fleet.

The analogy to Kubernetes is useful here. A Kubernetes control plane doesn't just run your pods. It continuously observes their state, compares it to your desired state, and takes corrective action when they diverge. If a pod crashes, the control plane replaces it. If a node goes unhealthy, the control plane reschedules. The infrastructure enforces your intent, continuously, without human intervention for each individual event.

A quality control plane does the same thing for agent behavior. Your "desired state" is expressed as quality SLOs. The scoring pipeline is your observation layer. The policy engine is your reconciliation loop. The feedback system is your self-healing mechanism.

Without this system, you're operating open-loop. You deploy, hope, and find out from customers when something breaks.

Score every conversation automatically

The foundation of the quality control plane is scoring every conversation, not sampling. Sampling is where most teams start and where the silent drift problem lives. If you score 10% of conversations, a 25% regression in the unscored 90% goes undetected for days.

The practical approach is LLM-as-a-judge: after each conversation ends, run the transcript through a scoring model with a fixed rubric. You get structured scores back, per dimension, that you can store and query over time.

Five dimensions cover most CX agent quality requirements:

Here's a scoring pipeline that runs asynchronously after each conversation completes:

import OpenAI from 'openai';
 
interface ConversationScore {
  conversationId: string;
  agentId: string;
  scores: {
    goalCompletion: number;
    accuracy: number;
    tone: number;
    toolUse: number;
    escalation: number | null;
  };
  composite: number;
  flags: string[];
  scoredAt: Date;
}
 
const SCORING_PROMPT = `
You are a quality evaluator for AI customer service agents. Score the following
conversation transcript on each dimension from 1 (very poor) to 5 (excellent).
 
Dimensions:
- goalCompletion: Did the agent resolve the customer's stated issue?
- accuracy: Was the information provided factually correct?
- tone: Was the agent's tone appropriate for the situation?
- toolUse: Were tools used correctly and at appropriate moments?
- escalation: If there was an escalation, was it handled correctly? null if no escalation.
 
Return JSON only. Include a "flags" array of short strings (e.g. "incorrect_policy_cited",
"unnecessary_escalation") for any notable issues. An empty array is fine.
`;
 
export async function scoringPipeline(
  conversationId: string,
  agentId: string,
  transcript: string
): Promise<ConversationScore> {
  const client = new OpenAI();
 
  const response = await client.chat.completions.create({
    model: 'gpt-4o',
    response_format: { type: 'json_object' },
    messages: [
      { role: 'system', content: SCORING_PROMPT },
      {
        role: 'user',
        content: `Transcript:\n\n${transcript}`,
      },
    ],
    temperature: 0,
  });
 
  const raw = JSON.parse(response.choices[0].message.content ?? '{}');
 
  const scores = {
    goalCompletion: raw.goalCompletion ?? 0,
    accuracy: raw.accuracy ?? 0,
    tone: raw.tone ?? 0,
    toolUse: raw.toolUse ?? 0,
    escalation: raw.escalation ?? null,
  };
 
  // Composite: weighted average (goal completion counts most)
  const scoredDimensions = [
    scores.goalCompletion * 0.35,
    scores.accuracy * 0.30,
    scores.tone * 0.20,
    scores.toolUse * 0.15,
  ];
  const composite = scoredDimensions.reduce((a, b) => a + b, 0);
 
  return {
    conversationId,
    agentId,
    scores,
    composite,
    flags: raw.flags ?? [],
    scoredAt: new Date(),
  };
}

Run this in a background worker, not in the request path. A webhook from your orchestration platform fires after each call ends, the worker picks it up and runs the scorer, and the result goes into your database. The customer never waits for this.

If you're using Chanl's scoring infrastructure directly, it looks like this instead:

const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
const scores = await chanl.scorecards.evaluate({
  transcriptId: conversationId,
  rubric: 'cx-standard',
});

Either approach works. The critical thing is that scoring happens on every conversation and the results are stored in a queryable, time-series format. See Chanl scorecards for how the rubric layer works.

Define your quality SLOs

Once you're scoring every conversation, you need to define what "good enough" means numerically. This is your SLO layer. Without it, you have data but no trigger, scores but no alerts, trends but no gates.

The SLO structure mirrors what you'd use for infrastructure. Each metric has a target (what you're aiming for), a warning threshold (where you want an alert), and a hard floor (where you stop new deployments). The key is tracking both the median (p50) and the tail (p99), because a high average can hide a terrible tail.

Tracking the right quality metrics before you hit production is easier than retrofitting SLOs after you've seen drift. But here's a starting table for a billing support agent:

Notice there's no single composite score that makes final decisions. A composite score is useful for monitoring trends, but it hides dimension-specific regressions. An agent can maintain a 3.8 composite while accuracy drops from 4.5 to 3.0, if tone simultaneously improves from 3.0 to 4.5. The composite looks stable. The accuracy regression is a serious problem.

Here's the SLO evaluation function that takes a batch of recent scores and returns whether each SLO is passing:

interface SLOConfig {
  metric: keyof ConversationScore['scores'];
  percentile: 50 | 99;
  target: number;
  warning: number;
  hardFloor: number;
}
 
interface SLOResult {
  metric: string;
  percentile: number;
  value: number;
  status: 'passing' | 'warning' | 'failing';
}
 
function percentile(values: number[], p: number): number {
  const sorted = [...values].sort((a, b) => a - b);
  const idx = Math.ceil((p / 100) * sorted.length) - 1;
  return sorted[Math.max(0, idx)];
}
 
export function evaluateSLOs(
  recentScores: ConversationScore[],
  sloConfigs: SLOConfig[]
): SLOResult[] {
  return sloConfigs.map((slo) => {
    const values = recentScores
      .map((s) => s.scores[slo.metric])
      .filter((v): v is number => v !== null && v !== undefined);
 
    if (values.length === 0) {
      return { metric: slo.metric, percentile: slo.percentile, value: 0, status: 'failing' };
    }
 
    const value = percentile(values, slo.percentile);
 
    let status: SLOResult['status'];
    if (value >= slo.warning) {
      status = 'passing';
    } else if (value >= slo.hardFloor) {
      status = 'warning';
    } else {
      status = 'failing';
    }
 
    return { metric: slo.metric, percentile: slo.percentile, value, status };
  });
}

Run this on a rolling window of the last 500 conversations, or the last 24 hours, whichever is larger. This gives you enough signal to distinguish noise from real trends. See Chanl analytics for how to query these windows without building the infrastructure yourself.

Build quality gates for deployment

Quality SLOs tell you when your production fleet is degrading. Quality gates stop you from making it worse. A gate is a check that runs before a new agent version goes live, asserting that the incoming version doesn't regress your SLOs compared to the current one.

The pattern that works well is a canary deployment with a quality gate. Send 5% of traffic to the new version, score those conversations for two to four hours, and only promote the full rollout if the canary's quality metrics are statistically comparable to the baseline.

interface DeploymentGateConfig {
  baselineAgentId: string;
  candidateAgentId: string;
  canaryDurationMs: number;
  minConversations: number;
  sloConfigs: SLOConfig[];
  maxRegressionPct: number; // e.g. 0.05 = allow up to 5% regression on any SLO
}
 
interface GateDecision {
  approved: boolean;
  reason: string;
  baselineResults: SLOResult[];
  candidateResults: SLOResult[];
  regressions: string[];
}
 
export async function deploymentQualityGate(
  config: DeploymentGateConfig,
  fetchScores: (agentId: string, windowMs: number) => Promise<ConversationScore[]>
): Promise<GateDecision> {
  const [baselineScores, candidateScores] = await Promise.all([
    fetchScores(config.baselineAgentId, config.canaryDurationMs),
    fetchScores(config.candidateAgentId, config.canaryDurationMs),
  ]);
 
  if (candidateScores.length < config.minConversations) {
    return {
      approved: false,
      reason: `Canary has only ${candidateScores.length} conversations. Need ${config.minConversations} before gate evaluation.`,
      baselineResults: [],
      candidateResults: [],
      regressions: [],
    };
  }
 
  const baselineResults = evaluateSLOs(baselineScores, config.sloConfigs);
  const candidateResults = evaluateSLOs(candidateScores, config.sloConfigs);
 
  const regressions: string[] = [];
 
  for (const candidate of candidateResults) {
    const baseline = baselineResults.find(
      (b) => b.metric === candidate.metric && b.percentile === candidate.percentile
    );
    if (!baseline) continue;
 
    const regressionPct = (baseline.value - candidate.value) / baseline.value;
 
    if (regressionPct > config.maxRegressionPct) {
      regressions.push(
        `${candidate.metric} p${candidate.percentile}: ${baseline.value.toFixed(2)} to ${candidate.value.toFixed(2)} (${(regressionPct * 100).toFixed(1)}% regression)`
      );
    }
  }
 
  if (regressions.length > 0) {
    return {
      approved: false,
      reason: `Quality gate failed: ${regressions.length} SLO regression(s) detected.`,
      baselineResults,
      candidateResults,
      regressions,
    };
  }
 
  return {
    approved: true,
    reason: 'All SLOs within acceptable range. Promoting canary to full rollout.',
    baselineResults,
    candidateResults,
    regressions: [],
  };
}

Wire this into your deployment pipeline. Before the final rollout step fires, call deploymentQualityGate(). If it returns approved: false, halt the promotion and page the team. The canary traffic cap means that even a failed deployment only affects 5% of users while the gate runs.

Deploy Gate

Pre-deploy quality checks

Score > 80%

92%

Latency < 500ms

234ms

Error Rate < 2%

3.1%

Deploy Blocked

Alert on drift, not just failure

Quality gates protect against deployment regressions. But drift, the gradual degradation that happened to the billing agent at the start of this article, doesn't come from a deployment. It comes from the world changing around a static agent. Tool APIs start returning slightly different data. The distribution of real-world questions shifts. A third-party service gets slower. None of these events trigger a deployment, so none of them trigger your gate.

You need a different signal for drift: sliding window detection. Instead of comparing a candidate to a baseline at deployment time, you continuously compare recent performance to your established baseline and alert when the gap becomes statistically significant.

Detecting agent drift before it reaches customers is harder than catching deployment regressions because there's no clear "before and after" event. The signal you're watching is a trend, not a step change.

The key insight is that a single bad conversation doesn't indicate drift. Drift is when the rolling mean of your scores descends consistently below the baseline. Standard deviation thresholds of 1.5x to 2x work well in practice: alert when the window mean drops more than 1.5 standard deviations below the established baseline.

interface DriftConfig {
  agentId: string;
  windowSize: number;       // number of conversations in the rolling window
  baselineWindowSize: number; // number of conversations to establish baseline
  stdDevThreshold: number;  // e.g. 1.5 = alert at 1.5 SDs below baseline
  metric: keyof ConversationScore['scores'] | 'composite';
}
 
interface DriftResult {
  agentId: string;
  metric: string;
  baselineMean: number;
  baselineStdDev: number;
  windowMean: number;
  zScore: number;
  isDrifting: boolean;
  severity: 'none' | 'warning' | 'critical';
}
 
function mean(values: number[]): number {
  return values.reduce((a, b) => a + b, 0) / values.length;
}
 
function stdDev(values: number[], mu: number): number {
  const variance = values.reduce((acc, v) => acc + Math.pow(v - mu, 2), 0) / values.length;
  return Math.sqrt(variance);
}
 
export function driftDetector(
  recentScores: ConversationScore[],
  baselineScores: ConversationScore[],
  config: DriftConfig
): DriftResult {
  const extract = (scores: ConversationScore[]): number[] =>
    config.metric === 'composite'
      ? scores.map((s) => s.composite)
      : scores
          .map((s) => s.scores[config.metric as keyof ConversationScore['scores']])
          .filter((v): v is number => v !== null && v !== undefined);
 
  const baselineValues = extract(baselineScores.slice(-config.baselineWindowSize));
  const windowValues = extract(recentScores.slice(-config.windowSize));
 
  const baselineMean = mean(baselineValues);
  const baselineStdDev = stdDev(baselineValues, baselineMean);
  const windowMean = mean(windowValues);
 
  // Z-score: how many SDs below baseline is the current window?
  const zScore = baselineStdDev > 0
    ? (baselineMean - windowMean) / baselineStdDev
    : 0;
 
  let severity: DriftResult['severity'] = 'none';
  if (zScore >= config.stdDevThreshold * 1.5) {
    severity = 'critical';
  } else if (zScore >= config.stdDevThreshold) {
    severity = 'warning';
  }
 
  return {
    agentId: config.agentId,
    metric: config.metric,
    baselineMean,
    baselineStdDev,
    windowMean,
    zScore,
    isDrifting: severity !== 'none',
    severity,
  };
}

Run this on a schedule, every 15 minutes for high-volume agents, every hour for lower-volume ones. If isDrifting is true, fire a Slack alert with the z-score and the window mean so the team can see how far off baseline things have gone.

If you're using Chanl's monitoring features, drift alerts are built in. You configure the window size and threshold, and the platform handles the detection and routing.

Close the feedback loop

Scoring every conversation and alerting on drift tells you when quality is degrading. The feedback loop is what you do about it. Without a feedback loop, your quality control plane is a monitoring system. With one, it's a self-improving system.

The feedback loop has two outputs. First, low-quality conversations go into a human review queue where patterns can be identified. Second, the most common failure patterns become new test scenarios, so the next version of the agent gets tested against the failure modes that actually happened in production.

That second output is where the compounding returns come from. Every production failure that makes it back into your test suite reduces the probability of that class of failure reaching production again. The agent drift problem doesn't go away, but your ability to catch it before deployment improves continuously.

interface FailedConversation {
  conversationId: string;
  agentId: string;
  score: ConversationScore;
  transcript: string;
  failureReasons: string[];
}
 
interface FeedbackRoutingConfig {
  scoreThreshold: number;       // Conversations below this go to review queue
  flagThreshold: number;        // Conversations with >= this many flags go to review
  highPriorityFlags: string[];  // Flags that always trigger immediate review
}
 
interface RoutingDecision {
  routeToReview: boolean;
  priority: 'immediate' | 'standard' | 'none';
  suggestedScenarioTitle?: string;
  reason: string;
}
 
export function feedbackRouter(
  conversation: FailedConversation,
  config: FeedbackRoutingConfig
): RoutingDecision {
  const { score } = conversation;
  const hasHighPriorityFlag = conversation.failureReasons.some((f) =>
    config.highPriorityFlags.includes(f)
  );
 
  if (hasHighPriorityFlag) {
    return {
      routeToReview: true,
      priority: 'immediate',
      suggestedScenarioTitle: `[CRITICAL] ${conversation.failureReasons[0]}`,
      reason: `High-priority flag detected: ${conversation.failureReasons[0]}`,
    };
  }
 
  if (score.composite < config.scoreThreshold) {
    // Generate a scenario title from the worst dimension
    const worstDimension = Object.entries(score.scores)
      .filter(([, v]) => v !== null)
      .sort(([, a], [, b]) => (a as number) - (b as number))[0];
 
    return {
      routeToReview: true,
      priority: 'standard',
      suggestedScenarioTitle: `Low ${worstDimension[0]} (agent ${conversation.agentId})`,
      reason: `Composite score ${score.composite.toFixed(2)} below threshold ${config.scoreThreshold}`,
    };
  }
 
  if (conversation.failureReasons.length >= config.flagThreshold) {
    return {
      routeToReview: true,
      priority: 'standard',
      reason: `${conversation.failureReasons.length} quality flags: ${conversation.failureReasons.join(', ')}`,
    };
  }
 
  return {
    routeToReview: false,
    priority: 'none',
    reason: 'Score and flags within acceptable range',
  };
}

In practice, you'll run the router on every scored conversation and batch the routeToReview: true ones into a daily digest for whoever owns agent quality on your team. Conversations flagged immediate get a real-time alert.

The suggestedScenarioTitle field is a small detail that has an outsized effect on how fast the loop actually closes. If you give reviewers a pre-populated scenario title, the friction of converting a failure into a test case drops enough that it actually happens. Without it, the review queue grows and the test suite doesn't.

The full control plane architecture

Here's how all four components connect:

The loop has two paths. The fast path is automated: scoring fires immediately after each conversation, SLO evaluation and drift detection run on a schedule, and gates run at deployment time. The slow path involves humans: failed conversations go to the review queue, reviewers turn them into scenarios, scenarios go into the regression suite, the regression suite feeds the gate.

Both paths reinforce each other. The automated path catches regressions fast. The human path improves the automated path over time.

Common mistakes teams make

These are the patterns that cause the quality control plane to fail silently after you build it.

Scoring too infrequently. The most common mistake is sampling 10-20% of conversations to save on LLM scoring costs. This works fine for a static system. For drift detection, it means you're seeing one in ten data points. A regression that spans 50 conversations might only show up as five scored ones, which can look like noise rather than signal. Score everything. The cost per conversation for a scoring call is low enough that it's worth it.

Treating the composite score as the signal. A composite score hides dimension-specific regressions, as shown in the SLO section. Build your alerts and gates on individual dimensions, not just the composite. The composite is useful for a 30-second status check, not for automated decision-making.

Ignoring tool failures as a quality proxy. Tool call failures are an early warning signal for quality degradation that doesn't require LLM scoring. If your agent calls get_account_details and gets a 500 error, it's probably going to have a bad conversation. Track tool failure rates separately from conversation scores and add them to your SLO dashboard. Tool failure rates often predict quality degradation by 15-30 minutes before it shows up in scored conversations.

No rollback plan. A quality gate that blocks a deployment is only useful if you can actually roll back. Make sure your agent versioning system can revert to the previous prompt and configuration in under five minutes. If a rollback takes an hour, teams start disabling the gate to avoid deployment delays, and the whole system erodes.

Setting the window too small. Drift detection on a window of 20 conversations generates a lot of false positives on normal traffic variation. Use at least 100 conversations for the rolling window, and 500 for the baseline. Low-volume agents need longer time windows (hours or days) rather than conversation count windows.

The quality control plane isn't a one-time build. It's an ongoing commitment to treating quality as infrastructure. That means reviewing alerts, closing the feedback loop consistently, and updating SLO thresholds as your understanding of what "good" means evolves.

The alternative, periodic manual review and hope, is the system that let a billing agent quietly lose 25 points of resolution rate over three weeks without anyone noticing.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed