Eval sampling for production agents: the 100/10/1 playbook

A team running a customer service agent gets their monthly AI bill. They budgeted for the agent itself -- generation costs, tool calls, memory lookups. What they didn't account for: LLM-as-judge running on 100% of traffic costs almost as much as the agent itself.

This is a real pattern. Teams add quality evaluation incrementally, one conversation at a time, until the eval stack is as expensive as the thing it's evaluating. At that point you face two bad options: pay it, or turn off evaluation and fly blind.

There's a third option. It's called tiered sampling, and it's what production teams with mature eval practices actually use. The structure: 100% of conversations get fast, cheap heuristic checks. A 5-10% random sample goes to LLM-as-judge. About 1% -- plus all conversations that fail Tier 1 or Tier 2 -- goes to human review. You maintain meaningful quality coverage at a fraction of the full-coverage cost.

This article walks through how to build a tiered eval system that actually works in production: what goes in each tier, how to route escalations, how to set your sampling rates, and how to make the three tiers talk to each other.

Why full coverage doesn't scale

LLM judge is genuinely useful. It catches semantic quality failures that heuristics can't see -- off-brand tone, incorrect information presented confidently, missed escalation opportunities, poor handling of an edge case. The fundamentals of LLM-as-judge evaluation are worth building into your system. The problem is running it on everything.

The math at scale: say your agent handles 100,000 conversations per day at $0.02 each. Full LLM judge coverage at $0.01 per evaluation adds $1,000 per day -- $30,000 per month just on scoring. Add human review on 10% of flagged conversations at $0.50 each, and you're looking at another $15,000 per month.

The operational problem compounds the cost problem. At 100% LLM judge coverage, your human review queue fills with thousands of conversations per day -- a volume no QA team can actually process. You end up with a review backlog that defeats the purpose of having a review queue.

Tiered sampling solves both. By concentrating expensive evaluation on a targeted sample, you make the human review queue manageable, reduce costs significantly, and maintain statistical confidence in your quality estimates.

Tier 1: Heuristics on everything

The fastest, cheapest evaluation runs on every conversation. These are signal-based catches that require no model inference, just deterministic checks against your conversation log.

Response length anomalies. Calculate your P5 and P95 response lengths per intent type. Flag anything outside those bounds. A 3-word response to a complex rebooking question is almost certainly a failure. A 500-word response to "what's my balance" is almost certainly a problem.

Blocked phrase detection. Regex checks for content your agent should never produce: competitor names, legal admissions, medical advice, profanity, or system prompt contents. These are categorical failures that don't need semantic evaluation -- a regex catches them in milliseconds.

Confidence drops. If your orchestration layer exposes confidence scores, flag conversations where the agent expressed low confidence on the primary intent but continued anyway without escalating.

Tool call errors. Any conversation with a tool call that returned an error -- especially one the agent didn't acknowledge to the customer -- is a candidate for review.

Consecutive null responses. An agent that loops back to "How can I help you?" twice in a row is confused. This pattern is detectable without inference.

Latency spikes. P99 latency isn't just a performance metric. Conversations with unusual latency often have unusual content -- extra tool calls, excessive self-correction, unresolved states.

Policy violations. Hard rules detectable structurally: agent revealed system prompt contents, agent made a commitment it can't keep (promised a refund on a non-refundable item), agent shared credentials.

The implementation is a set of deterministic checks that run against the structured conversation log. Cost: fractions of a cent per conversation.

interface ConversationLog {
  conversationId: string;
  turns: Turn[];
  toolCalls: ToolCall[];
  durationMs: number;
  intentType: string;
}
 
async function runTier1Checks(log: ConversationLog): Promise<CheckResult> {
  const checks = await Promise.all([
    checkResponseLengths(log),
    checkBlockedPhrases(log),
    checkToolErrors(log),
    checkConsecutiveNullResponses(log),
    checkLatencyAnomaly(log),
    checkPolicyViolations(log),
  ]);
 
  const failures = checks.filter(r => r.failed);
 
  return {
    passed: failures.length === 0,
    failures,
    escalateToTier2: failures.length > 0,
    tier: 1,
  };
}

Any conversation that fails Tier 1 goes straight to Tier 2. No sampling. Failures always escalate, never get excluded by the sampler.

Tier 2: LLM judge on a sample

The second tier runs LLM-as-judge on two sets of conversations: a random sample of the traffic that passed Tier 1, plus all conversations that failed Tier 1.

The random sample is where you catch what Tier 1 can't see -- semantic quality, factual accuracy, tone, escalation judgment. The Tier 1 failures come here for a more detailed verdict before deciding whether to route to human review.

What your LLM judge should evaluate

Most LLM judge prompts try to evaluate too many dimensions at once. The result is noisy aggregate scores that don't tell you what to fix. The pattern that works: focused evaluators, each scoring one dimension.

Intent accuracy. Did the agent correctly identify what the customer wanted? Output: yes/no with evidence.

Resolution quality. If the conversation closed as resolved, was it actually resolved? Output: 1-5 with reasoning.

Policy compliance. Did the agent follow your defined policies -- escalation rules, refund criteria, disclosure requirements? Output: pass/fail with specific citation.

Tone and brand voice. Did the response feel like your brand? Output: 1-5.

Escalation judgment. When the agent chose to escalate (or not escalate), was it the right call? Output: correct/incorrect with explanation.

Running each dimension as a separate prompt produces more reliable scores than trying to evaluate all dimensions in one call. The extra inference cost is usually worth the improvement in signal quality.

const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
async function runTier2Judge(conversationId: string): Promise<JudgeResult> {
  const result = await chanl.scorecards.evaluate({
    conversationId,
    dimensions: [
      "intent_accuracy",
      "resolution_quality",
      "policy_compliance",
      "tone_brand_voice",
      "escalation_judgment",
    ],
    model: "claude-opus-4-8",
    outputMode: "scored_with_evidence",
  });
 
  return {
    conversationId,
    scores: result.dimensions,
    overallScore: result.aggregate,
    passesThreshold: result.aggregate >= 3.5,
    flagForHumanReview: result.aggregate < 2.5 || result.hasCategoricalFailure,
    tier: 2,
  };
}

The threshold for escalating to Tier 3 is something you tune. Start at 2.5/5.0 for the overall score, or any categorical failure (a zero on any dimension). Watch how often your human reviewers overturn the LLM judge verdict. If they agree most of the time, your threshold is calibrated. If they're overturning more than 30%, adjust.

Setting your sampling rate

The right sample rate depends on how much signal you're getting from Tier 1, and how much your traffic varies by intent type.

Start at 5% and run for two weeks. At the end of the period, look at what the sample revealed: what percentage of sampled conversations had a quality issue that wasn't flagged by Tier 1? If it's above 15%, your heuristics are missing something important and you need to either add heuristics or increase the sample. If it's below 5%, your heuristics are catching most issues and you might be able to lower the rate.

Most teams land between 5% and 15%, depending on traffic diversity and heuristic maturity.

One nuance: sample rate shouldn't be uniform. Oversample conversations with:

• New tools enabled after a recent deployment
• Topics with known edge cases or high escalation rates
• High-value customers (high LTV or high churn risk)
• Conversations that include spending events (AP2 transactions)

function shouldSampleForTier2(
  log: ConversationLog,
  tier1Result: CheckResult
): boolean {
  // Tier 1 failures always escalate
  if (!tier1Result.passed) return true;
 
  // Oversample targeted categories
  if (log.hasNewToolEnabled || log.postPromptChange) return true;
  if (log.customerLTV > 10000) return true;
  if (log.intentType === "cancellation_intent") return true;
  if (log.hasSpendEvent) return true;
 
  // Base random sample
  return Math.random() < 0.05;
}

Deploy Gate

Pre-deploy quality checks

Score > 80%

92%

Latency < 500ms

234ms

Error Rate < 2%

3.1%

Deploy Blocked

Tier 3: Human review where it counts

Human review is your ground truth. It's expensive at scale, so you use it selectively. Three categories of conversations belong in the Tier 3 queue.

Systematic escalations from Tier 2. Any conversation where LLM judge scored below your threshold, or flagged a categorical failure. These need a human verdict to confirm or overturn the automated score.

Targeted pulls. Conversations you choose to review regardless of automated scores: new capability launches (the first week after enabling a new tool), complaints from customers who followed up, conversations with unusual spending patterns, any conversation that ended with a human handoff. You're not sampling these -- you're deliberately choosing them.

Calibration pulls. A small random set of conversations that passed both Tier 1 and Tier 2. This is how you catch when your automated scoring is systematically wrong. If your human reviewers find quality issues in 20% of your calibration pulls, your scoring thresholds are off.

The human review interface matters. Reviewers who navigate raw conversation logs make worse decisions and review fewer conversations per hour. Build a queue that shows the conversation, the LLM judge verdict with evidence, and a simple pass/fail form with a notes field. Speed and clarity produce better calibration data.

The notes are the highest-value output of Tier 3. When a reviewer overturns an LLM judge verdict and explains why, that explanation is training data for improving your judge prompt. Collect it, categorize it monthly, and use it in your calibration sessions.

How the three tiers talk to each other

The system only works if escalation is automatic and bidirectional. The routing logic:

• All conversations enter Tier 1.
• Tier 1 failures escalate to Tier 2 immediately.
• A random sample of Tier 1 passes also go to Tier 2 at your defined rate.
• Tier 2 failures escalate to Tier 3.
• Targeted and calibration pulls also go to Tier 3 regardless of scoring.
• Tier 3 human review verdicts feed back to calibrate Tier 2 thresholds.

The feedback loop from Tier 3 to Tier 2 is the part most implementations get wrong. Human reviewers overturn automated scoring, but that information never reaches the judge prompt. Over time, the judge and the human reviewers drift apart. The fix: monthly calibration sessions where you review overturned verdicts, identify patterns, and update the judge prompt.

Connecting tiers to your monitoring system

Each tier produces structured output that belongs in your analytics dashboards and production monitoring.

Tier 1 output: per-conversation pass/fail, which checks triggered, flag rate over time. The flag rate is a leading indicator. If it spikes after a deployment, something changed in your agent's behavior.

Tier 2 output: per-dimension scores, overall score distribution, false negative rate measured against Tier 3 overturns. The score distribution over time tells you whether quality is improving or degrading, even without full human review coverage.

Tier 3 output: human verdicts, overturn rates by dimension, notes text categorized and aggregated. The overturn rate by dimension tells you which scoring dimensions are weakest and need prompt improvement.

const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
async function logTierResult(
  conversationId: string,
  tierResult: TierResult
): Promise<void> {
  await chanl.calls.logEvent({
    conversationId,
    eventType: "eval_tier_result",
    payload: {
      tier: tierResult.tier,
      passed: tierResult.passed,
      dimensionScores: tierResult.dimensionScores,
      escalatedToNextTier: tierResult.escalate,
      flagReason: tierResult.flagReason ?? null,
    },
  });
}

When you join tier results with conversation metadata -- intent type, agent version, tool configuration, customer segment -- you can answer questions like: "Has quality improved since the prompt change on June 10th?" and "Is the new booking tool producing lower quality responses than the old one?" This is what score drift detection looks like in practice, with enough granularity to act on.

The calibration cadence

Tiered sampling only stays accurate if you maintain the connection between automated tiers and ground truth. A monthly calibration session keeps the system honest.

1. Pull 50-100 conversations from Tier 3 human review, split between escalations and calibration pulls.
2. Compute the overturn rate: what percentage of LLM judge verdicts did human reviewers change?
3. Categorize the overturns: which dimensions were wrong most often? What patterns appear in the notes?
4. Update Tier 2 judge prompts based on the patterns found.
5. Adjust Tier 1 heuristics if patterns show you're missing systematic failures.
6. Re-run your sample rate analysis: is 5% still the right rate given your current Tier 1 catch rate?

Monthly is the right cadence for most teams. Weekly if you're iterating quickly on prompts or tools. Quarterly if your system is stable and overturn rates are consistently below 10%.

Starting small, staying honest

You don't need to build the full three-tier system at once. Start with Tier 1 heuristics. These are cheap, fast, and immediately useful. Pick five checks that cover your most common failure modes. Run them for a month. Understand your flag rate.

Then add Tier 2 on a small sample. Start at 5%. Run for two weeks and look at what you're catching that Tier 1 missed.

Only add Tier 3 once you have a Tier 2 queue that's small enough to actually process. If Tier 2 flags 500 conversations per day that score below threshold, you need human review capacity to match. Build the interface at the same time as the queue.

The teams that ship this well are the ones who connect evaluation to action. Quality scores sitting in a dashboard without driving prompt changes, tool improvements, or monitoring alerts aren't worth the compute. Each tier should connect to a workflow where findings become improvements. The tiered system makes it affordable to maintain that loop at production scale -- and that's the whole point.

The team with the billing shock? They tiered their eval stack and cut their monthly eval cost by 85% without reducing their quality catch rate. They now know which conversations to care about, rather than drowning in every one.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed