LLM judges grade your agents wrong. Here's how to fix them.

You set up automated evaluation. Your agent scores 4.2 out of 5 on quality, week after week. Then a customer complaint lands in your inbox: the agent told them their account was in good standing during an escalation call. Their account had been suspended for two weeks.

You pull the session. The conversation looks good. The judge gave it a 4.5. The agent's response was polite, empathetic, and used exactly the kind of language that scores well on a quality rubric. It was also wrong about the most important fact in the exchange.

The judge did not catch it because you had not told it to check whether account status claims were accurate. You had told it to check tone and task completion. Those scored fine.

This is the gap between having an LLM judge and having a calibrated LLM judge. At production scale, you cannot review every session by hand. An LLM judge is the only practical tool. But an uncalibrated judge gives you scores that feel like quality signal and are not. Here is how to fix that.

Why you need LLM judges at all

LLM judges are the only practical way to evaluate agents at production scale. A customer service agent handling 500 conversations a day produces 500 transcripts, and human review of every session is not economical even at a few minutes each.

Sampling helps, but a 5% sample means you are reviewing 25 sessions and hoping the 475 you skipped look similar. For agents where a single bad response can trigger churn or escalation, 5% human coverage is not enough signal to catch emerging problems before customers feel them.

An LLM judge can evaluate all 500 sessions overnight, flag the ones that score below threshold, and surface a prioritized list for your reviewers in the morning. The humans review 10-20 sessions a day -- the ones most worth their time -- and the judge handles breadth coverage.

This is the right division of labor. It breaks when the judge's scores are systematically miscalibrated, because then "below threshold" does not reliably mean "actually bad" and the human review queue fills with acceptable sessions while genuinely problematic ones sit at a comfortable 4.2.

The biases that affect LLM judges most

LLM judges are not neutral evaluators. They have systematic tendencies that you should know about before you trust their scores.

Length bias. LLM judges tend to score longer responses higher, independent of content quality. A verbose response that restates the customer's problem, explains the resolution step by step, and ends with a polite offer to follow up will outscore a brief, accurate response on quality dimensions where length should be irrelevant. This is documented in multiple evaluation studies and it matters for CX agents specifically, because the right response length depends on the customer's question -- not on some abstract quality standard.

If your judge has length bias, your agent will drift toward wordier responses over time, because longer answers score better in eval and prompt updates that increase verbosity look like quality improvements. The real-world signal (resolution rate, customer satisfaction scores, handle time) will tell a different story.

Self-similarity bias. A judge model rates outputs that stylistically resemble its own training distribution more favorably. A judge trained on formal writing will penalize casual conversational tone even when that tone is appropriate for the customer's context. A judge that learned on US-standard customer service will score UK-style phrasing as slightly off. These are not neutral quality judgments. They are pattern matching to what the judge has seen most.

For CX agents where the right tone depends on whether the customer is frustrated, first-time, or high-value -- not on abstract formality standards -- self-similarity bias makes the judge systematically less accurate on edge-case conversations.

Position bias. In multi-turn evaluation, judges weight earlier turns more heavily than later ones. If the agent's first response was excellent but a mid-conversation tool failure led to a poor resolution, the judge will often score the session higher than a human reviewer would. The opening quality anchors the overall score.

This is specifically dangerous for detecting regression in how agents handle complex escalations, because those cases tend to have strong openings (generic, safe responses to "how can I help you?") followed by failures deeper in the conversation.

Coverage blindness. A judge scores what you tell it to score. If your rubric has five dimensions -- tone, clarity, task completion, accuracy, empathy -- it will not catch policy violations, factual errors on unchecked domains, or off-topic drift unless those are explicitly dimensions in the rubric. The account status error from the opening would not surface on any standard quality rubric.

How to calibrate your judge

Calibration is the process of measuring the gap between what your judge scores and what a knowledgeable human reviewer would score, then closing that gap.

The mechanics are straightforward. Collect 50 to 100 conversation examples that cover your common task types. Include both successful sessions and failing ones, and make sure you have representation from the edge cases -- complaints, escalations, unusual requests -- not just the high-volume easy interactions.

Have your most experienced reviewer (or a small group) score these sessions manually on the same dimensions your judge uses. This is your ground truth.

Run your judge on the same sessions. Compare scores dimension by dimension. Look for:

• Systematic offsets: if the judge consistently scores tone 0.5 points higher than humans across the board, that is a calibration offset you can correct for
• Dimension-level disagreements: if human-judge agreement is high on task completion but poor on accuracy, you have a rubric problem in the accuracy dimension specifically
• Task-type clusters: if the judge scores refund conversations well but complaint conversations poorly relative to humans, your calibration set for complaints is probably thin
• Confidence-accuracy mismatch: if the judge gives high-confidence scores (4.5-5.0) on conversations humans rate as mediocre, the judge is not using its confidence range appropriately

The goal of this exercise is not to get perfect agreement -- it is to understand where your judge is reliable and where it is not, so you know when to trust a score and when to pull the session for review.

import Chanl from "@chanl/sdk";
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
// Run your judge on calibration sessions
const calibrationResults = await Promise.all(
  calibrationSessions.map(async (session) => {
    const judgeScore = await chanl.scorecards.evaluate({
      sessionId: session.id,
      dimensions: [
        "task_completion",
        "tone_appropriateness",
        "accuracy",
        "policy_compliance",
        "resolution_quality",
      ],
    });
 
    return {
      sessionId: session.id,
      taskType: session.taskType,
      humanScore: humanLabels[session.id],
      judgeScore: judgeScore.dimensions,
      delta: computeDelta(humanLabels[session.id], judgeScore.dimensions),
    };
  })
);
 
// Compute per-dimension calibration gaps
const calibrationGaps = computeCalibrationGaps(calibrationResults);
console.log("Calibration gaps by dimension:", calibrationGaps);
// Output: { accuracy: -0.8, tone_appropriateness: +0.3, task_completion: +0.1 }
// Accuracy is the biggest problem: judge scores 0.8 points high relative to humans

Fixing the rubric problems you find

Once you know where your judge disagrees with humans, you can usually fix it at the rubric level without changing the judge model itself.

For length bias: add an explicit instruction to the judge prompt that score should not be affected by response length. Give examples of a concise correct response (high score) and a verbose incorrect response (low score). Explicitly prompt for "the response that best solves the customer's problem" rather than "the most thorough response."

For coverage gaps: add a dimension for each domain where factual accuracy matters. For a CX agent, this typically means account status accuracy, product information accuracy, and policy accuracy as separate dimensions from the general "accuracy" bucket. This forces the judge to check each domain, rather than averaging across the ones it does check.

# Accuracy dimension -- CX agent rubric
 
Score the factual accuracy of claims the agent made. Do not score based on response length or communication style.
 
Sub-dimensions to check:
1. Account status accuracy: did any claims about the customer's account status, balance, or subscription match the account data in the context?
2. Product information accuracy: were product names, prices, and feature descriptions correct?
3. Policy accuracy: did the agent correctly state return policy, support hours, or escalation paths?
 
Score each sub-dimension 1-5. Return the minimum score as the overall accuracy score.
A single factual error should not score above 2, regardless of other response quality.

For self-similarity bias in tone evaluation: give the judge explicit examples of appropriate tones for different customer situations. A frustrated customer on their third callback should get a different tone than a first-time customer asking a simple question. Frame the rubric around "appropriateness for this specific customer's situation" rather than abstract quality standards.

For position bias in multi-turn: explicitly instruct the judge to weight the conversation's resolution over its opening. Ask it to identify the point of highest tension or complexity in the session and evaluate the agent's handling of that moment specifically.

Building a practical judge workflow

Calibration is a point-in-time activity. To keep your judge reliable over time, you need an ongoing workflow.

Run calibration monthly or whenever your traffic mix shifts significantly toward new task types. A judge calibrated on your Q1 traffic may drift on Q2 traffic if new product launches or seasonal events change the conversation patterns your agent handles. This is also why tiered eval sampling matters: your calibration set should represent the full distribution of conversation types, not just the easy high-volume ones.

Build anomaly detection on top of your judge scores. Sessions that score unusually high or low relative to similar sessions are worth human review regardless of their absolute score. A session scoring 2.8 is obviously interesting. A session scoring 4.9 when similar sessions average 3.9 is interesting for different reasons -- either the agent did something unusually right, or the judge is being unusually generous for unclear reasons.

Track inter-rater reliability between your judge and human reviewers over time. If agreement was 85% in January and drops to 72% in March, something has changed -- either your agent's behavior, your traffic mix, or the judge model itself (model updates happen). Declining agreement is an early warning that your quality scores are becoming less reliable.

The scorecard feature in Chanl tracks per-dimension scores over time, which makes it easier to spot drift in specific dimensions rather than just watching the overall quality average. A drop in accuracy scores while tone scores hold steady points to a different problem than both dropping together.

When not to trust your judge

Even a well-calibrated judge has hard limits. Know what they are.

Sensitive escalations. When a conversation involves a customer threatening to cancel, expressing significant frustration, or reporting a serious error, judge scores are less reliable. These conversations have high variance in what constitutes a good response, and the stakes of a miscalibrated score are higher. Pull these for human review.

Domain knowledge requirements. If accurate evaluation requires knowledge your judge was not trained on -- specialized product details, regulatory requirements, industry-specific terminology -- the judge will substitute surface-level quality signals for domain accuracy. A judge that cannot tell whether an agent's explanation of a financial regulation is correct will score it based on clarity and tone instead.

Novel task types. When your agent handles a task type that was not in your calibration set, judge scores on those sessions are extrapolations. They might be right, but you have no evidence. Flag novel task types for higher human review rates until you have enough labeled examples to calibrate.

Catch-all sessions. Any conversation where the agent expressed uncertainty, attempted multiple approaches, or asked clarifying questions is worth human review. These edge cases are where agents are most likely to fail in ways a standard rubric misses, and they are the cases that teach you the most about where your rubric needs expansion.

The practical formula: automate your high-confidence, high-volume cases. Reserve human review for low-confidence, high-stakes, and novel cases. Use your calibration data to know which is which. The analytics feature can surface which conversation types have the highest score variance -- those are your highest-priority calibration targets.

Getting to scores you can trust

The goal of all this work is not perfect eval coverage. It is scores that actually mean something -- where a 4.2 reflects genuine quality and a 2.8 reliably identifies a session worth investigating.

The teams that catch the most production failures are not running more evals. They are running evals they can trust. An uncalibrated judge at 100% coverage gives you less actionable signal than a calibrated judge at 20% coverage, because at least with the calibrated judge you know what the scores mean.

If you take one thing from this: collect your calibration set before you rely on judge scores to make decisions. Fifty labeled examples is enough to find the biggest gaps. The account status error from the opening would have surfaced immediately if the calibration set had included one factually-wrong-but-polite response and a human had scored it low. If you are still deciding whether to move from vibes-based review to structured scoring, this breakdown of scorecards vs gut-check evaluation covers the tradeoffs.

Your agent's monitoring stack should treat judge scores as evidence, not verdicts. Evidence requires interpretation. Calibration is what makes that interpretation accurate.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed