Most of your agent traffic doesn't need your best model

At a post-mortem last quarter, a CX engineering team traced $47,000 of their monthly inference bill to a configuration they'd never revisited. Every request, from a simple balance inquiry to a complex multi-step dispute resolution, routed to Claude Opus. The simple requests took longer than they needed to, cost 8x what they should have, and the quality difference was unmeasurable because Haiku would have handled them just as well.

They'd set it up that way during prototyping, when the priority was correctness and the bill was small. They forgot to revisit it when production traffic scaled.

Model routing is the fix. It's the practice of matching each request to the most cost-effective model capable of handling it, rather than routing everything to one model and paying frontier prices for commodity tasks. Done well, it delivers 47 to 86% cost reductions without touching quality. Done poorly, it just adds latency and complexity. The difference is in how you build the routing logic.

Why routing matters more for agents than for chatbots

The cost math changes dramatically when you move from single-call LLM applications to agentic workflows.

A chat application makes one LLM call per user turn. The context is usually short, the task is bounded, and the per-call cost is small. For an agent handling a customer interaction, a single interaction might include intent classification, CRM lookup, policy retrieval, a drafted response, a follow-up clarification, and a final resolution confirmation. That's 5 to 10 LLM calls, a context window growing from 2,000 tokens to 40,000 across the interaction, and each call paying full input token prices for all the tokens that came before.

A single agent task uses 50,000 to 500,000 tokens when you include planning steps, tool call logging, and verification loops. At frontier model prices, that adds up fast. And the critical insight is that not every one of those LLM calls requires frontier reasoning. Intent classification is classification. Tool result summarization is summarization. These are not the same problem as multi-step reasoning under ambiguity.

Routing to the right model per call, rather than per session, is where the savings accumulate.

Three routing strategies

Routing approaches differ in where in the pipeline the routing decision happens and what information the router has access to.

Rule-based routing classifies the incoming request before the LLM call and routes based on that classification. Intent classifiers, regex patterns, and keyword matching are common. Rule-based routing is fast (sub-millisecond overhead), predictable (no LLM call required to route), and transparent. Its limitation is that it requires you to specify the rules explicitly, and real customer requests don't always fit clean categories.

Cascade routing sends the request to a cheap model first, evaluates the response, and escalates to a stronger model if quality doesn't meet the threshold. The cascade router doesn't need to classify the request upfront. It learns empirically which requests the cheap model can handle by trying. The cost of the cascade is an extra quality evaluation call on every request, which is small if you use a fast model for evaluation.

Confidence-based routing reads the model's own uncertainty signal (logprobs or explicit self-assessment) and escalates when the model expresses low confidence. This requires model support for logprob access or a confidence calibration pass. It works well for classification tasks where the model's logprobs correlate with actual accuracy, but it's harder to apply to generation tasks where uncertainty is harder to quantify.

In practice, production systems use all three in combination: rule-based routing handles the easy cases (balance inquiry, business hours lookup) before any LLM call, cascade handles everything else, and confidence-based escalation provides a safety net for high-stakes decisions.

Building a cascade router

The cascade architecture has four components: the primary cheap model, a quality evaluator, an escalation path to a strong model, and a routing log for monitoring.

interface RoutingPolicy {
  primaryModel: string;
  escalationModel: string;
  evaluatorModel: string;
  qualityThreshold: number;     // 0-10, escalate if below
  maxEscalationRate: number;    // alert if escalation rate exceeds this
}
 
interface RoutingResult {
  response: string;
  tier: "primary" | "escalation";
  primaryTokens: number;
  escalationTokens: number;
  qualityScore: number;
  escalationReason?: string;
}
 
const DEFAULT_POLICY: RoutingPolicy = {
  primaryModel: "claude-haiku-4-5-20251001",
  escalationModel: "claude-sonnet-4-6",
  evaluatorModel: "claude-haiku-4-5-20251001",  // use cheap model for evaluation too
  qualityThreshold: 7.0,
  maxEscalationRate: 0.25,
};
 
async function cascadeRoute(
  request: AgentRequest,
  policy: RoutingPolicy = DEFAULT_POLICY
): Promise<RoutingResult> {
  // Step 1: Try the cheap model
  const primaryResponse = await model.complete({
    model: policy.primaryModel,
    messages: request.messages,
    system: request.systemPrompt,
    max_tokens: request.maxTokens ?? 1024,
  });
 
  // Step 2: Evaluate response quality
  const qualityScore = await evaluateResponse(
    request,
    primaryResponse.content,
    policy.evaluatorModel
  );
 
  // Step 3: If quality meets threshold, return primary response
  if (qualityScore >= policy.qualityThreshold) {
    await logRoutingDecision({
      requestId: request.id,
      tier: "primary",
      model: policy.primaryModel,
      qualityScore,
      tokens: primaryResponse.usage.inputTokens + primaryResponse.usage.outputTokens,
    });
 
    return {
      response: primaryResponse.content,
      tier: "primary",
      primaryTokens: primaryResponse.usage.inputTokens + primaryResponse.usage.outputTokens,
      escalationTokens: 0,
      qualityScore,
    };
  }
 
  // Step 4: Escalate to strong model
  const escalationResponse = await model.complete({
    model: policy.escalationModel,
    messages: request.messages,
    system: request.systemPrompt,
    max_tokens: request.maxTokens ?? 2048,
  });
 
  await logRoutingDecision({
    requestId: request.id,
    tier: "escalation",
    model: policy.escalationModel,
    primaryQualityScore: qualityScore,
    escalationReason: `quality below threshold: ${qualityScore.toFixed(1)}`,
    tokens: escalationResponse.usage.inputTokens + escalationResponse.usage.outputTokens,
  });
 
  return {
    response: escalationResponse.content,
    tier: "escalation",
    primaryTokens: primaryResponse.usage.inputTokens + primaryResponse.usage.outputTokens,
    escalationTokens: escalationResponse.usage.inputTokens + escalationResponse.usage.outputTokens,
    qualityScore: qualityScore,
    escalationReason: `score ${qualityScore.toFixed(1)} below threshold ${policy.qualityThreshold}`,
  };
}
 
async function evaluateResponse(
  request: AgentRequest,
  response: string,
  evaluatorModel: string
): Promise<number> {
  const evaluation = await model.complete({
    model: evaluatorModel,
    system: `You evaluate AI agent responses for quality. Score the response 0-10.
Score 0-4: Response is wrong, incomplete, or harmful.
Score 5-6: Response attempts the task but has gaps or errors.
Score 7-8: Response correctly addresses the request with minor imperfections.
Score 9-10: Response is accurate, complete, and well-structured.
Return only a JSON object: {"score": <number>, "reason": "<brief explanation>"}`,
    messages: [
      { role: "user", content: `ORIGINAL REQUEST:\n${request.messages.at(-1)?.content}\n\nRESPONSE TO EVALUATE:\n${response}` }
    ],
    max_tokens: 100,
  });
 
  const parsed = JSON.parse(evaluation.content);
  return parsed.score;
}

The quality evaluator is itself an LLM call, which means you're paying for two model calls on every request. This is still cheaper than always using the expensive model because the two cheap model calls (primary + evaluator) cost a fraction of one escalation-model call. The math works as long as your cheap-tier resolution rate is above roughly 70%.

Total Calls

0+12%

Avg Duration

4:23-8s

Resolution

0%+3%

Live Dashboard

Active calls23

Avg wait0:04

Satisfaction98%

What to route where

Not every task type requires the same model capability. Knowing which tasks your cheap model handles well lets you set your routing rules and quality threshold appropriately.

Tasks that cheap models handle well:

• Single-turn factual lookup: balance inquiry, business hours, product specs
• Entity extraction from structured text: pulling account IDs, dates, and amounts from forms
• Intent classification: routing between departments or task types
• Response formatting: converting structured data to readable prose
• Tool result summarization: condensing a 500-token API response to 80 tokens for the next step

Tasks that benefit from stronger models:

• Multi-step reasoning where early errors cascade: troubleshooting sequences where a wrong diagnosis leads to wrong recommendations
• Ambiguous intent resolution: a customer message that could mean several different things
• High-stakes decisions: anything that involves authorization, account changes, or escalation to humans
• Tool call chaining with complex dependencies: when tool B's input depends on correctly interpreting tool A's output
• Novel scenarios outside the training distribution: the edge cases your agent hasn't seen before

For a typical CX agent, 60 to 70 percent of traffic is single-turn factual lookup or simple action confirmation. This is your primary tier. 20 to 25 percent is multi-turn problem resolution with clear structure. This often succeeds at the primary tier but escalates more frequently. 10 to 15 percent is complex or ambiguous work that regularly requires the escalation model.

Monitoring your router

The escalation rate is your live cost control dial. If it drifts up, your costs go with it. If it drops suddenly, your quality threshold might be too lenient.

import { chanl } from "@chanl/sdk";
 
interface RoutingMetrics {
  window: "1h" | "24h" | "7d";
  escalationRate: number;
  avgPrimaryTokens: number;
  avgEscalationTokens: number;
  avgCostPerRequest: number;
  qualityScoreP50: number;
  qualityScoreP10: number;  // low tail: poor quality on primary tier
}
 
async function getRoutingMetrics(window: RoutingMetrics["window"]): Promise<RoutingMetrics> {
  const logs = await chanl.analytics.getMetrics({
    event: "routing_decision",
    window,
    groupBy: "tier",
  });
 
  const totalRequests = logs.primary.count + logs.escalation.count;
  const escalationRate = logs.escalation.count / totalRequests;
 
  const avgCostPerRequest = calculateBlendedCost(
    logs.primary.count,
    logs.primary.avgTokens,
    logs.escalation.count,
    logs.escalation.avgTokens
  );
 
  return {
    window,
    escalationRate,
    avgPrimaryTokens: logs.primary.avgTokens,
    avgEscalationTokens: logs.escalation.avgTokens,
    avgCostPerRequest,
    qualityScoreP50: logs.primary.qualityScoreP50,
    qualityScoreP10: logs.primary.qualityScoreP10,
  };
}
 
// Alert if escalation rate crosses threshold
async function checkRoutingHealth(policy: RoutingPolicy): Promise<void> {
  const metrics = await getRoutingMetrics("1h");
 
  if (metrics.escalationRate > policy.maxEscalationRate) {
    await chanl.monitoring.alert({
      name: "high_escalation_rate",
      severity: "warning",
      message: `Escalation rate ${(metrics.escalationRate * 100).toFixed(1)}% exceeds threshold ${(policy.maxEscalationRate * 100).toFixed(0)}%`,
      context: { metrics },
    });
  }
 
  if (metrics.qualityScoreP10 < 5.0) {
    await chanl.monitoring.alert({
      name: "low_primary_quality_tail",
      severity: "warning",
      message: `10th percentile quality score ${metrics.qualityScoreP10.toFixed(1)} below floor`,
      context: { metrics },
    });
  }
}

Use Chanl's monitoring to set continuous escalation rate tracking and alert on anomalies. A sudden spike in escalation rate usually means one of three things: the traffic distribution changed (new customer segment, new product launch), the cheap model had a regression, or the quality threshold needs recalibration for a new task type. Each has a different fix, and you can only diagnose the difference if you're watching the escalation rate continuously.

Calibrating the quality threshold

The quality threshold is the most important parameter in your router. Too high and you're paying escalation model prices for tasks the cheap model handles fine. Too low and you're serving low-quality responses to customers.

Start by running your router in shadow mode: log routing decisions and quality scores but don't gate on them. Let 100% of traffic flow through the primary model and the evaluator, and collect quality scores for 1,000 to 2,000 requests. Look at the distribution of quality scores for requests you already know were resolved well (customer didn't escalate, CSAT was positive). The score where 90% of successful resolutions sit becomes your starting threshold.

Adjust after two to three weeks of production routing, once you can compare escalation rate with customer satisfaction metrics side by side. The agent unit economics article covers how to connect per-request cost to successful resolution rate, which gives you the ROI calculation for routing decisions. The cost optimization guide has context on the other cost levers (batch sizing, context compression) that complement routing.

The bill from the opening story

The team that found $47,000 in unnecessary Opus spend implemented a two-tier cascade with Haiku as the primary and Sonnet as the escalation path. Over the following month, their primary resolution rate stabilized at 74%. Their monthly inference bill dropped from $47,000 to $11,000. Quality scores held steady. The 26% that escalated to Sonnet actually got better responses than before, because the evaluator was catching cases where Haiku was producing plausible-but-wrong answers that would have gone undetected under the old always-Opus setup.

The routing layer didn't just save money. It surfaced a category of failures they hadn't been measuring.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed