How LLM Routing Cuts Agent Costs 40-85% in Production

You're running a customer service agent on Claude Sonnet. Every call comes in, every call hits the same model. Most of them are "What are your store hours?" and "Where's my package?" By the end of the month, you have a $40k bill. You pull the logs and discover that about 60% of that spend went to processing queries a much cheaper model would have answered just as well.

That's not a model problem. It's a routing problem.

Most teams default to a single model for their agent. It's the path of least resistance, and for early deployments, it's the right call. You're not sure what your traffic looks like, and optimizing a system you haven't built yet is a waste of time. But once you have production traffic, the cost profile becomes visible, and most teams find the same thing: a large fraction of their queries don't need a frontier model.

Model routing fixes this. You classify each incoming request by complexity and send it to the cheapest model that can handle it. The RouteLLM research benchmark reported cost reductions up to 85% on MT-Bench while preserving 95% of GPT-4 quality, with smaller savings (around 45%) on harder benchmarks like MMLU. The "well-tuned" part is the work. Here's how to do it without breaking your agent.

What Is Model Routing, Actually?

Routing means classifying each query before it reaches your language model, then sending it to the cheapest model tier that can handle it well. In code, it's a single function: query goes in, a model name comes out.

The reason it works is that query difficulty is not uniformly distributed. In a typical CX deployment, a large share of incoming queries are simple: they ask for factual information, request a status check, or confirm a policy. These queries don't benefit from a frontier model's deeper reasoning or longer context. They need a fast, coherent, factual answer, and models like GPT-4o Mini or Claude Haiku provide that at a fraction of the cost.

The pricing gap is substantial. A Sonnet-class frontier call runs about $0.015 per 1,000 output tokens. A fast model like GPT-4o Mini runs closer to $0.0006 per 1,000 output tokens, roughly 25x cheaper. At scale, routing even 50% of queries to the cheaper tier turns a $40k monthly bill into something closer to $22-25k. Route 70%, and you're under $15k.

The catch is that you need to route correctly. A complex complaint query that lands on Haiku and gets a shallow answer hasn't saved money; it's generated a callback, a human escalation, and a customer who's more frustrated than when they started. The routing decision has to be right.

Three Strategies for Routing in Production

There isn't one right routing strategy. The best choice depends on how predictable your traffic is and how much engineering time you want to invest. Three approaches work in production, and each breaks in a different place.

Rule-Based Routing

Rule-based routing maps recognized intents to model tiers. You define the routing logic explicitly. No inference needed.

type ModelTier = "fast" | "standard" | "frontier";
 
// Intents where a fast model is consistently sufficient
const FAST_TIER_INTENTS = new Set([
  "hours_inquiry",
  "location_inquiry",
  "return_policy_lookup",
  "order_status_simple",
  "shipping_estimate",
  "faq_general",
]);
 
// Intents that always need the frontier model
const FRONTIER_TIER_INTENTS = new Set([
  "billing_dispute",
  "account_compromise",
  "refund_escalation",
  "legal_question",
  "fraud_report",
  "compliance_inquiry",
]);
 
function routeByIntent(intent: string): ModelTier {
  if (FAST_TIER_INTENTS.has(intent)) return "fast";
  if (FRONTIER_TIER_INTENTS.has(intent)) return "frontier";
  return "standard"; // conservative default
}

This is the lowest-overhead approach. A hash lookup adds under 1ms. No model call needed for routing. The rule set is legible: any team member can read it and understand why a given intent routes where it does.

The weakness is coverage. You need to anticipate every intent category. New query types that your classifier hasn't seen default to your fallback tier. Set that fallback to "standard" rather than "fast". A query incorrectly defaulting to standard costs you a little more than optimal; a query incorrectly defaulting to fast can hurt the customer. When in doubt, route up.

Complexity Scoring

Complexity scoring routes based on observable features of the query itself, not just the classified intent. This handles queries that don't fit neatly into your intent taxonomy.

interface QuerySignals {
  tokenCount: number;
  questionCount: number;          // how many "?" characters
  requiresToolCall: boolean;       // mentions booking, refund, account changes
  sentimentSignal: "neutral" | "frustrated" | "angry";
  mentionsPersonalData: boolean;   // account number, name, email in message
  priorTurnFailed: boolean;        // previous message in session hit a tool error
}
 
function scoreComplexity(signals: QuerySignals): number {
  let score = 0;
 
  // Length signals depth of question
  if (signals.tokenCount > 40) score += 1;
  if (signals.tokenCount > 80) score += 2;
 
  // Multiple questions suggest multi-step reasoning needed
  if (signals.questionCount > 1) score += 2;
 
  // Tool calls require the model to reason about parameters correctly
  if (signals.requiresToolCall) score += 3;
 
  // Frustrated customers need more careful handling
  if (signals.sentimentSignal === "frustrated") score += 2;
  if (signals.sentimentSignal === "angry") score += 4;
 
  // Personal data increases the stakes of a wrong answer
  if (signals.mentionsPersonalData) score += 2;
 
  // Session already had a failure; escalate to be safe
  if (signals.priorTurnFailed) score += 5;
 
  return score;
}
 
function routeByComplexity(signals: QuerySignals): ModelTier {
  const score = scoreComplexity(signals);
  if (score <= 2) return "fast";
  if (score <= 7) return "standard";
  return "frontier";
}

Complexity scoring generalizes better than rule-based routing because it handles novel query types that fall outside your intent taxonomy. A new customer complaint phrased in a way your intent classifier has never seen will still score highly on the sentiment and tool-call signals.

The tradeoff is calibration. You need real traffic data to set your thresholds correctly. Before you have that data, start with conservative thresholds: push anything ambiguous to "standard". Once you have two to four weeks of production data, look at the score distribution and the quality outcomes for each tier. Adjust the tier boundaries to maximize cost savings while holding resolution rate constant.

Confidence-Based Escalation

Confidence-based escalation runs the cheap model first and only escalates when the cheap model signals it isn't certain. This is the most efficient approach when most of your queries genuinely are simple, because you only pay for the expensive model on the fraction that needs it.

const CONFIDENCE_THRESHOLD = 0.82;
 
async function routeWithEscalation(
  message: string,
  context: ConversationContext
): Promise<AgentResponse> {
  // Fast model attempts first
  const fastResponse = await callFastModel(message, context, {
    // Ask the model to express its own confidence
    systemAddendum: `After your response, include a JSON block:
{"confidence": 0.0-1.0, "escalate": true/false}
Set escalate to true if you are uncertain, if the query involves account changes,
financial decisions, or if you do not have enough context to answer completely.`,
  });
 
  const { confidence, escalate } = parseSelfAssessment(fastResponse);
 
  if (confidence >= CONFIDENCE_THRESHOLD && !escalate) {
    return { content: fastResponse.content, model: "fast" };
  }
 
  // Escalate with the fast model's attempt as additional context
  const frontierResponse = await callFrontierModel(message, context, {
    priorAttempt: fastResponse.content,
    escalationReason: `Fast model confidence: ${confidence}`,
  });
 
  return { content: frontierResponse.content, model: "frontier" };
}

The thing to watch is your escalation rate. If your fast model escalates on 40% of queries, you're paying for two model calls on nearly half your traffic. The break-even depends on your tier pricing, but generally you want escalation rates below 25% before confidence-based routing shows a net cost benefit over simple tier assignment.

Also be careful about self-reported confidence. Language models are not perfectly calibrated. Haiku may express high confidence on a query where it's actually producing a plausible but wrong answer. Treat self-assessed confidence as one signal among several, and validate it against your actual quality outcomes in production.

How Do You Estimate Savings Before Building?

Before committing to a routing layer, run the numbers on your actual traffic. The savings vary a lot by deployment, and for some teams the engineering cost isn't worth it. The calculation takes 15 minutes. Pull a sample of 1,000 recent conversations. Label each as "simple", "standard", or "complex" based on what you know about the query type and outcome. Count the distribution.

If 70% of your calls land in "simple", routing is worth building. If 30% land there, the complexity of maintaining a router might not justify the savings. The sweet spot for routing is deployments where a meaningful fraction of traffic is genuinely low-complexity, not deployments with uniformly hard queries.

This rough cost model sanity-checks the numbers:

interface RoutingCostModel {
  monthlyCallVolume: number;
  avgTokensPerCall: number;
  currentModelCostPer1kTokens: number;
  fastModelCostPer1kTokens: number;
  fractionRoutedToFast: number;
  classifierCostPerCall: number; // if using an ML classifier
}
 
function estimateMonthlySavings(model: RoutingCostModel): number {
  const totalTokens = model.monthlyCallVolume * model.avgTokensPerCall;
  const currentCost = (totalTokens / 1000) * model.currentModelCostPer1kTokens;
 
  const fastCallTokens = totalTokens * model.fractionRoutedToFast;
  const standardCallTokens = totalTokens * (1 - model.fractionRoutedToFast);
 
  const routedCost =
    (fastCallTokens / 1000) * model.fastModelCostPer1kTokens +
    (standardCallTokens / 1000) * model.currentModelCostPer1kTokens +
    model.monthlyCallVolume * model.classifierCostPerCall;
 
  return currentCost - routedCost;
}

Run this with your real numbers. If the projected savings exceed a few thousand dollars a month, the engineering investment is clearly worthwhile. If the savings are in the hundreds, consider whether you have better optimization levers first, like cutting unnecessary tool calls or improving context efficiency.

How Much Latency Does Routing Add?

Routing overhead ranges from under 1ms to roughly 200ms, and which end you land on comes down to the strategy. Rule-based and complexity scoring stay in the low single-digit to low-double-digit milliseconds. Confidence-based escalation is the costly one, since it runs a full fast-model call before it can decide whether to escalate.

A regex-based intent router adds under 1ms. A complexity scoring function that runs on the incoming message adds 2-5ms. A fine-tuned classifier model or embedding similarity lookup adds 5-20ms. Confidence-based escalation adds a full fast-model round-trip, typically 80-150ms, plus the frontier model call on escalations.

For most CX agents responding in 300-600ms, 15ms of routing overhead is invisible. But if you're targeting tighter latency, particularly for voice agents where response time affects perceived conversational flow, measure the overhead before committing to a strategy.

The latency concern with confidence-based escalation is real. When 25% of your calls escalate, those calls are effectively paying 1.5x the latency: fast model turn, confidence check, frontier model turn. If your p95 latency budget is already tight, this eats into it. In those situations, rule-based or complexity scoring keeps the routing decision off the critical path.

Gotchas That Catch Teams Off-Guard

Four problems reliably surprise teams building routing layers for the first time: safety-critical queries that look simple, routing drift as your product changes, calibration that decays after a model update, and the temptation to classify with another LLM. None of them are hard to avoid once you know they exist.

Safety-critical queries can look simple. "I think someone accessed my account" is 8 words. It's short, uses plain vocabulary, and would score low on a token-count-based complexity scorer. It's also exactly the kind of query where you want your best model. Build a hard exclusion list: any query containing keywords related to fraud, account security, legal disputes, or financial transactions routes to frontier regardless of the score. Don't let the classifier override you on these.

Routing drift happens. Your product ships a new feature in month three. Suddenly a wave of "how do I use the new scheduling tool?" queries hits your agent. Your classifier has never seen these, so they fall into your default tier. If that default is "fast" and the new feature is complex, quality suffers. Add routing distribution to your weekly metrics review. A sudden shift in the fraction of calls hitting each tier is a signal to investigate.

Calibration degrades with model updates. You tuned your complexity thresholds against GPT-4o Mini version N. GPT-4o Mini version N+1 ships with different capability characteristics. Your thresholds may no longer reflect the actual quality boundary. After any model update, run a calibration pass: sample calls from each tier, score them, and check that your tier assignments still correspond to quality differences.

Don't use another LLM to classify. A tempting approach is using a fast LLM to classify the query before routing. This adds a third model call on escalations and negates some of your savings. Use a fine-tuned classifier, a regex intent matcher, or an embedding similarity lookup instead. The classification doesn't need language model reasoning; it needs a fast, deterministic mapping.

Monitoring Both Paths Separately

Once routing is live, a single aggregate quality score misleads you. If your fast-model path is quietly producing worse outcomes, that signal gets diluted by the standard-model path. You need separate quality tracking per tier.

Track these metrics for each routing tier independently:

• Resolution rate: did the conversation end with the customer's issue resolved, without a callback or escalation?
• Transfer rate: how often did the fast-model path end in a human handoff?
• Repeat contact rate: did the same customer call back within 48 hours with the same issue?
• Conversation score: if you're running quality scoring, what's the average score per tier?

If your fast-model path has 2x the transfer rate of your standard path on the same intent categories, your classifier is routing queries that are too complex for the cheaper model. Tighten the complexity threshold or move the miscategorized intents to a higher tier.

Chanl's analytics lets you tag conversations by the routing tier used and filter quality metrics by tier, so you can run these comparisons without building a separate dashboard. The scoring rubrics run the same evaluation criteria against both paths, which makes quality divergence visible even before transfer and repeat-contact rates surface it.

This is where the trace-to-dataset loop becomes valuable for routing specifically. When a fast-model call produces a poor outcome, that trace is a candidate for your eval set. Over time, your eval dataset accumulates cases where the fast model failed, and you can use them to improve your classifier by studying what these queries have in common.

A Staged Rollout Strategy

Don't route everything on day one. The risk of a broad rollout is discovering quality problems at scale, with many customers already affected. Instead, route one intent at a time, prove parity against a control group, and expand only when the data backs it.

Start with your safest, most clearly simple intent category. Run it on fast for two weeks while keeping the same intent on standard in a control group. Compare resolution rates, transfer rates, and conversation scores between the two groups. If they're equivalent, expand to the next category.

// Week 1-2: Only route one low-risk intent
const ROUTING_ENABLED = new Set<string>([
  "hours_inquiry",    // lowest-stakes, highest-volume FAQ
]);
 
// Week 3-4: Add after verifying parity
// "location_inquiry",
// "return_policy_lookup",
 
// Week 5+: Continue expanding with data to justify each addition
// "order_status_simple",
 
function stageRouter(intent: string, flags: FeatureFlags): ModelTier {
  if (flags.routingEnabled && ROUTING_ENABLED.has(intent)) {
    return "fast";
  }
  return "standard";
}

This approach takes longer, but the cost is time, not quality. Your customers aren't the canary.

What Routing Doesn't Fix

Routing optimizes cost per request. It doesn't fix:

A poorly prompted agent that confuses customers regardless of which model answers. If your standard-tier agent has a 35% transfer rate, routing it to Haiku for simple queries will give you a cheaper 35% transfer rate on those queries, not a better experience.

Context rot in long conversations, where accumulated turns degrade the model's ability to track the thread. Routing helps on fresh single-turn queries, but once a conversation is several turns deep with a complex history, the routing decision matters less than context management. For deeper reading on that problem, the context engineering article covers it in detail.

Tool failures where the model calls the wrong API with incorrect parameters. A capable and a cheap model can both fail to call your booking system correctly if the tool description is wrong or ambiguous.

Fix quality first, then optimize cost. A cheaper bad agent is still a bad agent.

Get routing right, and that $40k bill from the top of this article turns into something closer to $15k without a single customer noticing the difference. The routing layer is part of your agent's infrastructure, the "Build" layer that decides how much each call costs before it reaches the customer. The "Monitor" layer is what tells you whether routing is actually working. Without per-tier quality tracking, routing is cost optimization with your eyes closed.

The Signal Briefing

Un email por semana. Cómo los equipos líderes de CS, ingresos e IA están convirtiendo conversaciones en decisiones. Benchmarks, playbooks y lo que funciona en producción.

Subscribe

500+ líderes de CS e ingresos suscritos