Extended thinking for CX agents: when slow beats fast

Your customer just asked your agent a simple question: "When does my order arrive?"

Your agent has the answer. It's in the tool response from get_order_status, retrieved in 200ms. But your agent is running with extended thinking enabled. It's working through the problem. Is there a delay? Are there exceptions? What if the carrier data conflicts with the warehouse record? Should it mention the return window?

Four seconds later, the customer gets a paragraph about shipping timelines.

You just paid for 3,800 reasoning tokens to answer a factual lookup question. And your customer, on the phone or in a chat window, felt that pause.

This is the reasoning model tradeoff, and getting it right for production CX agents is more nuanced than most teams expect.

What reasoning models actually add

Extended thinking gives a model the ability to reason through problems before committing to a response. Instead of the model going directly from input to output, it generates an internal scratchpad, a chain of reasoning steps that explores the problem, checks constraints, considers edge cases, and builds toward a conclusion.

The improvement is real and documented. On tasks involving multi-step logic, complex eligibility rules, ambiguous intent, or decisions that require holding several competing constraints in mind simultaneously, reasoning models consistently outperform standard models. For the right task class, extended thinking is the difference between an agent that gets it right first and one that confidently gets it wrong.

The cost is also real. Thinking tokens are generated sequentially before any output appears. A 4,000-token thinking budget adds 3-5 seconds of latency before your user sees anything. In chat, that's a visible delay. In voice, it's a dead air gap that sounds like the call dropped.

The challenge for CX teams is that both the benefit and the cost are real. The answer isn't "always use extended thinking" or "never use it." The answer is a classification problem: which tasks in your agent's repertoire actually need it?

The CX latency budget you're working with

Human conversation has natural timing expectations. In chat, response times under 1.5 seconds feel responsive. Between 1.5 and 3 seconds feel slightly slow but acceptable. Above 3 seconds, users start typing again, second-guessing whether their message went through, or abandoning the conversation.

Voice is stricter. Natural conversational turn-taking expects a response within about 800ms of a speaker stopping. Beyond that, the listener starts to wonder if the line is silent. Your voice AI pipeline already spends 200-300ms on STT, another 100-200ms on network, and another 100-200ms on TTS. That leaves you roughly 300-500ms for your agent's inference call before your latency budget is exhausted.

Extended thinking on a 4,000-token budget takes that 300ms inference call and turns it into a 4,000ms one. That's ten times over your latency budget, before the customer hears anything.

The voice AI latency pipeline article breaks down exactly where those milliseconds go. The point here is simpler: for voice, extended thinking is almost never the right default. For chat, it depends entirely on the task.

Task classification: where extended thinking earns its cost

The first step toward hybrid routing is understanding which tasks in your agent's distribution actually benefit from extended thinking. The honest answer is: fewer than you'd think.

Tasks that benefit from extended thinking:

Complex eligibility and policy decisions. When a customer asks "am I eligible for a refund?" the answer isn't a simple lookup. It involves checking the purchase date against the return window, the item category against the exception list, the customer tier against the policy overrides, and potentially the carrier's fault assignment from the shipping dispute. Getting this wrong means either an unhappy customer who should have gotten a refund or an incorrect refund that costs the business. Extended thinking helps an agent work through layered conditions correctly.

Ambiguous intent where mis-routing is costly. "I want to cancel" could mean cancel an order, cancel a subscription, or cancel an in-progress return. A fast model will pick the most likely interpretation and proceed. A reasoning model can work through the conversation context: what did the customer say, what did they do in the last three sessions, what does the history suggest. The cost of mis-routing here (a cancelled subscription when they meant to cancel an order) justifies the latency.

Multi-step dispute resolution. When a customer has a billing dispute involving credits, adjustments, a previous partial refund, and a tier-based discount calculation, getting the right answer requires arithmetic and constraint-checking across multiple records. Extended thinking handles this class of task well.

Tasks that should not use extended thinking:

Status queries. Order status, shipment tracking, account balance, subscription renewal date. These are answered by a tool call, not by reasoning. Extended thinking adds latency without adding accuracy.

FAQ responses. "What is your return policy?" "Do you ship to Canada?" "Is there a warranty?" Your knowledge base has these answers. Retrieving and returning them correctly doesn't require multi-step reasoning.

Simple action execution. "Update my address." "Add item to cart." "Set a reminder for my renewal." These are single tool calls. The agent's job is to parse the request, confirm with the customer if needed, and execute. Reasoning doesn't improve this.

First-response classification. When a customer first contacts your agent, the initial classification ("this is a billing question" vs "this is a shipping question") should be fast. Slow classification delays everything downstream. Route to extended thinking after you know what you're dealing with, not before.

Building the hybrid routing pattern

Hybrid routing gives you reasoning model quality on the tasks that need it, and fast inference speed on the tasks that don't. Here's how to implement it.

The core idea is a two-stage pipeline. An intake classifier evaluates the incoming request and routes it to either a fast path or a thinking path. The classifier itself should be fast and cheap, which makes it a good use case for a smaller, faster model.

import Anthropic from '@anthropic-ai/sdk';
 
interface TaskClassification {
  path: 'fast' | 'thinking';
  thinkingBudget?: number;
  reason: string;
}
 
async function classifyTask(
  query: string,
  context: ConversationContext
): Promise<TaskClassification> {
  const classifier = new Anthropic();
 
  const response = await classifier.messages.create({
    model: 'claude-haiku-4-5-20251001',
    max_tokens: 200,
    messages: [{
      role: 'user',
      content: `Classify this customer query for routing:
 
Query: "${query}"
Previous turns: ${context.turnCount}
Topic signals: ${context.topics.join(', ')}
 
Return JSON with: path ("fast" or "thinking"), thinkingBudget (null or 1000-8000), reason (one sentence).
 
Fast path: status checks, FAQ, simple actions, greetings.
Thinking path: eligibility decisions, disputes, ambiguous intent, multi-condition policy questions.`
    }],
  });
 
  return JSON.parse(response.content[0].type === 'text' 
    ? response.content[0].text 
    : '{"path":"fast","reason":"parse error"}');
}
 
async function routeToAgent(
  query: string,
  context: ConversationContext
): Promise<AgentResponse> {
  const classification = await classifyTask(query, context);
 
  if (classification.path === 'thinking') {
    return runWithExtendedThinking(query, context, classification.thinkingBudget ?? 3000);
  }
 
  return runFastInference(query, context);
}

The classifier call takes about 200ms. That's your overhead for the routing decision. For fast-path queries, you're still under 500ms total. For thinking-path queries, you're adding deliberate latency on tasks where it's justified.

async function runWithExtendedThinking(
  query: string,
  context: ConversationContext,
  thinkingBudget: number
): Promise<AgentResponse> {
  const client = new Anthropic();
 
  const response = await client.messages.create({
    model: 'claude-opus-4-8',
    max_tokens: thinkingBudget + 2000,
    thinking: {
      type: 'enabled',
      budget_tokens: thinkingBudget,
    },
    messages: buildMessages(query, context),
    tools: getCxTools(context.agentScope),
  });
 
  const thinkingBlock = response.content.find(b => b.type === 'thinking');
  const textBlock = response.content.find(b => b.type === 'text');
 
  return {
    content: textBlock?.text ?? '',
    thinkingTokensUsed: thinkingBlock ? thinkingBudget : 0,
    path: 'thinking',
  };
}

A note on budget sizing: start at 2,000 tokens for most CX reasoning tasks. Run your evaluation set and compare accuracy at 1,000, 2,000, 4,000, and 8,000 tokens. Most teams find that accuracy plateaus between 3,000 and 5,000 tokens for CX-class problems. Budgets above that threshold add latency without improving outcomes.

Setting the right budget by task type

Not all thinking-path tasks need the same budget. A rough starting point:

These are starting points, not fixed values. Calibrate against your actual conversation distribution. The right budget is the smallest one that gets your target accuracy on your eval set for that task class.

Monitoring thinking in production

Once you've deployed hybrid routing, you need visibility into whether thinking is earning its latency cost. The metrics that matter most:

Thinking token utilization rate. What fraction of your agent's conversations trigger the thinking path? If it's over 40%, your classifier is over-indexing on complexity. If it's under 5%, your classifier may be under-routing tasks that would benefit from reasoning.

Latency by path. Track P50, P90, and P99 separately for fast-path and thinking-path conversations. Your fast path should have P99 under 1.5 seconds. Your thinking path P50 should be under 5 seconds.

Thinking budget vs. outcome. Log both the allocated budget and the tokens actually used. If your agent consistently uses only 40% of an 8,000-token budget, you're allocating too much and paying more than necessary. If it consistently hits the ceiling, you may need a higher budget for that task class.

The Chanl analytics dashboard surfaces these per-session metrics automatically for agents built on the Chanl platform. The key insight is to segment by task type. An aggregate "average latency" masks the bimodal distribution between fast-path and thinking-path conversations.

Quality by path is where agent monitoring earns its value in this context. If thinking-path conversations don't score higher on accuracy and policy compliance in your scorecard evaluations, your routing criteria probably need recalibration. The thinking path should consistently outperform on complex tasks, or you're paying latency cost for no quality gain.

const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
await chanl.calls.logMetrics({
  sessionId: context.sessionId,
  routing: {
    path: response.path,
    classifierLatencyMs: classificationTime,
    thinkingTokensAllocated: thinkingBudget,
    thinkingTokensUsed: response.thinkingTokensUsed,
    totalLatencyMs: totalTime,
  },
  quality: {
    taskType: classification.reason,
    completedSuccessfully: !response.escalated,
  },
});

Where extended thinking breaks down

There are three failure modes worth knowing before you ship.

Overthinking simple requests. When a classifier incorrectly routes a simple query to the thinking path, the agent sometimes produces an overthought response. Instead of "Your order ships Thursday, October 15th," you get a paragraph about what Thursday means for delivery expectations, caveats about carrier delays, and a suggestion to check the tracking link. The answer is still correct, but the user asked a simple question and got a complex response. Monitor for response length inflation on thinking-path queries.

Thinking that contradicts the tool result. Extended thinking occasionally produces reasoning that disagrees with what the tool call actually returned. The model thinks through the problem and arrives at an answer, then the tool call returns a different one. Agents that are poorly prompted to trust tool results over their own reasoning will sometimes produce hybrid responses that blend both. Make your tool results authoritative: the reasoning step should plan which tools to call and in what order, not pre-calculate what the answer will be.

Budget exhaustion on time-constrained tasks. If you set a 6,000-token budget on a task that needs to respond in under 3 seconds, you'll hit your latency SLA before the budget runs out. Build timeout handling that returns a partial response or routes to human escalation if the thinking step exceeds your latency budget, rather than blocking the response entirely.

The broader principle

Reasoning models are a capability upgrade for tasks that genuinely require reasoning. The mistake is treating them as an upgrade for your agent overall. Swapping your fast model for a reasoning model across the board and expecting uniform improvement is how you end up with an agent that takes four seconds to confirm an order status.

The token cost optimization article has more on the cost side of this. The same principle applies to latency: optimize the path, not the model.

The teams getting the most value from extended thinking in 2026 are using it as a specialized capability, invoked deliberately on the task classes where it matters. Your CX agent will handle hundreds of conversation types. Most of them don't require extended thinking. A handful do, and those are exactly the ones where getting it right the first time is most valuable.

Classify first. Think only when thinking helps.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed