AI agents blew their 2026 budget by April. Here's the fix.

In April 2026, a team at a mid-market SaaS company discovered they had burned through their entire 2026 AI budget. Not half of it. All of it. In four months.

The culprit wasn't a rogue experiment or an accidental production deployment of an expensive model. It was production agents, running normally, handling real customer conversations, with no per-session cost cap. A small number of conversations entered retry loops. A few others hit complex multi-step tool chains that branched further than expected. None of them were stopped, because there was no mechanism to stop them.

This story is not unusual. Only 11 percent of organizations have AI agents in production, but among those that do, the majority underestimate the true cost by 40 to 60 percent. FinOps for AI is now the single most-desired skill in the discipline, precisely because nobody has yet figured out how to forecast non-deterministic costs at scale.

This article covers the four levers that stop this from happening to you: per-session token limits, prompt caching strategy for CX agents, model routing for cost tiers, and circuit breakers that stop runaway sessions before they drain the quarter.

Why agent costs are harder to predict than API costs

A REST API call has a fixed cost. You know the price before you make the call. AI agent conversations don't work like that.

The cost of a single agent conversation depends on: how many turns it takes to resolve, how many tool calls each turn spawns, how large the tool results are when they're added back to the context, whether any tool calls fail and trigger retries, how much conversation history accumulates before compression kicks in, and which model handled each turn. Every one of those is non-deterministic.

The result is that your p50 conversation cost might be $0.03, your p90 might be $0.15, and your p99 might be $2.40. That 80x spread between the median and the tail is where budgets die. If your fleet handles 10,000 conversations per day and 1 percent of them hit the p99 cost, the tail alone accounts for more than half your daily spend.

Traditional API budgeting doesn't account for this because traditional APIs don't have this property. A database query might be fast or slow, but it doesn't cost 80x more at the 99th percentile than at the 50th. Agent conversations do.

Understanding this is the first step. The next is knowing where the costs actually come from.

Where agent costs really come from

Model tokens are the most visible line item on your AI provider invoice, but they're not where most teams lose money. Studies of enterprise AI agent deployments in 2026 put model spend at under 8 percent of total cost of ownership for production systems.

The bigger cost drivers are:

Context accumulation. Every turn adds tokens to the context: user message, tool calls, tool results, model response. In a 20-turn conversation without compression, the 20th turn might be processing 5x as many input tokens as the first turn because the full history is being re-sent on every call. A simple compression strategy, summarizing older turns once they pass a threshold, cuts this significantly.

Retry overhead. When a tool call fails, agents retry. When a clarifying question doesn't get a useful response, agents rephrase and try again. Without a retry cap, a conversation that encounters repeated tool failures can cost 10x a normal conversation of the same type. The solution is both a per-turn retry limit and a total retry budget per session.

Tool result size. CRM lookups, calendar queries, and knowledge base searches can return large JSON payloads. Agents often pass the full payload back into the context even when only 10 percent of it is relevant. Preprocessing tool results to extract only what the current reasoning step needs cuts input tokens substantially, sometimes by 50 percent for data-rich tools.

System prompt inflation. System prompts grow over time as teams add policies, tool descriptions, examples, and brand guidelines. A system prompt that started at 800 tokens is often 4,000 tokens a year later, and it's re-sent on every turn of every conversation. Most teams don't notice because it's a slow accumulation.

Fixing these structural issues often reduces costs more than any optimizations to model selection or caching.

Setting per-session token limits

The most important single change you can make is adding a hard token budget to each agent session. This is the mechanism that would have stopped the April budget drain.

The implementation is simpler than it sounds:

interface SessionBudget {
  conversationType: string;
  maxInputTokens: number;
  maxOutputTokens: number;
  maxToolCalls: number;
  maxRetries: number;
}
 
const BUDGET_BY_TYPE: Record<string, SessionBudget> = {
  'faq-deflection': {
    conversationType: 'faq-deflection',
    maxInputTokens: 8_000,
    maxOutputTokens: 2_000,
    maxToolCalls: 3,
    maxRetries: 2
  },
  'appointment-booking': {
    conversationType: 'appointment-booking',
    maxInputTokens: 30_000,
    maxOutputTokens: 8_000,
    maxToolCalls: 10,
    maxRetries: 3
  },
  'refund-escalation': {
    conversationType: 'refund-escalation',
    maxInputTokens: 50_000,
    maxOutputTokens: 12_000,
    maxToolCalls: 15,
    maxRetries: 4
  }
};
 
class BudgetEnforcer {
  private used = { inputTokens: 0, outputTokens: 0, toolCalls: 0, retries: 0 };
 
  constructor(private budget: SessionBudget) {}
 
  checkBefore(estimatedInputTokens: number): 'allowed' | 'budget_exceeded' {
    if (this.used.inputTokens + estimatedInputTokens > this.budget.maxInputTokens) {
      return 'budget_exceeded';
    }
    if (this.used.toolCalls >= this.budget.maxToolCalls) {
      return 'budget_exceeded';
    }
    return 'allowed';
  }
 
  record(inputTokens: number, outputTokens: number, isRetry = false) {
    this.used.inputTokens += inputTokens;
    this.used.outputTokens += outputTokens;
    if (isRetry) this.used.retries++;
  }
 
  isOverBudget(): boolean {
    return (
      this.used.inputTokens > this.budget.maxInputTokens ||
      this.used.outputTokens > this.budget.maxOutputTokens ||
      this.used.toolCalls > this.budget.maxToolCalls ||
      this.used.retries > this.budget.maxRetries
    );
  }
}

Set the budgets based on your actual p90 cost for each conversation type, not a guess. Run a week of production traffic with detailed logging, look at the p90 token count per type, and set the limit at 1.5x to 2x that value. This catches true runaway sessions without blocking normal conversations that just run slightly long.

When a session hits its budget, the right response depends on the context. For voice agents, gracefully tell the caller you're routing them to a specialist and hand off to a human. For async agents, pause the session, log the state, and alert the on-call team. Never just drop the session without a handoff.

Prompt caching for CX agents

Prompt caching is one of the highest-leverage cost reductions available for CX agents, and most teams aren't using it correctly.

The core mechanism: the first time you send a prompt prefix to the model, the provider caches the processed key-value pairs for that prefix. On subsequent calls with the same prefix, those tokens are loaded from cache rather than reprocessed. Cache hits cost 10 to 25 percent of the normal input token price depending on the provider.

For CX agents, the system prompt is the natural cache prefix. It's the same across all conversations: same persona, same policies, same tool descriptions, same brand guidelines. A 3,000-token system prompt cached across 10,000 daily conversations saves 27 million input tokens per day.

The rule that most implementations get wrong: don't mix dynamic content into the cache prefix.

// Do this: system prompt as static cache prefix
const systemPrompt = await loadSystemPrompt('booking-agent'); // static, cached
 
const messages = [
  {
    role: 'system',
    content: systemPrompt,
    // mark for caching with your provider's syntax
    cache_control: { type: 'ephemeral' }
  },
  // Dynamic content below the cache boundary
  {
    role: 'user',
    content: buildConversationHistory(turns)  // changes every turn, not cached
  }
];
 
// Don't do this: tool results injected into the system prompt
const systemPromptWithContext = `
  ${systemPrompt}
  
  Recent tool results: ${JSON.stringify(lastToolResults)}  // breaks caching!
`;

Any content that changes between turns needs to sit below the cache boundary. Tool results, conversation history, user-specific context, real-time data: all dynamic, all below the cache prefix. The system prompt, persona definition, policy text, and static tool descriptions: all static, all above.

Studies from 2025-2026 testing across CX agent workloads show 41 to 80 percent input token cost reduction from proper caching, with a 13 to 31 percent improvement in time-to-first-token as a bonus.

One additional step that compounds the savings: trim tool descriptions in the system prompt to the minimum needed. Teams often include full OpenAPI spec excerpts in the system prompt for each tool. A 2,000-token tool description for a calendar API is almost always reducible to 400 tokens covering just the parameters and return shape the agent actually uses.

Model routing by turn complexity

Not every turn in a CX conversation requires a frontier model. A turn that acknowledges receipt of information and asks a clarifying question does not need the same model as a turn that has to reconcile conflicting tool results and make a refund decision.

The pattern is a lightweight classifier that runs before each model call and routes the turn to the appropriate model tier:

type ModelTier = 'fast' | 'standard' | 'frontier';
 
interface TurnContext {
  conversationTurn: number;
  pendingToolResults: number;
  requiresReasoning: boolean;
  isEdgeCase: boolean;
  priorTurnFailed: boolean;
}
 
function selectModelTier(context: TurnContext): ModelTier {
  // Frontier model: complex reasoning or failure recovery
  if (context.requiresReasoning || context.isEdgeCase || context.priorTurnFailed) {
    return 'frontier';
  }
 
  // Standard model: tool calls with moderate context
  if (context.pendingToolResults > 2 || context.conversationTurn > 8) {
    return 'standard';
  }
 
  // Fast model: early turns, simple acknowledgment or clarification
  return 'fast';
}
 
const MODEL_MAP: Record<ModelTier, string> = {
  fast: 'claude-haiku-4-5-20251001',
  standard: 'claude-sonnet-4-6',
  frontier: 'claude-opus-4-8'
};

The classifier itself can be hardcoded rules (as above) or a small model that evaluates turn complexity. The key insight is that requiresReasoning and isEdgeCase are detectable before the model call from the conversation state. A turn that has two conflicting tool results and a customer who escalated twice is clearly more complex than a first turn in a standard booking flow.

Teams running mixed-model routing on CX agent fleets report 40 to 60 percent cost reduction with less than 5 percent degradation in conversation completion rate. The completion rate is the right metric to watch, not accuracy on isolated turns, because model routing sometimes produces simpler responses on easy turns that still lead to successful conversation resolution.

Circuit breakers for cost

Session budgets stop individual conversations from going over. Circuit breakers stop you from discovering the problem after the fact.

A cost circuit breaker works at the fleet level, not the session level:

class FleetCostCircuitBreaker {
  private windowStart = Date.now();
  private windowCostUsd = 0;
  private state: 'closed' | 'open' | 'half-open' = 'closed';
 
  async recordConversationCost(costUsd: number, conversationType: string) {
    this.windowCostUsd += costUsd;
 
    // Alert if cost rate is 3x the expected rate for this window
    const expectedRateUsd = await this.getExpectedRate(conversationType);
    if (costUsd > expectedRateUsd * 3) {
      await this.alert({
        type: 'per_conversation_spike',
        conversationType,
        actualCostUsd: costUsd,
        expectedCostUsd: expectedRateUsd
      });
    }
 
    // Open circuit if fleet cost rate is too high
    const hourlyRateUsd = this.windowCostUsd / ((Date.now() - this.windowStart) / 3600_000);
    if (hourlyRateUsd > FLEET_HOURLY_BUDGET_USD) {
      await this.openCircuit();
    }
  }
 
  private async openCircuit() {
    this.state = 'open';
    await notify.ops({
      severity: 'critical',
      message: `Agent fleet cost circuit breaker tripped. New sessions paused.`,
      currentHourlyRateUsd: this.windowCostUsd,
      budgetUsd: FLEET_HOURLY_BUDGET_USD
    });
    // New sessions can't start until a human resets the circuit
  }
}

The circuit breaker operates at two levels: per-conversation (a single conversation costs 3x the expected amount for its type) and fleet (the hourly cost rate exceeds the fleet budget). Both levels generate alerts. The fleet-level circuit breaker pauses new sessions until a human reviews and resets it.

You want to be alerted before you're in trouble, not after. Set the fleet alert threshold at 70 percent of your hourly budget, not 100 percent. By the time you're at 100 percent, you're already over if there are conversations in flight.

Chanl's monitoring can expose per-conversation cost data with alerts, so you can see cost per agent type over time and get notified when the rate climbs. The analytics view lets you slice cost by conversation type, by day of week, and by agent configuration version, so when costs drift upward, you can trace it to a specific change: a system prompt addition, a new tool, a model change.

Building a FinOps practice for your agent fleet

FinOps for AI agents is still new enough that most teams are making it up as they go. A few practices that work well in production:

Weekly cost review by conversation type. Track median and p90 cost per type. Gradual upward drift in median cost means structural changes (system prompt growth, context accumulation). Sudden p90 spikes mean a new failure mode is appearing.

Cost attribution to product lines. If your agent fleet serves multiple product areas, allocate costs. This creates accountability and lets you compare cost-per-successful-outcome across different teams' agent configurations.

Version-controlled budget definitions. Keep your BUDGET_BY_TYPE definitions in the repo, not in a config dashboard. Changes to budgets go through code review like any other production change.

Monthly cost-quality review. Compare cost per conversation against conversation completion rate and customer satisfaction by conversation type. If the expensive conversation types are also the least successful, that's a sign that complexity is being added without corresponding quality improvement.

The article on token cost optimization goes deeper on the prompt-level changes that reduce cost without quality tradeoffs. And if you're trying to understand the unit economics of CX agents at a business level rather than a technical level, agent unit economics covers how to calculate cost per successful outcome and compare it against the cost of human handling.

The April budget drain wasn't a failure of technology. It was a failure of observability: nobody was watching per-session costs, and there was no mechanism to stop a session that was going off the rails. The fixes are all implementable in a week: session budgets, caching, model routing, and a circuit breaker. None of them require architectural changes. They just require treating agent cost as a first-class production metric, which it now clearly is.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed