Your agent re-reads its own manual on every call

The engineer opened the traces because the bill had spiked. She expected to find a token-hungry embedding model or an oversized RAG retrieval. What she found instead was the same 18,000-token block appearing on every single request. The agent's system prompt. The instructions it already had from the last call, and the call before that, and every call this month.

Nobody had touched the system prompt in weeks. It wasn't growing. It was just showing up every time, in full, and the model was processing it from scratch every time, because nobody had turned on caching.

Datadog's 2026 State of AI Engineering report found this is not unusual. On average, 69% of all input tokens across production AI systems go to system prompts. And only 28% of LLM calls use any form of cached-read tokens. The gap between those two numbers is where most agent budgets bleed out.

This article is about closing that gap: understanding what's in your system prompt, why it stays there, how to fix it without breaking your agent, and how to make sure the fix actually holds in production.

Why your system prompt is 20,000 tokens

A typical production CX agent accumulates roughly 18,000-22,000 tokens in its system prompt through four additive layers: the persona and constraints (~1,200 tokens), the full tool list with descriptions (~8,400 tokens), operational procedures (~4,100 tokens), and embedded domain knowledge (~3,800 tokens). Each layer made sense when someone added it. None of them re-evaluate themselves.

System prompt bloat comes in layers, and most of them were sensible decisions at the time.

The foundation is the persona and core instructions: who the agent is, what it should and shouldn't do, how to handle escalations. Maybe 1,000 tokens, usually fine.

On top of that sits the tool list. In a non-trivial agent, you might have 10-30 tools, each with a name, description, parameter schema, and example usage. Tool descriptions are prompts -- they're the primary signal the model uses to select the right function call. Every word matters. But at 300-500 tokens per tool, 20 tools puts you at 6,000-10,000 tokens before you've written a single instruction.

Then come the operational procedures. The things that don't fit as tool descriptions because they're conditional: "when the customer says they want to cancel, first offer the pause option, and only then proceed to the cancellation flow." These are legitimate business rules, and they end up in the system prompt because there's nowhere else to put them.

Then the domain knowledge. The product catalog summary. The current pricing tiers. The list of integrations the customer might ask about. Often another 3,000-5,000 tokens.

System prompt anatomy for a typical CX support agent:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Core persona + instructions    ~1,200 tokens
  Tool definitions (22 tools)    ~8,400 tokens
  Operational procedures         ~4,100 tokens
  Domain knowledge               ~3,800 tokens
  Edge case handling             ~1,600 tokens
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Total                         ~19,100 tokens
 
In a 15-step agentic task, this gets sent 15 times.
Uncached cost: 15 x 19,100 = 286,500 input tokens per task.
Cached cost:  1x write + 14x read at 10% = ~45,740 effective tokens.

The math isn't subtle. A 15-step task that uses caching effectively consumes roughly one-sixth the input tokens of the same task without caching. At $3 per million input tokens (Sonnet pricing), that's the difference between $0.86 and $0.14 per task. Across ten thousand customer conversations, it's the difference between $8,600 and $1,400.

How prompt caching works and why 72% of agents skip it

Prompt caching works by letting the model provider cache the processed key-value representation of a fixed prefix. On the first request, you pay full price to compute it. On subsequent requests with the same prefix, the cached version loads in milliseconds and you pay a fraction of the original cost.

For Anthropic's Claude, cached input tokens cost 90% less than uncached. OpenAI's cached input tokens cost 50% less. Google's Gemini has its own caching tiers. The exact numbers vary, but the order of magnitude is consistent: caching cuts input token costs by 50-90%.

The catch is that the cached prefix must be bit-for-bit identical across requests. Any change invalidates the cache and forces a full rewrite. This is why 72% of agents don't use it -- not because they don't know caching exists, but because their system prompt changes in ways they didn't design for.

import Anthropic from "@anthropic-ai/sdk";
 
const client = new Anthropic();
 
// The system prompt must be static to be cached.
// Any dynamic content must live AFTER the cache boundary.
const SYSTEM_PROMPT = `You are a customer support agent for Acme Corp.
Your tools allow you to look up orders, check account status, and process refunds.
Follow these policies when handling requests:
[... 19,000 tokens of stable content ...]`;
 
async function runAgentStep(userMessage: string, conversationHistory: Anthropic.MessageParam[]) {
  const response = await client.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 4096,
    system: [
      {
        type: "text",
        text: SYSTEM_PROMPT,
        cache_control: { type: "ephemeral" }, // Mark this for caching
      },
    ],
    messages: [
      ...conversationHistory,
      { role: "user", content: userMessage },
    ],
  });
 
  // Check whether the cache was hit
  const usage = response.usage;
  console.log({
    inputTokens: usage.input_tokens,
    cacheWriteTokens: usage.cache_creation_input_tokens,
    cacheReadTokens: usage.cache_read_input_tokens, // These cost 10% of normal
  });
 
  return response;
}

The cache_read_input_tokens field tells you exactly how much of each request came from cache. If that number is zero on requests past the first one, your cache is being invalidated somewhere.

Finding what's breaking your cache

Cache invalidation has four common culprits in production agents. Check them in this order because the first two account for most cases.

Timestamps and session IDs in the system prompt. The most common cause. Someone added "Current date: " to help the agent know what day it is. Now every request has a different prefix and the cache never warms up. Move time-sensitive content to the user message or a separate user-turn injection after the cached system prompt.

Dynamic tool lists. If your agent enables or disables tools based on customer tier or session state, and that list lives in the system prompt, the prompt changes with every routing decision. Fix: define the full tool set in the cached system prompt and handle authorization at the tool execution layer, not the tool discovery layer.

A/B testing without cache buckets. If you're running a prompt experiment and both variants share the same cache, neither gets a warm cache because they write different content to the same prefix position. Use separate cache keys per variant, or route each variant through a separate Chanl prompt slot with its own cache state.

Tool descriptions that drift. As we covered in the MCP tool description drift article, even small edits to tool descriptions change the prompt byte-for-byte and invalidate the cache. Gate tool description changes with a CI check that forces cache warmup measurement on the new version before merging.

import { Chanl } from "@chanl/sdk";
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY! });
 
async function measureCacheHealth(agentId: string, windowHours = 24) {
  const metrics = await chanl.calls.getMetrics({
    agentId,
    windowHours,
    includeTokenBreakdown: true,
  });
 
  const cacheHitRate =
    metrics.totalCacheReadTokens /
    (metrics.totalCacheReadTokens + metrics.totalCacheWriteTokens + metrics.totalUncachedTokens);
 
  const systemPromptRatio =
    metrics.averageSystemPromptTokens / metrics.averageInputTokens;
 
  return {
    cacheHitRate: `${(cacheHitRate * 100).toFixed(1)}%`,
    systemPromptRatio: `${(systemPromptRatio * 100).toFixed(1)}%`,
    estimatedMonthlySavingsIfCached: `$${(
      metrics.totalUncachedTokens * 0.000003 * 0.9
    ).toFixed(2)}`,
    cacheInvalidationEvents: metrics.cacheInvalidations,
  };
}
 
// In practice you'd schedule this and alert on cache hit rate below 70%
const health = await measureCacheHealth("cx-support-agent");
console.log(health);
// { cacheHitRate: '31.2%', systemPromptRatio: '71.4%',
//   estimatedMonthlySavingsIfCached: '$2,140.00',
//   cacheInvalidationEvents: 847 }

Trimming the system prompt before you cache it

Caching a bloated system prompt is better than not caching, but the right sequence is trim first, then cache. Every token you remove from the static system prompt is a token you never pay for, cached or not.

Audit your system prompt in three passes.

Pass 1: What's always needed? The agent's persona, core behavioral constraints, and the tool list -- these have to be present on every request. Don't touch them in this pass.

Pass 2: What's sometimes needed? Operational procedures that only apply to specific scenarios ("if the order is more than 90 days old..."), domain knowledge about product lines the customer didn't ask about, escalation scripts for situations that arise in maybe 5% of conversations. These are candidates for dynamic loading.

Pass 3: What's never needed? Outdated edge case handling from a previous product version, redundant restatements of the same constraint, and copy-pasted documentation that exists at the tool description level too. Delete these.

// Instead of embedding all procedures in the system prompt,
// load relevant ones based on conversation context
async function loadRelevantProcedures(conversationContext: string): Promise<string> {
  const procedures = [
    { trigger: "cancel", content: await import("./procedures/cancellation.txt") },
    { trigger: "refund", content: await import("./procedures/refund-policy.txt") },
    { trigger: "upgrade", content: await import("./procedures/upgrade-flow.txt") },
    { trigger: "billing", content: await import("./procedures/billing-disputes.txt") },
  ];
 
  // Simple keyword match; in production use embeddings for better recall
  const relevant = procedures.filter((p) =>
    conversationContext.toLowerCase().includes(p.trigger),
  );
 
  if (relevant.length === 0) return "";
 
  return `\n\nRelevant procedures for this conversation:\n${
    relevant.map((p) => p.content.default).join("\n\n")
  }`;
}
 
async function buildAgentMessages(
  staticSystemPrompt: string, // Cached portion
  userMessage: string,
  conversationHistory: { role: string; content: string }[],
): Promise<{ system: string; messages: { role: string; content: string }[] }> {
  // Dynamic procedures go into a user-turn injection, not the system prompt
  const procedures = await loadRelevantProcedures(
    conversationHistory.map((m) => m.content).join(" "),
  );
 
  return {
    system: staticSystemPrompt, // This gets cached
    messages: [
      ...conversationHistory,
      ...(procedures
        ? [{ role: "user", content: `[Context for this turn: ${procedures}]` }]
        : []),
      { role: "user", content: userMessage },
    ],
  };
}

Moving procedures to dynamic user-turn injections has two benefits. The system prompt stays cacheable, and you can update procedures without invalidating the cache. The agent gets the same guidance; you pay for it only when it's relevant.

The rate limit problem hiding inside the token problem

Token waste and rate limits are the same problem in two different invoices. When your agent sends 20,000 uncached tokens per request, it burns through token-per-minute quota at 6x the rate of a properly cached agent. Fewer requests fit inside the quota, and the ones that don't get 429 errors.

Datadog's report surfaced this clearly: 5% of LLM call spans failed in February 2026, and rate limits were the dominant failure mode. Most of those teams weren't running too many agents. They were running the same number of agents but sending far more tokens per request than necessary.

When your agents send 20,000-token system prompts on every request without caching, you're burning through your token-per-minute quota faster than necessary. The more tokens per request, the fewer concurrent requests your quota supports. Teams that hit rate limits at scale aren't always running too many agents -- sometimes they're running the same number of agents but sending 5-10x more tokens than they need to.

Rate limit errors also compound in multi-step agents. If step 7 of a 15-step task hits a rate limit, the agent either fails or waits and retries. Both outcomes affect the customer experience. The cheapest retry is the one that never happens because your token budget per request was manageable.

The fix is the same as the cost fix: cache the system prompt, trim what doesn't need to be there, and move dynamic content to user-turn injections. The quota reduction is a side effect you get for free.

Monitoring token health in production

Setting up caching and trimming the system prompt solves the problem once. Keeping it solved requires monitoring that catches when cache invalidation creeps back in.

Three metrics to track per agent, per day:

Cache hit rate. Should be above 70% for any agent with a multi-step task pattern. Below 50% means something is changing the cached prefix on more than half your requests. Alert on it.

System prompt token ratio. Track system_prompt_tokens / total_input_tokens as a rolling average. If it rises above 75%, either your system prompt is growing or your user-turn context is shrinking (sometimes a sign that the agent is cutting conversations short). Either way, worth investigating.

Effective tokens per task. For agents that run multi-step tasks, total the input tokens across all steps and divide by task count. This is the number that should drop when you implement caching correctly. If it stays flat after enabling caching, your cache is being invalidated.

import { Chanl } from "@chanl/sdk";
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY! });
 
// Pull daily token health metrics and surface anything that needs attention
async function dailyTokenHealthReport(agentId: string) {
  const [metrics, recentCalls] = await Promise.all([
    chanl.calls.getMetrics({ agentId, windowHours: 24, includeTokenBreakdown: true }),
    chanl.calls.list({ agentId, limit: 1000, windowHours: 24 }),
  ]);
 
  // Group multi-step task calls by session to get effective tokens per task
  const sessionGroups = recentCalls.data!.reduce(
    (acc, call) => {
      const key = call.sessionId ?? call.id;
      acc[key] = (acc[key] ?? 0) + call.inputTokens;
      return acc;
    },
    {} as Record<string, number>,
  );
 
  const avgTokensPerTask =
    Object.values(sessionGroups).reduce((a, b) => a + b, 0) /
    Object.values(sessionGroups).length;
 
  const systemPromptRatio =
    metrics.averageSystemPromptTokens / metrics.averageInputTokens;
 
  const cacheHitRate =
    metrics.totalCacheReadTokens /
    Math.max(
      metrics.totalCacheReadTokens + metrics.totalCacheWriteTokens,
      1,
    );
 
  return {
    agentId,
    date: new Date().toISOString().split("T")[0],
    cacheHitRate: parseFloat((cacheHitRate * 100).toFixed(1)),
    systemPromptRatioPercent: parseFloat((systemPromptRatio * 100).toFixed(1)),
    avgTokensPerTask: Math.round(avgTokensPerTask),
    alerts: [
      ...(cacheHitRate < 0.7 ? ["Cache hit rate below 70% -- check for invalidation"] : []),
      ...(systemPromptRatio > 0.75 ? ["System prompt exceeds 75% of input tokens"] : []),
    ],
  };
}

If you're using Chanl analytics, the token breakdown and cache hit rate show up in your agent dashboard alongside conversation quality metrics. The goal is to catch a rising system prompt ratio before it shows up in your cloud bill. It's also worth running Chanl scorecards against a sample of conversations after you trim the system prompt -- lighter context occasionally changes how the agent handles edge cases, and you want to catch that before it reaches production.

What to do this week

Start with measurement, not optimization. Pull one week of traces for your highest-volume agent and calculate the three numbers: cache hit rate, system prompt token ratio, and effective tokens per task. If you haven't thought carefully about what's consuming your context window, the context window crisis article covers the broader token budget problem and is a useful frame for this audit. If all three are healthy (hit rate above 70%, system prompt below 65%, task tokens trending down), you're in good shape.

If cache hit rate is below 50%, find the invalidation source before doing anything else. Log the first 100 characters of your system prompt on every request and look for variation. Nine times out of ten it's a timestamp, a session ID, or a dynamic tool list.

If the system prompt ratio is above 70% and your hit rate is good, your prompt is too big for what it contains. Run the three-pass audit, move optional procedures to dynamic loading, and measure whether effective tokens per task drops within 48 hours.

The goal isn't to minimize the system prompt for its own sake. It's to make sure the tokens you send are earning their cost by changing the agent's behavior. A token that's being reprocessed for the 40th time this session without contributing new information is a token you're paying for twice.

The engineer who opened those traces found 18,000 tokens on every request. After caching the static prefix and moving three procedure blocks to dynamic loading, she got that down to 4,200 effective tokens per request. The conversations got faster. The bill got smaller. The agent got better at its job -- because with a tighter context, the model had less irrelevant scaffolding competing with what the customer actually said.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed