Prompt caching: the cost cut most agent teams skip

Your CX agent handles 10,000 conversations a day. Before answering a single question, it sends the same 4,000-token system prompt every time (the agent's name, its instructions, 30 tool definitions, a policy excerpt from your knowledge base). At that volume, you're re-processing 40 million tokens daily just to tell the model who it is. Most of that cost disappears once you add a cache breakpoint.

Prompt caching is one of the simplest performance wins available to production agent teams, and one of the most commonly skipped. A 2026 study on long-horizon agentic tasks found it cuts API costs by 41-80% and trims time-to-first-token (TTFT) by 13-31%. This article walks through how it works, where to put the cache boundary, when it backfires, and how to monitor hit rates so you know the cache is actually doing its job.

How prompt caching works

Prompt caching stores a snapshot of your context prefix on the API server. When you mark a cache breakpoint in your messages, the server saves everything up to that point after the first request. Subsequent requests that share the same prefix read from that snapshot instead of re-processing the tokens. You pay a cache-read price (roughly 10x cheaper than standard input pricing on Claude) and get a faster first token back.

The cache is keyed on exact byte content. If even one token before the breakpoint changes between requests (a timestamp, a dynamic placeholder, a session ID), you miss the cache and pay full input-token price. That's the core constraint: everything before your breakpoint must be byte-for-byte identical across calls.

Cache entries don't live forever. On Anthropic's API, a cache entry has a 5-minute TTL that refreshes on each hit. A CX agent handling back-to-back turns within a conversation keeps the cache warm indefinitely because each turn resets the clock. A background workflow with long pauses between calls will see more misses. Designing around this TTL is part of getting the economics right.

The API tells you when a hit happened. The response includes cache_read_input_tokens (tokens served from cache) and cache_creation_input_tokens (tokens that created or replenished the cache). Tracking these two numbers is how you verify the cache is working.

import Anthropic from "@anthropic-ai/sdk";
 
const client = new Anthropic();
 
const response = await client.messages.create({
  model: "claude-opus-4-8",
  max_tokens: 1024,
  system: [
    {
      type: "text",
      text: SYSTEM_PROMPT,
      cache_control: { type: "ephemeral" }, // mark the cache breakpoint
    },
  ],
  messages: conversationHistory,
});
 
const { cache_read_input_tokens, cache_creation_input_tokens, input_tokens } =
  response.usage;
 
console.log(`Regular tokens: ${input_tokens}`);
console.log(`Cache hit tokens: ${cache_read_input_tokens}`);
console.log(`Cache creation tokens: ${cache_creation_input_tokens}`);

Where to put the cache boundary

The right place for your cache boundary is immediately after the last piece of content that stays identical between requests. For most CX agents, that's the end of your tool definitions.

A typical agent context has three zones:

1. System prompt -- instructions, persona, policies. Stable within and across sessions.
2. Tool definitions -- schemas for every tool the agent can call. Changes only on deployment, never mid-session.
3. Conversation history and tool results -- grows every turn. Always different.

Zones 1 and 2 belong before the cache breakpoint. Zone 3 stays after it.

const response = await client.messages.create({
  model: "claude-opus-4-8",
  max_tokens: 1024,
  system: [
    {
      type: "text",
      text: SYSTEM_PROMPT,
    },
    {
      type: "text",
      text: TOOL_USAGE_INSTRUCTIONS, // last static element
      cache_control: { type: "ephemeral" }, // breakpoint goes here
    },
  ],
  tools: TOOL_DEFINITIONS, // tool schemas in the tools array also benefit from caching
  messages: conversationHistory, // dynamic tail -- never cached
});

If your agent does a RAG pass at session startup (fetching a customer's recent orders or a policy document before the first turn), those chunks can also go before the breakpoint, as long as they don't change mid-session. A startup knowledge injection that stays fixed for the conversation lifetime is cheap to include in the static prefix.

The question to ask about any piece of context: "Will this content be byte-for-byte identical on the next request within this session?" If yes, put it before the breakpoint. If not, leave it after.

When caching makes things worse

Caching the wrong content is worse than not caching at all. Here's the failure mode to avoid.

If you put the breakpoint at the end of your full messages array (after conversation history, after tool results), you're caching a different snapshot every turn. Turn 1 caches a 3,000-token prefix. Turn 2 adds a new assistant message and a tool result, then tries to match the turn-1 cache. It misses. Now you've paid for cache creation on turn 1 AND a full input-token charge on turn 2. You end up paying more than without caching.

This is what the 2026 research called "naive full context caching." It's the reason some teams turned caching on, saw costs go up, and turned it off again.

Cache creation carries a small overhead because the server has to store the snapshot. If you're creating a new cache entry every turn instead of hitting one, that overhead compounds. The fix is simple: move the breakpoint to a stable position.

Tool definitions are the hidden token cost

MCP-connected agents tend to carry large tool schemas. A tool schema for a CRM integration might be 300-500 tokens. An agent with 30 tools runs to 9,000-15,000 tokens of tool definitions alone, before the system prompt.

Without caching, those definitions go out on every single request. At 15,000 tokens of tools, $15 per million input token pricing, and 500 turns a day, that's $112.50 per day in tool schema re-processing for a moderately busy agent. With a cache hit rate of 80%, that drops to roughly $22.50 per day. A $90 per day saving from one configuration change.

If you're not sure which tools your agent is loading or how many tokens they cost, Chanl's Tools dashboard shows the schema size for each registered tool and flags which ones contribute most to per-turn token cost. Tools you're loading but rarely calling are both a cost problem and a cache hit rate problem because they inflate your static prefix without adding value.

Caching injected knowledge base chunks

Some agents inject knowledge base content at session start (a policy document, a customer's purchase history, the current product catalog). If that content is fetched once and stays fixed for the session, it belongs in the static prefix.

The pattern is to run your retrieval pass before the first message turn, then construct the system prompt with the retrieved content included. Mark the cache breakpoint after this combined block.

async function startSession(customerId: string) {
  // Fetch at session start, before any conversation turns
  const customerProfile = await fetchCustomerProfile(customerId);
  const relevantPolicies = await fetchPolicies(customerProfile.tier);
 
  const STATIC_PREFIX = `
${BASE_SYSTEM_PROMPT}
 
## Customer context
Name: ${customerProfile.name}
Tier: ${customerProfile.tier}
Lifetime orders: ${customerProfile.orderCount}
 
## Relevant policies
${relevantPolicies.text}
  `.trim();
 
  return {
    system: [
      {
        type: "text" as const,
        text: STATIC_PREFIX,
        cache_control: { type: "ephemeral" as const }, // cache the entire startup block
      },
    ],
  };
}

The customer profile changes between sessions but stays constant within one conversation, which makes it safe to cache for the session lifetime. You'd re-run the startup pass at the beginning of each new conversation and get a fresh cache entry for the new customer context.

Monitoring cache hit rates

Turning caching on isn't enough. You need to verify it's working and catch when it breaks.

Two things go wrong silently: the breakpoint ends up in the wrong position (dynamic content slipping before it), or the session pattern breaks the TTL (calls arriving more than 5 minutes apart so the cache expires between turns). Both show up as low hit rates in your usage metadata.

interface CacheStats {
  hitRate: number;
  hitTokens: number;
  missTokens: number;
  estimatedSavingsUSD: number;
}
 
function computeCacheStats(usage: {
  cache_read_input_tokens?: number;
  cache_creation_input_tokens?: number;
}): CacheStats {
  const hitTokens = usage.cache_read_input_tokens ?? 0;
  const missTokens = usage.cache_creation_input_tokens ?? 0;
  const total = hitTokens + missTokens;
  const hitRate = total > 0 ? hitTokens / total : 0;
 
  // Claude pricing: $1.50/M cache reads vs $15/M regular input
  const savingsPerToken = 0.000015 - 0.0000015;
  const estimatedSavingsUSD = hitTokens * savingsPerToken;
 
  return { hitRate, hitTokens, missTokens, estimatedSavingsUSD };
}
 
// Emit this metric on every agent turn
const stats = computeCacheStats(response.usage);
if (stats.hitRate < 0.7) {
  console.warn(`Cache hit rate below threshold: ${(stats.hitRate * 100).toFixed(1)}%`);
}

A healthy CX agent in back-to-back conversation mode should hit above 80%. Rates below 70% usually point to one of three root causes:

Dynamic content leaking into the static prefix. Any timestamp, user ID, or session value injected into your system prompt text busts every cache hit. Even a Date.now() call inside the string template is enough to invalidate the entire prefix.

Tool definition changes between requests. If your tool schemas are generated dynamically (with varying field ordering or generated descriptions), they won't cache-hit even if the content is semantically identical. Serialize them deterministically and store the result as a module-level constant.

Session gaps exceeding the TTL. If your agent handles async tasks with pauses longer than 5 minutes between turns, the cache expires. Consider sending a no-op message at session start to seed the cache, then managing keep-alive if the workflow is async.

The Chanl Analytics dashboard tracks per-session token breakdown including cache hit and miss counts. You can set an alert that fires when hit rate drops below a threshold across a rolling window, which usually catches deployment bugs (like a code change that accidentally started injecting dynamic content into the system prompt) before the cost spike shows up in your billing report.

Caching across a full multi-turn conversation

In multi-turn conversations, the cache stays effective as long as you keep the breakpoint in the same position every turn, even as conversation history grows. The system prompt sits in the system parameter with its cache marker. The conversation history lives in the messages array. These are separate API fields, so the cache boundary between them is always clean.

class CachedAgentSession {
  private history: Anthropic.MessageParam[] = [];
  private systemConfig: Anthropic.TextBlockParam[];
 
  constructor(
    private client: Anthropic,
    systemPrompt: string
  ) {
    // Build the static system config once per session
    this.systemConfig = [
      {
        type: "text",
        text: systemPrompt,
        cache_control: { type: "ephemeral" },
      },
    ];
  }
 
  async send(userMessage: string): Promise<string> {
    this.history.push({ role: "user", content: userMessage });
 
    const response = await this.client.messages.create({
      model: "claude-opus-4-8",
      max_tokens: 1024,
      system: this.systemConfig, // same every turn -- always hits cache after turn 1
      messages: this.history,   // grows every turn -- always processed fresh
    });
 
    const assistantText =
      response.content[0].type === "text" ? response.content[0].text : "";
 
    this.history.push({ role: "assistant", content: assistantText });
    return assistantText;
  }
}

Turn 1 pays full price and creates the cache entry. Turns 2, 3, 4, and onward all read from it. As this.history grows, the dynamic tail gets longer, but the static prefix remains identical and the cache keeps hitting.

How OpenAI's automatic caching differs

OpenAI also offers prompt caching, but the implementation is automatic rather than explicit. You don't add a cache marker -- the API automatically caches your context prefix based on length and reuse patterns. This is convenient but less controllable.

The practical difference for agent teams: Claude's explicit cache_control markers let you choose exactly where the cache boundary sits. OpenAI's automatic caching uses a 128-token granularity and applies to prefixes above 1,024 tokens. If you have dynamic content mixed into the first 1,024 tokens of your context, it won't cache, and you won't know why from the usage metadata alone.

For CX agents with large, reusable system prompts, the explicit control is worth using. It also gives you the usage metadata to verify the cache is working, which the automatic approach doesn't always surface as clearly.

Monitoring with the Chanl SDK

If you're using Chanl to run agents, you can pull per-session cache metrics through the analytics client and track which sessions are getting the benefit.

import { Chanl } from "@chanl/sdk";
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
const metrics = await chanl.calls.getMetrics({
  timeRange: "24h",
  groupBy: "session",
  fields: ["cache_hit_rate", "cache_tokens_saved", "total_input_tokens", "cost_usd"],
});
 
const poorCacheSessions = metrics.sessions.filter(
  (s) => s.cache_hit_rate < 0.7
);
 
console.log(`Sessions with cache hit rate below 70%: ${poorCacheSessions.length}`);
console.log(`Total cache savings today: $${metrics.totals.cache_tokens_saved_usd.toFixed(2)}`);

This surfaces in Chanl's Monitoring dashboard as an anomaly when a deployment change breaks your cache configuration. Set it up as an alert so it fires before the cost spike reaches your billing report.

Putting it together

Back to that 10,000-conversation-a-day agent: at 4,000 tokens of static context, 80% cache hit rate, and $15/M input pricing, caching saves roughly $4,320 per month in re-processed system prompt tokens alone. That's before accounting for tool definitions. For most teams, this is the highest return-per-hour optimization available -- one cache breakpoint, one monitoring metric, done.

The discipline is in the monitoring. Cache hit rate is the one number that tells you whether your configuration is actually working, and it's the number most teams don't track. A single code change (a developer adding a Date.now() to the system prompt for a debug log) will silently crater your hit rate until someone notices. Add it to your dashboards alongside TTFT and cost, and set an alert when it drops below 70%.

For the next layer of latency optimization, speculative tool calling covers pre-fetching tool results the agent is likely to need before it explicitly requests them. And token cost optimization covers the broader picture of where tokens go in a production agent across the full turn lifecycle.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed