Tool result caching: the latency and cost wins hiding in your stack

Look at your agent's tool call logs for one shift. Pick any FAQ lookup, pricing table fetch, or product catalog query. Count how many times it ran. Now count how many times the underlying data actually changed.

For most CX agent deployments, those two numbers are very different. The agent is re-fetching the same data every call, sometimes dozens of times per hour, for information that changes once a day at most. Each re-fetch costs time (300-500ms for an external API call), money (per-request API charges, database compute), and context window tokens when the result gets injected back into the conversation.

Tool result caching eliminates the redundant fetches. The implementation is simpler than most teams expect, and the impact on both cost and latency is significant enough that it belongs in your first wave of production optimizations.

What tool result caching is, and how it differs from prompt caching

Most teams have heard of prompt caching. Anthropic, OpenAI, and Google support it natively: if the beginning of your prompt stays constant across requests (system prompt, tool definitions, long preamble), the provider caches those input tokens and charges you a fraction of normal cost on cache hits.

Prompt caching handles the input side. Tool result caching handles the output side.

When an agent calls a tool, it receives a result: a JSON response from an API, rows from a database, the contents of a knowledge base article. That result gets injected back into the agent's context window as a new message. The LLM then processes the combined context and decides what to do next. If you cache the tool's output, the entire call-wait-inject sequence collapses to a cache lookup.

The latency difference is substantial. An external API call takes 200-500ms. A database query returning 50 rows takes 80-150ms. A Redis cache lookup takes under 5ms. An in-memory cache lookup takes under 1ms. For a conversational agent where each turn involves 3-5 tool calls, removing even half of those external round-trips changes how the conversation feels to the user.

The implementation path for prompt caching is mostly automatic once you format your prompts correctly. Tool result caching is something you build, but the building isn't complicated. That asymmetry is why most guides cover prompt caching thoroughly and barely mention tool result caching.

TVCache: the research behind this pattern

A February 2026 paper on arXiv (TVCache: A Stateful Tool-Value Cache for Post-Training LLM Agents) formalized tool result caching for agentic workflows. The name stands for Tool-Value Cache, making explicit that the target is tool output values rather than input tokens.

The paper's most useful contribution is the stateless/stateful classification. Naive exact-match caching fails for LLM agents because the same tool call can legitimately have different optimal responses depending on conversation history. A database query for "customer account status" should return the same result for the same customer regardless of what else has been said in the conversation. That's a stateless call: output depends only on inputs, safe to cache. A call to "get recommended next action" might need a different response depending on what the agent has already done in the session. That's a stateful call: output depends on session context, not safe to cache.

In TVCache's benchmark tests on agentic task workloads, the stateless/stateful split was the single most important design decision, more than cache size or TTL policy. Getting it right meant higher throughput and correct behavior. Getting it wrong meant incorrect behavior and cache pollution.

The practical implication is that you need to classify your tools before you cache them. Not all tools are cacheable, and caching the wrong ones produces incorrect agent behavior, not just suboptimal performance.

Connected Integrations12 active

Salesforce

Slack

Google

Stripe

HubSpot

Intercom

Zapier

Shopify

GitHub

Jira

Gmail

PostgreSQL

What to cache and what to skip

The caching decision should be made per tool, not per agent or per session. Here is the framework that works in practice.

Cache these:

• Product catalogs, pricing tables, and feature lists. They change on a schedule (daily, weekly) and are read constantly across every call.
• FAQ and knowledge base content. The same content gets served to hundreds of customers with slightly different phrasing.
• User profile and account metadata. Changes infrequently but is read at the start of nearly every interaction.
• Historical analytics and reports. Usually immutable once generated.
• Reference data: country codes, shipping zones, taxonomy trees, supported file formats.

Do not cache these:

• Any tool that has side effects. Creating an order, sending an email, updating a record -- caching would suppress the side effect on a cache hit, which is wrong.
• Real-time data: inventory counts, stock prices, live queue lengths, current session state. The purpose of calling the tool is to get the current value.
• Write-then-read sequences: if a tool writes data and another tool immediately reads it back, the read will not reflect the just-written state if cached.
• Authentication tokens and per-user session state. These are both security-sensitive and session-specific.
• Anything whose staleness within the TTL window would cause the agent to take an incorrect action.

The default rule: read-only, slow, expensive, and stable means cache it. Write, real-time, or conversation-state-dependent means don't.

Implementing the cache layer

The simplest cache layer sits between your agent and its tools as a thin proxy. On each tool call, the proxy checks the cache before executing the actual tool. On a miss, it executes the tool and stores the result with a configured TTL before returning it to the agent.

import { Redis } from 'ioredis';
 
const redis = new Redis(process.env.REDIS_URL!);
 
// Per-tool cache configuration
const CACHE_CONFIG: Record<string, { ttlSeconds: number; cacheable: boolean }> = {
  get_product_catalog:   { ttlSeconds: 3600,  cacheable: true  },
  get_pricing_table:     { ttlSeconds: 900,   cacheable: true  },
  get_user_profile:      { ttlSeconds: 86400, cacheable: true  },
  get_faq_content:       { ttlSeconds: 43200, cacheable: true  },
  get_inventory_count:   { ttlSeconds: 0,     cacheable: false },
  create_support_ticket: { ttlSeconds: 0,     cacheable: false },
  send_email:            { ttlSeconds: 0,     cacheable: false },
  get_next_action:       { ttlSeconds: 0,     cacheable: false }, // stateful
};
 
export async function callToolWithCache(
  toolName: string,
  toolArgs: Record<string, unknown>,
  executeTool: (args: Record<string, unknown>) => Promise<unknown>
): Promise<{ result: unknown; cached: boolean }> {
  const config = CACHE_CONFIG[toolName];
 
  if (!config?.cacheable) {
    const result = await executeTool(toolArgs);
    return { result, cached: false };
  }
 
  const cacheKey = `tool:${toolName}:${JSON.stringify(toolArgs)}`;
  const hit = await redis.get(cacheKey);
 
  if (hit) {
    return { result: JSON.parse(hit), cached: true };
  }
 
  const result = await executeTool(toolArgs);
  await redis.setex(cacheKey, config.ttlSeconds, JSON.stringify(result));
 
  return { result, cached: false };
}

The cache key combines tool name and arguments, so different argument values produce different cache entries. Two calls to get_user_profile with different customerId values produce two separate cache entries with independent TTLs.

Wire this proxy into your tool execution layer. In most agent frameworks, tool calls go through a dispatch function. Replace the direct tool invocation with callToolWithCache and the caching behavior is applied automatically without touching individual tool implementations.

For visibility into which tools are being called and what your hit rates look like, Chanl's tool management layer captures tool call timing and frequency per session. Pair that with cache hit/miss logging from your Redis layer and you have the data to tune TTLs and identify the highest-impact caching candidates.

Semantic caching for higher hit rates

Exact-match caching has a ceiling. It only hits when tool arguments are byte-for-byte identical. For conversational agents, that threshold is often too strict.

Two customers asking "what's your return policy for laptops" and "can I return electronics I bought last month" might need the same FAQ content. The exact tool call arguments differ slightly because one routes to category: 'laptops' and the other to category: 'electronics', but both return the same information. An exact-match cache misses both.

Semantic caching addresses this with vector similarity. The cache stores embeddings of previous tool inputs alongside their results. When a new tool call arrives, its embedding is compared against cached entries. If the similarity exceeds a threshold, the cached result is returned without executing the tool.

import { Chanl } from '@chanl/sdk';
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
export async function callToolWithSemanticCache(
  toolName: string,
  naturalQuery: string, // The semantic key -- often the user's phrasing
  toolArgs: Record<string, unknown>,
  executeTool: (args: Record<string, unknown>) => Promise<unknown>
): Promise<{ result: unknown; cached: boolean; similarity?: number }> {
  const semanticHit = await chanl.tools.searchCache({
    toolName,
    query: naturalQuery,
    similarityThreshold: 0.88,
  });
 
  if (semanticHit) {
    return {
      result: semanticHit.result,
      cached: true,
      similarity: semanticHit.similarity,
    };
  }
 
  const result = await executeTool(toolArgs);
 
  await chanl.tools.cacheResult({
    toolName,
    query: naturalQuery,
    result,
    ttlSeconds: 3600,
  });
 
  return { result, cached: false };
}

The similarity threshold is the key tuning parameter. Too low (around 0.7) and you get false positives: semantically different queries return the wrong cached response. Too high (around 0.98) and you're barely above exact matching. For FAQ content and reference data, 0.85-0.90 is a reasonable starting point. For queries that reference specific entities (customer IDs, order numbers, account details), use exact matching instead.

Semantic caching typically increases hit rates by 20-40% compared to exact matching for conversational agents, because users phrase the same underlying request differently and the variation is larger than it looks.

Cache invalidation

Two connected problems. The first is expiration: making sure cached data doesn't outlive its usefulness. The second is active invalidation: updating the cache when the underlying data changes before the TTL expires.

TTL-based expiration handles most cases. Set a staleness tolerance per tool based on how often the underlying data actually changes, and let entries expire naturally. For a pricing table that updates daily, a 1-hour TTL is safe and conservative. For user preferences that rarely change, 24 hours is fine. For reference data that's stable for weeks, you can go longer and gain substantial hit rate improvements.

Active invalidation matters when data changes on a trigger rather than a schedule. If an agent updates a customer's account, the cached profile for that customer is immediately stale. The cleanest pattern is to delete relevant cache entries when a write tool executes successfully.

// Map from write tools to the read tools they invalidate
const INVALIDATION_MAP: Record<string, string[]> = {
  update_user_profile:   ['get_user_profile'],
  update_pricing:        ['get_pricing_table'],
  create_order:          ['get_order_history'],
  resolve_ticket:        ['get_open_tickets', 'get_ticket_status'],
};
 
export async function invalidateOnWrite(
  toolName: string,
  toolArgs: Record<string, unknown>
): Promise<void> {
  const toInvalidate = INVALIDATION_MAP[toolName] ?? [];
  const entityId = toolArgs.customerId ?? toolArgs.entityId ?? '';
 
  for (const readTool of toInvalidate) {
    const pattern = `tool:${readTool}:*${entityId}*`;
    const keys = await redis.keys(pattern);
    if (keys.length > 0) {
      await redis.del(...keys);
    }
  }
}

Keep the invalidation map explicit and maintain it alongside your tool definitions. If it drifts from reality, you serve stale data. Treating the map as documentation of the relationships between your tools is a good discipline regardless of caching.

Measuring what you have

A cache that isn't measured isn't managed. Three numbers per tool tell you everything you need:

Hit rate is the percentage of calls served from cache. A healthy cache for read-heavy tools should exceed 50%. Below 30% usually means your TTL is too short or your key strategy is too specific. Above 80% for a single tool suggests you might extend the TTL and gain even more.

Latency delta compares p50 and p95 for cached versus uncached calls. For an external API with 300ms average latency, you should see cached calls return in under 10ms. If the delta is small, the cache may not be deployed at the right layer.

Cost delta is the cost per session with caching enabled versus disabled. For tool-heavy CX agents, 40-60% reduction is typical for read-heavy workloads. For agents that mostly reason without calling external tools, the impact is smaller.

Chanl's analytics and monitoring layers surface tool call timing and frequency per session. You can identify which tools account for most of your latency budget before deciding what to cache, and validate that the cache is actually being used by watching the metrics after you deploy.

The cost calculation

Here's the rough math for a typical CX agent stack.

An agent handling 500 calls per day, with 8 tool calls per call, runs 4,000 tool calls per day. Assume 60% of those are cacheable read operations (pricing, FAQ, profile lookups), giving 2,400 cacheable calls. If the cache hits 55% of those, you're serving 1,320 calls from cache per day.

At 350ms average latency for an external API call, 1,320 fewer external calls saves about 462 seconds of wait time per day, distributed across calls. Users notice this as faster responses, not as a number. The tool-layer cost savings (if you're paying per-API-request) depend on your specific vendor, but 40-60% reduction in per-call API charges is consistent with what teams report after implementing basic caching.

The implementation effort for a basic exact-match cache is a few days. Semantic caching with vector similarity adds another week. Both are worth it at any meaningful call volume.

The guide on prompt caching for production agents covers the complementary technique that handles the input token side of cost optimization. The guide on parallel tool calls covers reducing latency by running independent tool calls simultaneously. Tool result caching completes the picture by eliminating redundant tool calls entirely.

Together, these three patterns address the three main sources of agent latency and cost: input token re-processing (prompt caching), tool call serialization (parallel calls), and redundant data fetching (tool result caching). Most teams implement them in that order because prompt caching is free with existing providers, parallel calls require code changes, and result caching requires a cache infrastructure layer. All three matter at production scale.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed