Design agent tools that won't double-charge on retry

A customer calls your support line to dispute a charge. Your agent looks up the account, confirms the error, and calls issue_refund. The CRM API hangs. Your agent framework times out after five seconds and retries. The CRM was slow, not down -- the first call went through, but the response never arrived. Now the customer has two refunds and your accounting team has a reconciliation problem.

This is the idempotency problem. It's not a rare edge case. It's what happens to every production agent that doesn't plan for retries.

Your agent will retry. Build for it.

Retries are inevitable in production agent systems. LLM API calls fail 1-5% of the time due to rate limits, timeouts, and server errors. External tool APIs fail at similar rates. In a 10-step workflow with a 2% per-step failure rate, you have roughly an 18% chance of at least one step failing per session. At 5% per step, that's over 40%.

Your agent framework will retry automatically for transient errors. The LLM itself might decide to retry after seeing an error response in a tool result. Your reliability layer (covered in detail in circuit breakers for AI agents) provides another retry layer on top of that.

Every one of those retries is a second call to a tool that may have already succeeded. If your tools aren't designed to handle duplicate calls, a retry isn't a safety net. It's a bug waiting to cause customer-facing damage.

What makes a tool naturally idempotent

Reads are already safe. Calling get_customer_profile twice returns the same data both times with no side effects. You can retry reads as many times as you need without risk.

Writes are the problem. When a tool creates a record, processes a payment, sends a message, or updates a state machine, calling it twice causes two executions. Two refunds. Two confirmation emails. Two CRM updates that may conflict.

Natural idempotency means the operation is inherently safe to repeat. A set_account_status(accountId, status) call that writes a boolean flag is effectively idempotent -- setting a flag to "cancelled" twice leaves the system in the same state as setting it once. A create_refund(accountId, amount) call is not. The second call creates a second refund.

The question isn't whether a tool is inherently idempotent. It's whether you can design it to behave idempotently. That's what idempotency keys give you.

The idempotency key pattern

An idempotency key is a unique string that represents exactly one logical operation. Your tool server checks whether it has seen this key before. If it has, it returns the stored result without executing again. If it hasn't, it executes and stores the result.

Here's the minimal shape:

async function issueRefund(
  accountId: string,
  amount: number,
  idempotencyKey: string
): Promise<RefundResult> {
  // Check for existing result
  const cached = await redis.get(`idem:${idempotencyKey}`)
  if (cached) {
    return JSON.parse(cached) as RefundResult
  }
 
  // Execute the operation
  const result = await paymentProcessor.createRefund({ accountId, amount })
 
  // Store result with 48-hour TTL
  await redis.set(
    `idem:${idempotencyKey}`,
    JSON.stringify(result),
    'EX', 172800
  )
 
  return result
}

The first call executes and caches. Every subsequent call with the same key returns the cached result. The customer gets one refund regardless of how many times the agent retried.

Three properties a key must have to work correctly:

Deterministic. The same logical operation must always generate the same key. If your agent retries step 4 of session sess_abc123, it needs to produce the same key on retry as it did on the original attempt. Random UUIDs break this: a new UUID on retry means the server sees a new operation and executes again, defeating the entire mechanism.

Scoped. Different logical operations must have different keys. A refund in session A and a refund in session B are different operations. A bare accountId as the key would deduplicate across sessions when you only want to deduplicate within a session.

Time-bounded. Keys should expire. A 24-72 hour TTL is appropriate for most CX operations. An indefinitely-cached key from six months ago would block a legitimate new refund for the same customer.

Generating keys in an agent context

The most reliable approach for agent tool calls is a compound key built from session ID and a monotonic call counter:

function buildIdempotencyKey(sessionId: string, callIndex: number): string {
  return `${sessionId}-${callIndex}`
}

sessionId is a stable identifier for the current conversation session. callIndex is a counter that increments for each tool call within the session.

When the agent retries call number 4 in session sess_abc123, it generates sess_abc123-4. The server sees a key it already processed and returns the cached result. No duplicate execution.

Here's how to track the call index in your session context:

class AgentSession {
  private callCounter = 0
 
  constructor(public readonly sessionId: string) {}
 
  getNextIdempotencyKey(): string {
    return `${this.sessionId}-${++this.callCounter}`
  }
 
  getKeyForRetry(originalCallIndex: number): string {
    return `${this.sessionId}-${originalCallIndex}`
  }
}

On a retry, pass the original index. On a new call, increment. The distinction is what makes retries safe while still allowing new operations.

Connected Integrations12 active

Salesforce

Slack

Google

Stripe

HubSpot

Intercom

Zapier

Shopify

GitHub

Jira

Gmail

PostgreSQL

Server-side implementation patterns

Redis SETNX

Redis's set-if-not-exists combined with a TTL gives you fast atomic idempotency checking:

async function withIdempotency<T>(
  key: string,
  ttlSeconds: number,
  fn: () => Promise<T>
): Promise<T> {
  const cacheKey = `idem:${key}`
 
  const cached = await redis.get(cacheKey)
  if (cached !== null) {
    return JSON.parse(cached) as T
  }
 
  const result = await fn()
  await redis.set(cacheKey, JSON.stringify(result), 'EX', ttlSeconds)
 
  return result
}

Wrap every mutating tool with this function:

async function issueRefund(args: RefundArgs): Promise<RefundResult> {
  return withIdempotency(
    args.idempotencyKey,
    48 * 60 * 60,
    () => paymentProcessor.createRefund(args)
  )
}

Postgres UNIQUE constraint

If you're already storing operation records in Postgres, a unique column on idempotency_key handles deduplication at the database layer:

ALTER TABLE refund_operations
  ADD COLUMN idempotency_key TEXT UNIQUE NOT NULL;
 
CREATE INDEX idx_refund_idempotency ON refund_operations (idempotency_key);

Then insert with conflict handling:

async function issueRefund(
  accountId: string,
  amount: number,
  idempotencyKey: string
): Promise<RefundResult> {
  const existing = await db.query(
    'SELECT result FROM refund_operations WHERE idempotency_key = $1',
    [idempotencyKey]
  )
 
  if (existing.rows.length > 0) {
    return existing.rows[0].result as RefundResult
  }
 
  const result = await paymentProcessor.createRefund({ accountId, amount })
 
  await db.query(
    `INSERT INTO refund_operations (account_id, amount, idempotency_key, result)
     VALUES ($1, $2, $3, $4)
     ON CONFLICT (idempotency_key) DO NOTHING`,
    [accountId, amount, idempotencyKey, JSON.stringify(result)]
  )
 
  return result
}

Redis is faster for high-frequency operations. Postgres is simpler when you're already storing the operation record and want deduplication logic co-located with the data rather than in a separate cache tier.

The concurrent retry problem

Here's the scenario that breaks naive idempotency: two instances of your agent retry the same step simultaneously. Both check for the key. Both see nothing. Both proceed to execute. You've bypassed your deduplication logic.

This happens in multi-agent systems with fan-out patterns and in any framework that retries on multiple threads. It's more common than developers expect.

Fix it with atomic operations. In Redis, a Lua script that checks and sets atomically prevents the race:

const checkAndSetScript = `
  local existing = redis.call('GET', KEYS[1])
  if existing then
    return existing
  end
  redis.call('SET', KEYS[1], ARGV[1], 'EX', ARGV[2])
  return nil
`
 
async function atomicIdempotencyCheck(
  key: string,
  result: string,
  ttl: number
): Promise<string | null> {
  return redis.eval(checkAndSetScript, 1, key, result, ttl.toString())
}

In Postgres, ON CONFLICT DO NOTHING combined with checking affected rows handles the race cleanly at the database level. Only one insert wins; the other returns zero affected rows and falls through to the SELECT.

How the agent passes idempotency keys

Two approaches, each with tradeoffs.

Keys as explicit parameters. The agent includes the idempotency key in the tool call arguments. Your tool schema has a required idempotencyKey: string field. This is transparent to the model and observable in your tool call logs. The tradeoff: it adds a field to every mutating tool's schema, and there's a small risk the LLM fills the field with a badly-formed value if it doesn't receive a system-generated one.

Keys via injection. Your agent runtime attaches the idempotency key before forwarding the tool call, keeping the tool schema clean. The key is always system-generated. The tradeoff: requires middleware in your tool client that isn't visible to the model.

For most setups, explicit parameters are simpler to implement and debug. You can wrap your tool call handler to inject the key transparently:

const MUTATING_TOOLS = new Set([
  'issue_refund',
  'create_booking',
  'send_confirmation_email',
  'cancel_subscription',
  'update_account_status'
])
 
async function callTool(
  session: AgentSession,
  toolName: string,
  args: Record<string, unknown>
): Promise<unknown> {
  const enrichedArgs = MUTATING_TOOLS.has(toolName)
    ? { ...args, idempotencyKey: session.getNextIdempotencyKey() }
    : args
 
  return mcpClient.callTool(toolName, enrichedArgs)
}

Maintain an explicit set of mutating tool names. Only attach keys to those. Read-only tools don't need them.

Testing retry safety before production

Chanl scenarios let you simulate retry conditions in a controlled environment before they occur in production.

Write a test that calls each mutating tool twice with the same idempotency key:

import Chanl from '@chanl/sdk'
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY })
 
const result = await chanl.scenarios.run({
  name: 'Refund tool retry safety',
  steps: [
    {
      action: 'call_tool',
      tool: 'issue_refund',
      args: {
        accountId: 'acct_test_001',
        amount: 49.99,
        idempotencyKey: 'test-sess-4'
      }
    },
    {
      action: 'call_tool',
      tool: 'issue_refund',
      args: {
        accountId: 'acct_test_001',
        amount: 49.99,
        idempotencyKey: 'test-sess-4'
      }
    },
    {
      action: 'assert',
      check: 'payment_processor.refunds.count({ accountId: "acct_test_001" }) === 1'
    }
  ]
})

Also test the concurrent case: fire two requests simultaneously and verify exactly one execution completes. This catches the race condition that sequential tests miss.

Add these tests to your CI pipeline. Any tool that mutates external state needs a retry safety test before it ships.

What to monitor once it's deployed

Add these signals to your agent monitoring dashboard:

Idempotency hit rate. What percentage of tool calls are served from the cache rather than executed? A hit rate above 5% suggests frequent retrying. That's worth investigating -- either your tools are unreliable or your agent is retrying more aggressively than expected.

Concurrent conflict rate. How often does your database-level deduplication catch a simultaneous retry? Should be near zero under normal operation. Spikes indicate retry storms from a stuck agent or misconfigured retry policy.

Key expiry collisions. If you see duplicate executions after your TTL window, your TTL is too short for your workflow duration. A multi-day booking flow needs a longer TTL than a real-time support chat.

Log the idempotency key with every tool call result. When a support ticket comes in about a double charge, you want to trace the key back to the session and understand exactly what the agent did and when.

Build it in first, not as a retrofit

Every team that retrofits idempotency after a double-charge incident spends more time cleaning up than teams that built it in from the start.

The steps are the same for every team: maintain a list of mutating tools, generate a deterministic compound key for each call, add atomic server-side deduplication, and test retry safety in CI. None of those steps are expensive. The alternative -- discovering idempotency gaps when a customer calls with a duplicate charge -- costs far more.

The customer in the opening scenario didn't care about your retry policy or your circuit breaker thresholds. They cared that the second refund appeared on their statement. Idempotency is what prevents that from happening.

The durable execution guide covers the complementary layer: checkpointing workflows at the session level so your agent can resume mid-task after a crash. Idempotency keys protect individual tool calls. Durable execution protects the workflow that contains them. Together they give you the full picture for production reliability.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed