Build agents that fix themselves when they fail

Your customer service agent finished the call with a confident summary. The caller said goodbye. But the refund was never issued. No exception was raised. No alert fired. The agent decided the task was complete when it wasn't.

This is the failure mode that doesn't show up in your error logs, doesn't trigger your monitoring, and won't surface until a customer calls back angry. Most agents today have exactly two failure responses: retry and crash. They'll retry the same API call a few times before giving up, or they'll surface an exception and halt. What they won't do is reason about why they failed, adapt their approach, and try something different.

That's what self-healing adds. Given that silent degradation is already the dominant failure mode for long-running agents, building recovery logic into your agent's core design isn't a nice-to-have for production CX systems.

What self-healing actually means

Self-healing doesn't mean your agent magically fixes production bugs. It means your agent can detect when it's in a failure state, classify what kind of failure it is, and execute a recovery path appropriate for that specific failure class.

That distinction from basic retry matters because most recovery logic is too coarse. Exponential backoff works when a downstream API is temporarily unavailable. It does nothing when the agent is stuck in a reasoning loop, calling the same tool with slightly different parameters because it never checks whether its goal is already satisfied. And exponential backoff tells you nothing about termination failures, where the tool calls all succeeded but the underlying task wasn't actually completed.

A May 2026 paper formalizing the problem (arxiv:2605.06737) defines four failure classes accounting for the majority of production agent failures:

• Execution errors: tools fail, APIs return errors, timeouts occur
• Output errors: the tool call succeeds but returns invalid or unexpected data
• Reasoning failures: the agent's plan is internally consistent but wrong
• Termination failures: the agent finishes before the task is actually done

Each class needs a different detection signal and a different recovery response. The mistake most teams make is handling all four with the same generic retry wrapper.

The four failure classes your agents hit

Production agents fail in four distinct ways: execution errors (tools fail), output errors (tools succeed but return bad data), reasoning failures (logic loops), and termination failures (agent stops before the task is done). Each needs different detection and a different recovery path. Treating all four the same way is why most retry logic falls short.

Execution errors are the easy case. Your tool call returns a 429 or a network timeout. The signal is clear, the recovery is mechanical: back off and retry, or route to a fallback tool. Most agents handle these already.

Output errors are more subtle. The API responds with status 200, but the data is wrong: a null where you expected a string, a truncated list, a stale record from a caching layer. The agent ingests it without question and builds its next reasoning step on bad data. By turn four, the agent is confidently proceeding down a path with a wrong premise embedded in working memory.

Reasoning failures are the most expensive. The agent looks busy. It's calling tools, generating observations, updating its working memory. But it's looping: trying the same search with slightly different phrasing, checking the same record repeatedly, or pursuing a subtask that can't solve the main goal. A 2026 analysis found that reasoning loops account for 15.7% of all agent failures in production, and a single loop can burn tens of dollars in API credits before timeout kicks in.

Termination failures are the most dangerous for CX agents. The agent sends a response, logs the session as complete, and closes. But it never issued the refund, never sent the confirmation email, never created the follow-up ticket. The customer thinks the problem is solved. It isn't. This failure class produces no exception and no error signal. Your dashboards show a successful session.

How to detect that your agent is failing

Three techniques cover the non-obvious failures: schema validation catches bad tool output before it enters your agent's reasoning chain, argument hashing catches reasoning loops by tracking repetition, and a separate completion check catches tasks that the agent declared done before they were. Execution errors already surface as exceptions. These three don't.

Validating tool outputs before reasoning on them

Output error detection requires checking tool results against a schema before incorporating them into the agent's reasoning chain. Most frameworks don't enforce this. The agent receives a tool response and treats it as ground truth.

import { z } from 'zod';
 
const OrderSchema = z.object({
  orderId: z.string(),
  status: z.enum(['pending', 'processing', 'shipped', 'delivered', 'cancelled']),
  amount: z.number().positive(),
  customerId: z.string(),
});
 
type ValidationResult<T> =
  | { valid: true; data: T }
  | { valid: false; error: string };
 
function validateToolResult<T>(
  toolName: string,
  result: unknown,
  schema: z.ZodSchema<T>
): ValidationResult<T> {
  const parsed = schema.safeParse(result);
  if (!parsed.success) {
    return {
      valid: false,
      error: `${toolName} returned unexpected shape: ${parsed.error.message}`,
    };
  }
  return { valid: true, data: parsed.data };
}
 
// In your agent tool-call handler
const rawOrder = await tools.getOrder(orderId);
const validated = validateToolResult('getOrder', rawOrder, OrderSchema);
 
if (!validated.valid) {
  recoveryState.recordFailure({ class: 'output-error', tool: 'getOrder', error: validated.error });
  return recoverFromOutputError(recoveryState);
}

The rule: never let an unvalidated tool result flow forward in your agent's reasoning chain. An agent that ingests malformed data will produce confident-sounding output for the rest of the session with a wrong premise embedded at the base.

Detecting reasoning loops

Loop detection tracks tool call history. If your agent calls the same tool with semantically similar arguments more than twice in a short window, it's stuck. Sorting argument keys before hashing catches the common case where key order varies between calls even though the intent is identical.

import { createHash } from 'crypto';
 
class LoopDetector {
  private history: Array<{ tool: string; hash: string }> = [];
 
  record(toolName: string, args: Record<string, unknown>): void {
    const normalized = JSON.stringify(args, Object.keys(args).sort());
    const hash = createHash('md5').update(normalized).digest('hex');
    this.history.push({ tool: toolName, hash });
  }
 
  isLooping(windowSize = 6): boolean {
    if (this.history.length < windowSize) return false;
    const window = this.history.slice(-windowSize);
    const unique = new Set(window.map(c => `${c.tool}:${c.hash}`));
    return unique.size < Math.ceil(windowSize / 2);
  }
 
  repetitionCount(): number {
    const window = this.history.slice(-12);
    const unique = new Set(window.map(c => `${c.tool}:${c.hash}`));
    return window.length - unique.size;
  }
}

Checking completion before declaring done

The completion check is separate from generating a response. Before your agent sends a final message, it runs a judgment against the original task specification. Use a separate model instance with a minimal context, not the same conversation window. The same context window will often confirm completion because that's what the conversation trajectory predicts, not because the task is actually done.

interface CompletionCheck {
  complete: boolean;
  reason: string;
  missingSteps: string[];
}
 
async function checkCompletion(
  originalTask: string,
  actionLog: string[],
  llm: LLMClient
): Promise<CompletionCheck> {
  const response = await llm.complete({
    model: 'claude-haiku-4-5-20251001',
    messages: [{
      role: 'user',
      content: [
        `Task: ${originalTask}`,
        '',
        'Actions taken:',
        ...actionLog.map((a, i) => `${i + 1}. ${a}`),
        '',
        'Is this task genuinely complete? Return JSON: { complete, reason, missingSteps }',
        'Only mark complete if every step in the original task has been verified as done.',
      ].join('\n'),
    }],
    responseFormat: { type: 'json_object' },
  });
 
  return JSON.parse(response.content);
}

The model choice matters here. A fast, cheap model makes this check affordable enough to run on every session. You don't need frontier capability to judge whether a task description matches an action log.

Recovery strategies for each failure class

Detection gets you the signal. Recovery decides what to do with it. The key insight from the self-healing research is that each failure class needs a different response, not just a different retry count.

Execution errors: context-aware retry

Before retrying a failed tool call, note what failed and why in the agent's working memory. If the payment API rate-limited you, that context helps the agent decide whether to wait or try a different path. The notes fed into working memory aren't just for logging -- they feed back into the agent's next reasoning step.

async function retryWithContext<T>(
  toolCall: () => Promise<T>,
  toolName: string,
  memory: string[],
  maxRetries = 3
): Promise<T> {
  let lastError: Error = new Error('Unknown');
 
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      return await toolCall();
    } catch (error) {
      lastError = error as Error;
      const isRateLimit =
        lastError.message.includes('429') || lastError.message.includes('rate limit');
      const waitMs = isRateLimit ? Math.pow(2, attempt) * 1000 : 500;
 
      memory.push(
        `${toolName} failed (attempt ${attempt}/${maxRetries}): ${lastError.message}. ` +
        `Waiting ${waitMs}ms.`
      );
 
      await new Promise(resolve => setTimeout(resolve, waitMs));
    }
  }
 
  memory.push(`${toolName} exhausted retries. Trying alternative approach.`);
  throw lastError;
}

Output errors: simplify or reroute

When a tool returns invalid data, try a simplified version of the request before routing to a fallback. A tool that returns malformed results for a complex, multi-filter query might work fine when you remove the optional constraints and make a narrower request.

async function recoverFromOutputError(
  toolName: string,
  originalArgs: Record<string, unknown>,
  tools: ToolRegistry,
  memory: string[]
): Promise<unknown> {
  memory.push(`Output validation failed for ${toolName}. Trying simplified request.`);
 
  const simplified = simplifyArgs(originalArgs);
  const simplifiedResult = await tools.call(toolName, simplified);
  const revalidated = validateToolResult(toolName, simplifiedResult, getSchema(toolName));
 
  if (revalidated.valid) {
    memory.push(`Simplified request succeeded. Continuing with available data.`);
    return revalidated.data;
  }
 
  const fallback = tools.findFallback(toolName);
  if (fallback) {
    memory.push(`Rerouting to ${fallback.name}.`);
    return tools.call(fallback.name, originalArgs);
  }
 
  throw new Error(`No valid data source for ${toolName} after recovery attempts`);
}

Reasoning loops: interrupt and replan

Breaking out of a loop and continuing where you left off usually just shifts the loop to a different set of tool calls. The more effective approach: summarize what's been accomplished, clear the working state for the stuck subtask, and replan from the current state with fresh context.

async function handleReasoningLoop(
  state: AgentState,
  originalTask: string,
  llm: LLMClient
): Promise<AgentState> {
  const progressSummary = await llm.complete({
    model: 'claude-haiku-4-5-20251001',
    messages: [{
      role: 'user',
      content: `Summarize what has been accomplished:\nTask: ${originalTask}\nActions: ${state.memory.join('\n')}`,
    }],
  });
 
  const freshState: AgentState = {
    ...state,
    memory: [
      `Previous progress: ${progressSummary.content}`,
      'Previous approach hit a loop. Replanning from current state.',
    ],
    loopCount: (state.loopCount ?? 0) + 1,
  };
 
  if (freshState.loopCount > 2) {
    return escalateToHuman(freshState, `Repeated reasoning loop on: ${originalTask}`);
  }
 
  return freshState;
}

The loop count is the budget. If you're replanning more than twice on the same task, it's a signal the task is outside the agent's capability or your prompt for that intent needs work. Escalate rather than burn tokens on diminishing returns.

Termination failures: the completion loop

When the completion check catches an incomplete task, feed the missing steps back into context and continue. Give it a hard iteration budget before escalating.

async function runWithCompletionCheck(
  task: string,
  agent: Agent,
  llm: LLMClient,
  maxIterations = 5
): Promise<AgentResult> {
  let state = await agent.run(task);
  const actionLog: string[] = [...state.memory];
 
  for (let i = 0; i < maxIterations; i++) {
    const check = await checkCompletion(task, actionLog, llm);
 
    if (check.complete) {
      return { success: true, state, completionIterations: i + 1 };
    }
 
    const continuation = await agent.continue(state, [
      `Task not yet complete. Reason: ${check.reason}`,
      `Missing: ${check.missingSteps.join(', ')}.`,
      `Complete these steps now.`,
    ].join(' '));
 
    state = continuation;
    actionLog.push(...state.newActions);
  }
 
  return escalateToHuman(state, `Task still incomplete after ${maxIterations} iterations`);
}

Deploy Gate

Pre-deploy quality checks

Score > 80%

92%

Latency < 500ms

234ms

Error Rate < 2%

3.1%

Deploy Blocked

Testing self-healing logic before production

Self-healing logic only helps if it fires under the right conditions. You need to deliberately trigger each failure class in your test suite, not just test the happy path.

import { describe, it, expect, jest } from '@jest/globals';
 
describe('self-healing recovery', () => {
  it('detects a reasoning loop and triggers replan', async () => {
    const stubbedSearch = jest.fn().mockResolvedValue({ results: [] });
    const agent = buildAgent({ tools: { customerSearch: stubbedSearch } });
 
    const result = await agent.run('Find order 12345 and issue a refund');
 
    expect(stubbedSearch.mock.calls.length).toBeLessThan(6);
    expect(result.recoveryLog).toEqual(
      expect.arrayContaining([
        expect.objectContaining({ failureClass: 'reasoning-loop' }),
      ])
    );
  });
 
  it('catches an incomplete task before the session closes', async () => {
    const updateStatus = jest.fn().mockResolvedValue({ updated: true });
    const sendEmail = jest.fn().mockResolvedValue({ sent: true });
 
    const agent = buildAgent({ tools: { updateStatus, sendEmail } });
    await agent.run('Mark order 789 as shipped and email the customer');
 
    expect(sendEmail).toHaveBeenCalled();
  });
 
  it('escalates after repeated replanning attempts', async () => {
    const agent = buildAgent({ tools: { lookupOrder: jest.fn().mockResolvedValue(null) } });
    const result = await agent.run('Find and cancel order 99999');
 
    expect(result.status).toBe('escalated');
    expect(
      result.recoveryLog.filter(r => r.failureClass === 'reasoning-loop').length
    ).toBeGreaterThanOrEqual(2);
  });
});

Running these in CI alongside your happy-path tests gives you regression coverage for the recovery paths, not just the happy path. An agent that passes functional tests but fails recovery tests will produce silent failures in production.

If you want to test under realistic conversation conditions, Chanl's scenario runner lets you construct test conversations that deliberately trigger each failure mode and then evaluate whether the recovery produced an acceptable outcome for the customer.

Connecting recovery signals to monitoring

Self-healing is most useful when you can see what it's catching. Recovery events without observability are just noise you can't act on.

Chanl's scorecards evaluate completed conversations against your quality criteria. Combining "agent recovered from a failure" and "outcome quality was poor" in the scorecard metadata gives you the most actionable signal for improvement: these are the cases where recovery ran but still didn't produce a good experience.

import { Chanl } from '@chanl/sdk';
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
await chanl.scorecards.evaluate({
  sessionId: session.id,
  rubric: 'cx-agent-quality',
  metadata: {
    recoveryAttempts: recoveryLog.length,
    failureClasses: recoveryLog.map(r => r.failureClass),
    recoveredSuccessfully: recoveryLog.every(r => r.outcome === 'recovered'),
    escalated: session.status === 'escalated',
  },
});

Monitoring then gives you per-failure-class rates over time. If your reasoning loop rate trends up after a prompt change, that's your signal the change degraded something. If your execution error rate spikes on a Tuesday morning, that's probably an upstream API issue, not your agent.

What self-healing doesn't solve

Self-healing makes your agent more resilient to failures it can classify and recover from. It doesn't make your agent capable of tasks it simply can't handle. There's a version of the completion loop that just extends the time your agent spends failing before finally escalating.

The budget parameters -- max retries, max replan iterations, escalation thresholds -- are judgment calls. Set them too loose and you burn tokens on hopeless recovery attempts. Set them too tight and you escalate cases the agent could have resolved.

Start conservative. Watch your escalation rate. Loosen budgets only when data shows the agent resolves the cases you're currently escalating. And if certain failure classes are consistently high despite recovery, look at root causes: persistent memory reduces termination failures, better prompt design reduces reasoning loops, and circuit breakers handle the infrastructure failures that produce execution errors.

Recovery logic is a safety net. The goal is to need it less over time, not to make it more aggressive.

The May 2026 research found that self-healing frameworks increase task success rates by 27% compared to basic retry across a broad range of agent tasks. But those gains come from targeted recovery per failure class, not from more aggressive retrying. That's where most teams leave improvement on the table: they add more retries when they should add better detection.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed