Fault Injection Is the Missing Layer in Agent Testing

Your agent sailed through QA. Every scenario passed. Call success rate hit 96% on test. Three weeks into production, the CRM started hitting its rate limit during peak hours, and that number dropped to 67%. No one caught it in testing because the test suite only covered happy paths.

ReliabilityBench, an evaluation paper published in January 2026 (arxiv 2601.06112), tested LLM agents across a range of production-like stress conditions: repeated runs with slight variation, semantically equivalent inputs phrased differently, and injected infrastructure faults including timeouts, rate limits, partial responses, and schema drift. Rate limiting was the single most damaging failure type. More than timeouts. More than partial responses. More than wrong answers.

The paper calls the problem a reliability surface. Most teams measure one point on it -- single-run success on clean inputs -- while production exposes the entire surface every day. This post is about mapping that surface before your users do.

Why Happy-Path Tests Miss the Real Failures

Happy-path testing checks that your agent does the right thing when everything works. Tools respond on time. APIs return correct schemas. The customer's intent is unambiguous. Those conditions hold on test infrastructure because your test infrastructure is designed to hold them. Production infrastructure is not.

The failures that degrade production success rates aren't reasoning failures. Your agent knows what to do. The failures are infrastructure failures: the CRM returns HTTP 429, the payment processor times out after 29 seconds, the knowledge base returns a truncated result because the backend is under load, the tool schema changed in a minor API update. None of those conditions appear in typical test suites. All of them happen in production.

ReliabilityBench built a framework to measure exactly this gap. They defined a unified reliability surface R(k, ε, λ):

• k is the number of independent runs of the same task. pass^k measures success on all k runs, which is what production requires -- not just one good run.
• ε (epsilon) is the perturbation level. At ε=0.2, 20% of inputs are semantically equivalent rewrites of the original: different phrasings, different order of details, different vocabulary.
• λ (lambda) is the fault rate. At λ=0.1, 10% of tool calls return a simulated infrastructure failure of one of the defined types.

At ε=0 and λ=0, agents look great. The paper found baseline success rates around 96-97% for the models tested. Add ε=0.2 and success dropped to 88.1%. Add fault injection on top of that and it dropped further. Rate limiting caused the steepest degradation across all four tested domains: scheduling, travel, customer support, and e-commerce.

The finding is logical once you trace through it. A timeout can be retried. A partial response can sometimes be handled by asking again. A rate limit response (HTTP 429 with a Retry-After header) requires the agent to understand what it received, recognize that the right behavior is to wait and retry rather than continue, and then continue correctly after the wait window. Most agents don't handle this gracefully. They hit the 429, receive an unexpected error type, and either fail the tool call or loop into retries that trigger more 429s.

What Semantic Perturbation Tests Reveal

The perturbation dimension is the one that surprises most teams. Semantically equivalent inputs should produce semantically equivalent outputs. "I want a refund," "Can I get my money back," and "I'd like to return this for a full refund" all express the same customer intent. An agent that handles the first phrasing correctly but not the third has a vocabulary sensitivity problem that will show up in production with real customers who say things your training data didn't cover.

Perturbation testing works by generating semantically equivalent rewrites of your test inputs and checking that the end-state is identical across all variants. Not identical text output -- identical tool calls, identical data mutations, identical resolution outcome. ReliabilityBench calls these action metamorphic relations: a pair of inputs where the relation between them implies a required relation between their outputs.

// Semantically equivalent inputs -- all express the same refund request
const refundVariants = [
  "I want a refund for order 18833",
  "Can you refund my order? It's 18833.",
  "I'd like my money back for order number 18833",
  "Please process a return for order 18833",
  "This charge on order 18833 needs to be reversed",
];
 
// Run all variants through the agent
const results = await Promise.all(
  refundVariants.map((input) =>
    runAgent({ message: input, customerId: "cust_001" })
  )
);
 
// Check end-state equivalence: all variants should call issue_refund
// with the same order_id and amount, regardless of phrasing
const endStates = results.map((r) => extractToolCallState(r.trace));
 
const allEquivalent = endStates.every(
  (state) =>
    state.toolCalled === "issue_refund" &&
    state.args.orderId === "18833" &&
    Math.abs(state.args.amount - endStates[0].args.amount) < 0.01
);
 
if (!allEquivalent) {
  const failedVariants = endStates
    .map((s, i) => ({ variant: refundVariants[i], state: s }))
    .filter((_, i) => !isEquivalent(endStates[i], endStates[0]));
 
  console.error("Metamorphic test FAILED:", failedVariants);
}

When a perturbation test fails, the failure tells you something actionable: which phrasing variant your agent handles differently, and whether the difference is in intent classification, tool selection, or parameter extraction. That's a specific debugging target. Without perturbation testing, you'd find this in production when a real customer happened to phrase their request in the variant your agent was trained to misread.

The ReliabilityBench results showed that domain-tuned models maintained consistent behavior across perturbation levels much better than base models. Domain-Tuned maintained 72.8% pass^8 while ReAct-GPT4 dropped 19.4% from its pass@1 baseline when perturbations were applied. The difference is distribution coverage: domain-tuned models have seen enough of how real customers phrase things that semantic variation is already in their training distribution, where base models treating novel phrasings as genuinely novel inputs lose performance.

Injecting Faults: Rate Limits, Timeouts, Schema Drift

The λ dimension of the reliability surface is what you build with fault injection. Instead of letting production expose your agent to infrastructure failures, you expose it deliberately in test, where you can measure the degradation and respond before users encounter it.

Three fault types matter most for CX agents:

Rate limits (HTTP 429). CRM APIs, payment processors, and telephony providers all have rate limits. Under peak load, your agent may hit them mid-workflow. A well-designed agent reads the Retry-After header, waits the specified duration, and continues. Most agents either crash on the unexpected status code or loop into retries that compound the rate limiting. Test this explicitly before it happens in production.

Timeouts. Backend systems that work fine at 99th percentile latency sometimes take 30 seconds on the one call in a hundred where the database is under load. Your agent's tool timeout configuration needs to match your actual SLA requirements, and you need to test that timeout handling produces a useful fallback rather than an unhandled exception.

Schema drift. Tool response schemas change. An order lookup that returned { "status": "shipped" } starts returning { "order_status": "in_transit", "carrier": "UPS", "updated_at": "2026-06-19T14:22:00Z" } after a backend update. If your agent is reading output.status directly, it gets undefined on the new schema and may hallucinate a value, fail silently, or produce a confusing response to the customer. Schema drift is especially dangerous because it doesn't cause an immediate visible error -- it causes subtle quality degradation that's hard to attribute.

Here's a fault proxy that handles all three:

type FaultConfig = {
  rateLimitRate?: number;       // fraction of calls that return HTTP 429
  timeoutRate?: number;         // fraction of calls that hit the timeout threshold
  partialResponseRate?: number; // fraction of calls that return truncated data
  schemaDriftEnabled?: boolean; // apply schema mutations to all responses
};
 
class FaultProxy {
  constructor(
    private realTools: Record<string, (...args: unknown[]) => Promise<unknown>>,
    private config: FaultConfig
  ) {}
 
  async call(toolName: string, args: unknown[]): Promise<unknown> {
    const rand = Math.random();
 
    if (this.config.rateLimitRate && rand < this.config.rateLimitRate) {
      throw Object.assign(new Error("Too Many Requests"), {
        status: 429,
        headers: { "retry-after": "5" },
      });
    }
 
    if (this.config.timeoutRate && rand < this.config.timeoutRate) {
      await new Promise((_, reject) =>
        setTimeout(() => reject(new Error("Tool timeout after 30000ms")), 30)
      );
    }
 
    const result = await this.realTools[toolName](...args);
 
    if (this.config.schemaDriftEnabled) {
      return applyRandomSchemaMutation(result);
    }
 
    if (
      this.config.partialResponseRate &&
      rand < this.config.partialResponseRate
    ) {
      return truncateAtFieldBoundary(result);
    }
 
    return result;
  }
}

Building Your Reliability Surface

The reliability surface is a function, not a number. To map it, you run the same scenario suite at different values of k, ε, and λ and record how success rates change across the parameter space.

A practical starting point for most CX teams:

Step 1: Baseline (k=4, ε=0, λ=0). Run your scenario suite four times each with clean inputs and no fault injection. Record pass^4. If your agent has 90% single-run success but 65% four-run consistency, you have a baseline inconsistency problem that exists before you add any faults. The reliability-capability gap post covers why this inconsistency exists and what drives it -- but for testing purposes, a baseline pass^4 below 80% on well-defined scenarios means the agent needs work before fault injection results are meaningful.

Step 2: Perturbation sweep (k=1, ε=0 to 0.2, λ=0). Run each scenario with five phrasings of the same intent, progressively more different from the training distribution. Where does success rate start dropping? Degradation at ε=0.05 means your agent is vocabulary-sensitive. Stability through ε=0.2 means it's handling real-world phrasing variation reasonably well.

Step 3: Fault injection (k=4, ε=0.1, λ=0.05 to 0.2). Run scenarios with 5%, 10%, and 20% rate limit injection rates. Measure which scenarios degrade fastest. For CX workloads, the scenarios with the most tool calls will degrade first -- more calls means more chances to hit a rate limit.

Step 4: Domain-specific faults. Add the actual error responses from your production backends: your CRM's real 429 message format, your payment processor's actual timeout behavior, your knowledge base's actual truncation pattern when it's under load. Generic fault injection tells you about structural weaknesses. Domain-specific injection tells you about your actual production risk.

The result is a surface that shows you where your reliability budget is. If your SLA requires 90% success rate and your surface shows you hit 83% at λ=0.1, you need either a lower real-world fault rate (engineering your backend reliability), a more fault-tolerant agent (handling 429s with backoff, partial responses with explicit retry logic), or a narrower SLA claim.

Catching Regressions in CI Before Production

Fault injection testing belongs in your CI pipeline, not in ad-hoc manual testing. The risk is that a model update or a tool schema change shifts your reliability surface without anyone noticing -- and the first indication is a production incident during peak hours.

A CI integration for fault injection runs on every merge to main. It's slower than your unit tests and you don't need every variation, but it needs to run:

import { Chanl } from "@chanl/sdk";
 
async function runReliabilitySuite(): Promise<void> {
  const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
  // Run at lambda=0.1 (10% fault rate) and epsilon=0.1 (10% perturbation)
  const results = await chanl.scenarios.run({
    suite: "customer-support-reliability",
    faultInjection: {
      rateLimitRate: 0.1,
      timeoutRate: 0.05,
      partialResponseRate: 0.05,
    },
    repetitions: 4,
    perturbationLevel: 0.1,
  });
 
  const passK = results.scenarios.filter((s) => s.passedAllRuns).length /
    results.scenarios.length;
 
  console.log(
    `Reliability surface at lambda=0.1, epsilon=0.1: pass^4 = ${(passK * 100).toFixed(1)}%`
  );
 
  // Fail the build if reliability dropped more than 5 points from baseline
  if (passK < results.baseline.passK - 0.05) {
    console.error(
      `Reliability regression detected: baseline ${(results.baseline.passK * 100).toFixed(1)}% -> current ${(passK * 100).toFixed(1)}%`
    );
    process.exit(1);
  }
 
  // Also report which fault types caused the most degradation
  for (const [faultType, impact] of Object.entries(results.faultImpact)) {
    if (impact.successRateDrop > 0.1) {
      console.warn(
        `High fault sensitivity: ${faultType} caused ${(impact.successRateDrop * 100).toFixed(1)}% success drop`
      );
    }
  }
}

What you're tracking over time is the reliability surface, not individual test pass rates. A model update might improve single-run success on clean inputs and simultaneously degrade performance under faults. If you only measure single-run clean success, the regression is invisible.

This is the same pattern that catches schema drift regressions. If a tool schema changes in a backend update, your clean-input scenarios might still pass (the agent figures it out from context), but your fault injection runs at ε=0.1 will show degradation as the agent tries to handle both the phrasing variation and the schema change simultaneously. The fault injection surface is more sensitive to these subtle changes than the happy-path suite.

What Domain-Tuned Models Do Differently Under Pressure

The most actionable ReliabilityBench finding for teams building on foundation models is the gap in fault tolerance between base models and domain-tuned ones. Domain-tuned models weren't just more accurate -- they degraded more slowly as fault rates and perturbation levels climbed.

At the 20% perturbation level, Domain-Tuned maintained 72.8% pass^8 while ReAct-GPT4 dropped 19.4% from its baseline. The researchers attribute this to distribution coverage: domain-tuned models have seen enough of the space of real inputs, including weird phrasings, tool error responses, and edge cases, that these conditions are less disorienting at inference time.

The practical implication: your fault injection test results are a leading indicator of where to focus fine-tuning. If your agent's reliability surface degrades sharply at rate limits, you want training examples that show correct Retry-After handling. If it degrades under schema drift, you want examples showing the agent recognizing missing fields and asking for clarification rather than inferring incorrect values.

Your production monitoring closes this loop. When a production failure traces back to a rate limit or a partial response, that trace is a labeled training example for the exact failure mode your agent needs to handle better. The failures you observe and log today become the fault injection test cases you run against the next model version. Conversation intelligence in your monitoring pipeline surfaces these failures with the context needed to use them as training signal.

The Right Sequence for Adding Fault Testing

Fault injection testing doesn't replace your existing test suite -- it extends it after the basics are solid. The sequence that works:

Start with happy-path scenarios. Verify basic behavior on clean inputs before adding complexity. If your agent can't handle the straightforward cases, fault injection results won't be interpretable.

Add perturbation testing. Run five semantic variants of each scenario. This catches vocabulary sensitivity before you get to fault injection and is cheap to run.

Add rate limit injection first. It's the most damaging fault type, it's easy to inject (a wrapper that returns 429 on N% of calls), and it immediately tells you whether your agent has any fault-handling logic at all.

Add timeout and partial response injection. These reveal different failure modes: timeout handling (does the agent retry? bail? hang?) and incomplete data handling (does the agent recognize the data is truncated or treat it as authoritative?).

Add schema drift testing. Generate mutated versions of your tool schemas -- rename a field, add required fields, change a value type -- and run your scenarios against them. The scenarios that fail under schema drift are your fragile ones.

The scenario suite in Chanl's testing platform lets you configure all of this per test run and tracks the reliability surface across deploys. The agent regression testing post covers how to structure the CI pipeline around it. What matters is that fault injection becomes routine, not something you add after a production incident.

The single most useful thing you can add to your test suite this week is a rate limit injection run on your highest-volume workflow. Find the API that handles your most common tool call, wrap it in a fault proxy that returns 429 on 10% of calls, run 20 representative scenarios four times each, and measure pass^4. If it's above 80%, your fault handling is reasonable. If it's below 60%, you have a reliability gap that production is going to find for you -- probably during peak hours on a Friday.

El briefing de Signal

Un email por semana. Cómo los equipos líderes de CS, ingresos e IA están convirtiendo conversaciones en decisiones. Benchmarks, playbooks y lo que funciona en producción.

Subscribe

500+ líderes de CS e ingresos suscritos