When your LLM provider goes down, your agent shouldn't

On April 20, 2026, at 9:14 PM Eastern, the on-call pager fired for a team running a voice AI agent at a mid-market insurance company. Calls were failing at 40%. The culprit was three words in the Anthropic status page: "Investigating API Issues."

The outage lasted six hours. The team had no failover. Every call that hit the agent during those six hours returned an error. By the time the incident resolved, they had missed 1,200 customer interactions.

That team had spent months on prompt engineering, tool integrations, and evaluation pipelines. They hadn't spent an afternoon on failover. This article covers what that afternoon looks like.

Why LLM uptime isn't good enough for production agents

LLM API uptime looks impressive on paper. Anthropic's 90-day rolling uptime in early 2026 sits around 98.95%. OpenAI's is similar. But apply some arithmetic and the picture changes.

98.95% uptime equals 0.05% downtime. Over a year, that is roughly 44 hours of potential unavailability. A critical infrastructure dependency with 44 hours of annual downtime would get every engineering team's attention. LLM APIs get a pass because teams think of them as development tools, not production dependencies. They're wrong.

The math is worse for agents running 24/7. A customer service agent handling 500 calls per day means roughly 21 calls per hour around the clock. A six-hour outage without failover means 126 missed customer interactions in a single incident. A patient agent, a billing agent, or a scheduling agent has similar exposure.

The 99.99% standard that mature SaaS infrastructure achieves (about 52 minutes of annual downtime) requires multiple redundant providers. No single LLM API meets it. Meeting it with AI agents means treating provider failover as a first-class architectural requirement.

How provider outages actually manifest

Understanding the failure modes helps you design the right response:

Hard API errors are the easiest to detect. The provider returns a 503, 529 (overloaded), or connection refused. Your error handling catches it immediately. This is what the Anthropic incident on April 20 looked like at peak: direct connection failures.

Elevated latency is harder. The provider is technically responding but taking 45 seconds instead of 3. Your timeout fires eventually, but not before customers have waited half a minute. This is the failure mode that stress-testing misses, because staging traffic volumes rarely reproduce it.

Degraded quality is the hardest to detect. The provider is up, responding at normal speed, but the model is returning lower-quality outputs due to an internal issue. You won't catch this without evaluators running against production traffic. We covered how to set those up in the agent observability piece.

Rate limit exhaustion is self-inflicted but has the same effect: your agent can't complete requests. If your production traffic spikes during a campaign or seasonal event, you'll hit rate limits before your provider hits an outage.

A complete failover strategy handles all four. The first two call for circuit breakers and timeouts. The third calls for quality evaluation. The fourth calls for per-provider rate limit tracking.

Three failover architectures

The right architecture depends on how many providers you're willing to maintain and what quality tradeoffs you'll accept.

Dual-provider active-passive is the minimum viable setup. One primary, one standby. You route all traffic to the primary and switch to the standby only on failure. Simple to implement, easy to reason about, and covers the common case where one provider has an outage. The risk is that both providers could be down simultaneously, which is rare but not impossible.

Multi-provider with health-weighted routing adds intelligence. Each provider has a health score based on its recent error rate, latency, and any known incidents. Traffic routes to the highest-scoring provider. If two providers are both healthy, you can split traffic to maintain active familiarity with both endpoints. This approach is harder to implement but gives you better data on provider quality before you actually need to fail over.

Tiered fallback chain is the most resilient. Primary, secondary, and a self-hosted or open-weight model as a final backstop. The self-hosted tier never has third-party API downtime. You'll accept a quality tradeoff when you reach it, so you label the degraded state clearly.

For most production CX agents, dual-provider active-passive is the right place to start. The multi-provider approach adds operational complexity that isn't justified until you've instrumented the simpler version and seen where the failures are.

Building the failover layer

The core challenge is making providers interchangeable. The major LLM APIs have different request schemas, streaming formats, and capability sets. A clean abstraction layer lets you swap providers without rewriting agent logic.

type ProviderName = "anthropic" | "openai" | "google" | "local";
 
interface ProviderConfig {
  name: ProviderName;
  client: LLMClient;
  timeoutMs: number;
  priority: number; // lower = higher priority
}
 
interface ProviderHealth {
  name: ProviderName;
  errorRate: number; // rolling 5-minute error rate
  p95LatencyMs: number;
  circuitOpen: boolean;
  lastChecked: string;
}
 
class ProviderRouter {
  private providers: ProviderConfig[];
  private health: Map<ProviderName, ProviderHealth>;
  private circuitBreakers: Map<ProviderName, CircuitBreaker>;
 
  constructor(providers: ProviderConfig[]) {
    this.providers = providers.sort((a, b) => a.priority - b.priority);
    this.health = new Map();
    this.circuitBreakers = new Map(
      providers.map((p) => [p.name, new CircuitBreaker(p.name)])
    );
  }
 
  async complete(messages: Message[], options?: CompletionOptions): Promise<Completion> {
    const orderedProviders = this.getHealthyProviders();
 
    for (const provider of orderedProviders) {
      try {
        const result = await this.callWithTimeout(
          provider,
          messages,
          options
        );
        this.recordSuccess(provider.name);
        return result;
      } catch (err) {
        this.recordFailure(provider.name, err as Error);
        console.warn(`Provider ${provider.name} failed, trying next`, err);
        // Continue to the next provider in the loop
      }
    }
 
    return this.gracefulDegradation(messages);
  }
 
  private getHealthyProviders(): ProviderConfig[] {
    return this.providers.filter((p) => {
      const breaker = this.circuitBreakers.get(p.name);
      return !breaker?.isOpen();
    });
  }
 
  private async callWithTimeout(
    provider: ProviderConfig,
    messages: Message[],
    options?: CompletionOptions
  ): Promise<Completion> {
    const timeout = new Promise<never>((_, reject) =>
      setTimeout(() => reject(new Error("Provider timeout")), provider.timeoutMs)
    );
    return Promise.race([
      provider.client.complete(messages, options),
      timeout,
    ]);
  }
 
  private gracefulDegradation(messages: Message[]): Completion {
    const lastUserMessage = messages.filter((m) => m.role === "user").at(-1);
    return {
      content: `I'm having trouble connecting right now. ${
        lastUserMessage
          ? "I received your message and will follow up shortly."
          : "Please try again in a moment."
      }`,
      role: "assistant",
      degraded: true,
    };
  }
 
  private recordSuccess(provider: ProviderName): void {
    const breaker = this.circuitBreakers.get(provider);
    breaker?.recordSuccess();
  }
 
  private recordFailure(provider: ProviderName, err: Error): void {
    const breaker = this.circuitBreakers.get(provider);
    breaker?.recordFailure(err);
  }
}

The getHealthyProviders() method checks circuit breakers before routing. Failed providers stay out of rotation until their circuit breaker decides they're safe to retry.

Circuit breakers: the pattern that prevents cascades

A circuit breaker stops you from hammering a failed provider, which matters because every failed request to a down provider is a request that could have gone to a healthy one. Without circuit breakers, your fallback logic may spend 5 seconds timing out against Anthropic before routing to OpenAI, for every single request, throughout a 6-hour outage.

type CircuitState = "closed" | "open" | "half-open";
 
class CircuitBreaker {
  private state: CircuitState = "closed";
  private failureCount = 0;
  private lastFailureTime: number | null = null;
  private openedAt: number | null = null;
 
  private readonly failureThreshold = 3;
  private readonly recoveryWindowMs = 30_000; // 30 seconds
  private readonly halfOpenTimeoutMs = 60_000; // 1 minute
 
  constructor(private readonly providerName: string) {}
 
  isOpen(): boolean {
    if (this.state === "open") {
      const now = Date.now();
      if (this.openedAt && now - this.openedAt > this.halfOpenTimeoutMs) {
        this.state = "half-open";
        return false; // allow one test request
      }
      return true;
    }
    return false;
  }
 
  recordSuccess(): void {
    this.failureCount = 0;
    this.lastFailureTime = null;
    if (this.state === "half-open") {
      this.state = "closed";
      this.openedAt = null;
      console.info(`Circuit closed for ${this.providerName} -- recovered`);
    }
  }
 
  recordFailure(err: Error): void {
    const now = Date.now();
 
    // Reset count if last failure was outside recovery window
    if (this.lastFailureTime && now - this.lastFailureTime > this.recoveryWindowMs) {
      this.failureCount = 0;
    }
 
    this.failureCount++;
    this.lastFailureTime = now;
 
    if (this.state === "half-open" || this.failureCount >= this.failureThreshold) {
      this.state = "open";
      this.openedAt = now;
      console.warn(
        `Circuit opened for ${this.providerName}: ${this.failureCount} failures. Error: ${err.message}`
      );
    }
  }
}

The half-open state is the critical piece. After the halfOpenTimeoutMs cooldown, the breaker allows one test request through. If the provider is back, the circuit closes and normal traffic resumes. If the test fails, the circuit opens again for another cooldown interval. This prevents you from prematurely returning traffic to a provider that's still recovering.

The message format problem

The biggest technical challenge in multi-provider failover isn't the routing logic. It's message format compatibility.

Anthropic's API takes {"role": "user", "content": [{"type": "text", "text": "..."}]}. OpenAI takes {"role": "user", "content": "..."}. Google's Gemini API uses {"role": "user", "parts": [{"text": "..."}]}. Tool use schemas differ even more substantially.

The solution is a canonical internal format that your routing layer normalizes to each provider's schema before sending:

interface CanonicalMessage {
  role: "system" | "user" | "assistant" | "tool";
  content: string;
  toolCallId?: string;
  toolName?: string;
  toolResult?: string;
}
 
function toAnthropicMessages(messages: CanonicalMessage[]): AnthropicMessage[] {
  const system = messages.find((m) => m.role === "system")?.content;
  const conversation = messages
    .filter((m) => m.role !== "system")
    .map((m) => ({
      role: m.role === "tool" ? "user" : m.role,
      content:
        m.role === "tool"
          ? [{ type: "tool_result", tool_use_id: m.toolCallId, content: m.toolResult }]
          : [{ type: "text", text: m.content }],
    }));
  return { system, messages: conversation };
}
 
function toOpenAIMessages(messages: CanonicalMessage[]): OpenAIMessage[] {
  return messages.map((m) => ({
    role: m.role === "tool" ? "tool" : m.role,
    content: m.content,
    ...(m.role === "tool" && { tool_call_id: m.toolCallId }),
  }));
}

Build these normalizers once, test them thoroughly, and the routing layer becomes simple: pick a healthy provider, normalize the message format, call the API. The rest of your agent code never knows which provider handled the request.

Deploy Gate

Pre-deploy quality checks

Score > 80%

92%

Latency < 500ms

234ms

Error Rate < 2%

3.1%

Deploy Blocked

Monitoring provider health in production

Failover logic is inert until something goes wrong. What you need from your monitoring stack is enough signal to know when circuits are opening, how often you're hitting the fallback tier, and how provider quality compares across the fleet.

import Chanl from "@chanl/sdk";
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
// Register quality gates for the provider routing layer
chanl.qualityGates.register({
  name: "llm-provider-health",
  checks: [
    {
      name: "primary_error_rate",
      metric: "provider.anthropic.error_rate_5m",
      threshold: 0.03, // alert if primary error rate exceeds 3%
      severity: "warning",
    },
    {
      name: "failover_rate",
      metric: "provider.failover_activations_per_hour",
      threshold: 5, // alert if failing over more than 5 times/hour
      severity: "critical",
      message: "Provider failover activating frequently -- check primary provider status",
    },
    {
      name: "all_providers_degraded",
      metric: "provider.healthy_count",
      threshold: 1, // alert if fewer than 1 healthy provider
      comparison: "less_than",
      severity: "critical",
      message: "All configured LLM providers are degraded",
    },
  ],
  alertChannels: ["slack:#agent-alerts", "pagerduty:cx-oncall"],
});
 
// Record provider outcomes on every request
export async function recordProviderOutcome(
  provider: ProviderName,
  outcome: "success" | "error" | "timeout" | "degraded",
  latencyMs: number
): Promise<void> {
  await chanl.analytics.track({
    event: "provider.request",
    properties: {
      provider,
      outcome,
      latencyMs,
      timestamp: new Date().toISOString(),
    },
  });
}

The failover_activations_per_hour metric is the one to watch. One or two failovers per day is normal noise. Five per hour means your primary provider is in trouble and you should check their status page.

The Chanl monitoring dashboard groups these provider metrics alongside your agent quality scores, so you can see the moment a provider issue starts degrading customer experience rather than discovering it through support tickets.

Testing your failover before you need it

Most teams discover their failover logic is broken during an actual outage. The better approach is deliberate testing that confirms the failover path works before you rely on it.

Chaos testing involves intentionally routing requests to a broken endpoint and verifying the fallback chain activates:

import { describe, it, expect } from "vitest";
import { ProviderRouter } from "./provider-router";
import { MockBrokenClient, MockWorkingClient } from "./test-helpers";
 
describe("ProviderRouter failover", () => {
  it("routes to secondary when primary is down", async () => {
    const router = new ProviderRouter([
      { name: "anthropic", client: new MockBrokenClient(), priority: 1, timeoutMs: 1000 },
      { name: "openai", client: new MockWorkingClient("openai-response"), priority: 2, timeoutMs: 5000 },
    ]);
 
    const result = await router.complete([{ role: "user", content: "Hello" }]);
    expect(result.content).toBe("openai-response");
  });
 
  it("opens circuit after threshold failures", async () => {
    const router = new ProviderRouter([
      { name: "anthropic", client: new MockBrokenClient(), priority: 1, timeoutMs: 500 },
      { name: "openai", client: new MockWorkingClient("openai-response"), priority: 2, timeoutMs: 5000 },
    ]);
 
    // First 3 requests trip the circuit breaker
    for (let i = 0; i < 3; i++) {
      await router.complete([{ role: "user", content: "Test" }]);
    }
 
    // After circuit opens, requests should not even attempt primary
    const startTime = Date.now();
    await router.complete([{ role: "user", content: "Test" }]);
    const elapsed = Date.now() - startTime;
 
    // Should be fast because it skips the timeout on the broken primary
    expect(elapsed).toBeLessThan(500);
  });
 
  it("returns graceful degradation when all providers fail", async () => {
    const router = new ProviderRouter([
      { name: "anthropic", client: new MockBrokenClient(), priority: 1, timeoutMs: 100 },
      { name: "openai", client: new MockBrokenClient(), priority: 2, timeoutMs: 100 },
    ]);
 
    const result = await router.complete([{ role: "user", content: "Hello" }]);
    expect(result.degraded).toBe(true);
  });
});

Run these tests in CI so a refactor of your routing layer can't silently break the failover path. A passing unit test suite isn't a guarantee of production reliability, but it ensures the logic hasn't regressed since the last time someone touched that code.

Load testing across the failover boundary is the other half. Use a tool like k6 or Artillery to send traffic to your agent at production-equivalent rates while a chaos proxy drops 100% of requests to the primary provider. Verify that your p95 latency doesn't blow out, that the failover activates within your acceptable window, and that quality scores from your evaluators don't crash. We covered the full circuit breaker pattern and load testing setup in the circuit breakers for AI agents article.

Graceful degradation messaging

When all providers fail, you still have a choice about what customers experience. The worst outcome is a generic 500 error or a silent failure. The best outcome is a message that acknowledges the situation, gives the customer a reasonable expectation, and preserves the relationship.

For voice agents, graceful degradation means a scripted acknowledgment and a callback offer: "I'm having a technical issue right now. Can I have someone call you back within the hour?" For chat agents, it means an honest message with a contact alternative: "I'm running into a connection issue. You can reach our team directly at..."

Build the degradation response into your routing layer as a first-class output, not as an afterthought. Give it an explicit degraded: true flag so your monitoring can track how often customers hit the degraded state. If degraded responses climb above a threshold, that's an alert condition, not just a log line.

The operational reality

Building failover into your agent architecture adds two to four days of engineering work: the routing abstraction, circuit breakers, message normalizers, monitoring, and tests. That's a one-time investment.

The alternative is a team on incident call at 2am, manually rerouting traffic or building the failover layer under pressure while customers are hitting errors. The 1,200 missed interactions from the April outage example took three weeks of customer recovery work. The engineering investment pays back quickly.

Start with dual-provider active-passive. Add circuit breakers. Instrument the routing layer with your monitoring stack. Run chaos tests in CI. The full implementation described here, including the normalizer layer, is a week of work for a small team. That week of work is what separates a production-grade agent from one that's one cloud incident away from a customer experience crisis.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed