Five things your CX agent needs that its framework won't provide

Three months after your CX agent goes live, customer complaints start coming in. "Your bot forgot everything I told it last week." "I called yesterday and had to explain the whole thing again." You pull the logs. No errors. No exceptions. The framework did exactly what it was built to do. The problem isn't the framework. The problem is the five layers of infrastructure that were never there.

This happens to almost every team that ships a production CX agent. The demo works perfectly. The conversation loop is clean. Tool calls return the right data. Then customers start experiencing the agent the way you never did during development: across multiple sessions, across channels, under the kind of failure conditions you can't simulate in a Jupyter notebook.

LLM frameworks are excellent at the conversation layer. They are not infrastructure. Understanding where that line sits, and what you need on the other side of it, is the whole game.

What LLM frameworks actually give you (and what they don't)

LLM frameworks solve a real, hard problem: they handle the mechanics of getting a message to a model, managing tool call syntax, routing responses, and streaming output back to the user. That's the conversation layer, and they handle it well.

The frameworks in the left column are genuinely good. VAPI and Retell handle voice orchestration reliably. LangChain and the Vercel AI SDK make multi-step reasoning and tool use tractable. The right column is not a criticism of those tools. It's a description of what they were never designed to do.

Every cell in that right column is something your production CX agent needs from day one. If you don't have them, you find out from your customers, not from your monitoring.

Persistent memory

The agent doesn't remember customers between sessions because it has no place to store what it learned. Session state solves the wrong problem. You need memory that persists across every conversation a customer has ever had with your agent, and a mechanism for injecting the relevant parts of that history into a new conversation's context window.

Session state is "what did we discuss in this conversation." Persistent memory is "who is this person and what do I already know about them." Those are different systems serving different purposes.

Here's a simplified memory store that shows the core pattern. The retrieval step does a semantic similarity search so you're injecting relevant memories, not just the most recent ones:

interface Memory {
  id: string;
  customerId: string;
  content: string;
  embedding: number[];
  createdAt: Date;
  tags: string[];
}
 
interface MemoryStore {
  save(customerId: string, content: string, tags?: string[]): Promise<void>;
  retrieve(customerId: string, query: string, limit?: number): Promise<Memory[]>;
  injectContext(customerId: string, currentQuery: string): Promise<string>;
}
 
// Cosine similarity between two embedding vectors
function cosineSimilarity(a: number[], b: number[]): number {
  const dot = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
  return dot / (magA * magB);
}
 
async function createMemoryStore(db: Database, embedder: Embedder): Promise<MemoryStore> {
  return {
    async save(customerId, content, tags = []) {
      const embedding = await embedder.embed(content);
      await db.insert("memories", {
        id: crypto.randomUUID(),
        customerId,
        content,
        embedding,
        tags,
        createdAt: new Date(),
      });
    },
 
    async retrieve(customerId, query, limit = 5) {
      const queryEmbedding = await embedder.embed(query);
      const memories = await db.query<Memory>(
        "SELECT * FROM memories WHERE customer_id = ?",
        [customerId]
      );
      return memories
        .map((m) => ({ ...m, score: cosineSimilarity(queryEmbedding, m.embedding) }))
        .sort((a, b) => b.score - a.score)
        .slice(0, limit);
    },
 
    async injectContext(customerId, currentQuery) {
      const memories = await this.retrieve(customerId, currentQuery);
      if (memories.length === 0) return "";
      const lines = memories.map((m) => `- ${m.content}`).join("\n");
      return `What you already know about this customer:\n${lines}`;
    },
  };
}

You call injectContext before constructing your system prompt. The returned string goes into the prompt preamble, before any instructions. The model receives a grounded picture of who it's talking to. Without this, every conversation is a first meeting.

If you want to go deeper on designing a production memory system, the full build guide covers vector storage selection, eviction strategies, and memory consolidation patterns.

Chanl's memory feature handles the embedding, storage, and retrieval layer so you configure what to remember, not how to store it.

Tool reliability

Tool calls fail in production. They fail 3-8% of the time due to timeouts, network blips, schema mismatches, and downstream APIs having a bad moment. Without retry logic and graceful degradation, each failure strands the agent mid-conversation.

Think about the math. A 20-turn support conversation might involve 8 tool calls: fetch account details, check order status, look up policy, write a note. At a 5% per-call failure rate, the probability of at least one failure across 8 calls is about 33%. One in three conversations hits a tool failure. What happens then depends entirely on whether you've built for it.

A retry wrapper is the minimum. You want configurable backoff, a timeout budget, and a graceful fallback for when retries are exhausted:

interface ToolExecutorOptions {
  maxRetries?: number;
  timeoutMs?: number;
  backoffMs?: number;
  fallback?: () => unknown;
}
 
async function executeWithRetry<T>(
  toolFn: () => Promise<T>,
  toolName: string,
  options: ToolExecutorOptions = {}
): Promise<T | null> {
  const { maxRetries = 3, timeoutMs = 5000, backoffMs = 300, fallback } = options;
 
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const result = await Promise.race([
        toolFn(),
        new Promise<never>((_, reject) =>
          setTimeout(() => reject(new Error(`Tool ${toolName} timed out after ${timeoutMs}ms`)), timeoutMs)
        ),
      ]);
      return result;
    } catch (err) {
      const isLastAttempt = attempt === maxRetries;
      console.warn(`Tool ${toolName} failed (attempt ${attempt}/${maxRetries}):`, err);
 
      if (isLastAttempt) {
        if (fallback) {
          console.info(`Using fallback for tool ${toolName}`);
          return fallback() as T;
        }
        return null;
      }
 
      await new Promise((r) => setTimeout(r, backoffMs * attempt));
    }
  }
 
  return null;
}
 
// Usage: wrap any tool call
const orderDetails = await executeWithRetry(
  () => orderService.getOrder(orderId),
  "get_order",
  {
    maxRetries: 3,
    timeoutMs: 4000,
    fallback: () => ({ status: "unavailable", message: "Order details temporarily unavailable" }),
  }
);

The fallback matters as much as the retry. When retries are exhausted, you don't want the agent to silently receive null and hallucinate a response. The fallback gives the agent something honest to work with: "I wasn't able to retrieve your order status right now."

Graceful degradation is a product decision, not just a code pattern. Decide in advance what each tool failure mode should look like to the customer, and encode it in the fallback. That decision belongs in your agent design, not improvised at runtime.

For a full treatment of tool system design, including schema validation, tool composition, and the tradeoffs between synchronous and async tool calls, see build your own AI agent tool system.

Chanl's tools feature provides managed tool execution with built-in retry, timeout, and schema validation across MCP and OpenAPI integrations.

Connected Integrations12 active

Salesforce

Slack

Google

Stripe

HubSpot

Intercom

Zapier

Shopify

GitHub

Jira

Gmail

PostgreSQL

Pre-production scenario testing

You can't write a unit test for a conversation. Conversations are stateful, multi-turn, and probabilistic. The question "does this agent handle a frustrated customer who provides the wrong account number three times" has no deterministic answer that a unit test can capture.

Scenario testing uses AI-powered simulated users. You define a persona and a goal, the simulator runs realistic conversations against your agent, and you score the outcomes. The value isn't one passing test. It's a regression suite that runs every time you change a prompt, swap a model, or modify a tool.

Here's a minimal scenario runner stub showing the structure:

interface Scenario {
  id: string;
  name: string;
  persona: {
    background: string;
    intent: string;
    emotionalState: "neutral" | "frustrated" | "confused" | "urgent";
  };
  successCriteria: string[];
  maxTurns?: number;
}
 
interface ScenarioResult {
  scenarioId: string;
  passed: boolean;
  turns: number;
  goalAchieved: boolean;
  escalated: boolean;
  criteriaResults: Record<string, boolean>;
  transcript: Array<{ role: "user" | "agent"; content: string }>;
}
 
async function runScenario(
  scenario: Scenario,
  agentEndpoint: string,
  evaluatorModel: string
): Promise<ScenarioResult> {
  const transcript: ScenarioResult["transcript"] = [];
  let turn = 0;
  const maxTurns = scenario.maxTurns ?? 20;
 
  // Simulated user sends the opening message based on the persona
  let userMessage = await generateUserMessage(scenario.persona, transcript, evaluatorModel);
 
  while (turn < maxTurns) {
    transcript.push({ role: "user", content: userMessage });
 
    const agentResponse = await callAgent(agentEndpoint, userMessage, transcript);
    transcript.push({ role: "agent", content: agentResponse });
 
    // Check if the scenario goal is reached or conversation should end
    const evaluation = await evaluateTranscript(
      transcript,
      scenario.successCriteria,
      evaluatorModel
    );
 
    if (evaluation.goalAchieved || evaluation.escalated || turn >= maxTurns - 1) {
      return {
        scenarioId: scenario.id,
        passed: evaluation.criteriaResults
          ? Object.values(evaluation.criteriaResults).every(Boolean)
          : false,
        turns: turn + 1,
        goalAchieved: evaluation.goalAchieved,
        escalated: evaluation.escalated,
        criteriaResults: evaluation.criteriaResults ?? {},
        transcript,
      };
    }
 
    userMessage = await generateUserMessage(scenario.persona, transcript, evaluatorModel);
    turn++;
  }
 
  return {
    scenarioId: scenario.id,
    passed: false,
    turns: maxTurns,
    goalAchieved: false,
    escalated: false,
    criteriaResults: {},
    transcript,
  };
}

The key insight is that the evaluator model is separate from the agent being tested. You're not asking the agent to grade itself. An independent model checks the transcript against the success criteria after the conversation finishes.

Run your scenario suite as part of CI. A prompt change that regresses three previously passing scenarios should not ship.

For a deeper look at testing strategies including adversarial personas, multi-agent simulation, and regression scoring, see voice AI testing strategies that actually work.

Chanl's scenarios feature gives you a library of pre-built personas and criteria, a runner that connects directly to your deployed agent, and a dashboard showing pass rates over time.

Conversation observability

Latency metrics tell you how fast your agent responded. They say nothing about whether the response helped the customer. Conversation observability means scoring every conversation on quality dimensions that actually matter for CX outcomes.

Knowing response latency is under 800ms while your agent is incorrectly telling customers their order was delivered when it wasn't is not useful monitoring. You need quality scores alongside performance metrics.

The quality score is the highest-signal metric and the hardest to collect without tooling. You need a separate evaluator model reading the transcript and scoring it against a rubric. Build that into your pipeline.

Latency matters too, but frame it correctly. For voice agents, time-to-first-token matters for naturalness. For chat, it matters less than accuracy. Track latency by channel and by tool call separately, so a slow CRM lookup doesn't inflate your overall latency metric.

For a full treatment of what to monitor and how to structure alerts, see AI agent observability: what to monitor in production.

Chanl's analytics and monitoring features give you per-conversation quality scores, tool success rate dashboards, and alert rules that trigger on quality drops, not just latency spikes.

Cross-channel session state

A customer calls your support line Monday about a billing dispute. The agent handles it well and opens a ticket. Tuesday, the customer follows up via live chat. The chat agent has no idea the call happened. It asks the customer to re-explain the situation. That customer is now frustrated, and you earned that.

Cross-channel session state means a customer's context travels with them regardless of which channel they contact you on next.

This is different from persistent memory, which is about who the customer is. Cross-channel state is about what issue is currently in flight. The same customer might have three open issues at different stages across different channels. Each agent needs to know the current state of the issue that prompted this particular contact.

type Channel = "voice" | "chat" | "email" | "messaging";
 
interface ChannelEvent {
  channel: Channel;
  sessionId: string;
  timestamp: Date;
  summary: string;
  outcome: "resolved" | "escalated" | "pending" | "abandoned";
  openTicketId?: string;
}
 
interface CrossChannelSession {
  customerId: string;
  issueId: string;
  createdAt: Date;
  updatedAt: Date;
  currentStatus: "open" | "resolved" | "escalated";
  events: ChannelEvent[];
  activeTicketId?: string;
}
 
class SessionManager {
  constructor(private store: SessionStore) {}
 
  async getOrCreateSession(customerId: string, issueId: string): Promise<CrossChannelSession> {
    const existing = await this.store.findSession(customerId, issueId);
    if (existing) return existing;
 
    const session: CrossChannelSession = {
      customerId,
      issueId,
      createdAt: new Date(),
      updatedAt: new Date(),
      currentStatus: "open",
      events: [],
    };
 
    await this.store.saveSession(session);
    return session;
  }
 
  async recordEvent(customerId: string, issueId: string, event: ChannelEvent): Promise<void> {
    const session = await this.getOrCreateSession(customerId, issueId);
    session.events.push(event);
    session.updatedAt = new Date();
 
    if (event.outcome === "resolved") session.currentStatus = "resolved";
    if (event.outcome === "escalated") session.currentStatus = "escalated";
    if (event.openTicketId) session.activeTicketId = event.openTicketId;
 
    await this.store.saveSession(session);
  }
 
  async buildHandoffContext(customerId: string, issueId: string): Promise<string> {
    const session = await this.store.findSession(customerId, issueId);
    if (!session || session.events.length === 0) return "";
 
    const eventSummaries = session.events
      .sort((a, b) => a.timestamp.getTime() - b.timestamp.getTime())
      .map((e) => `${e.channel} (${e.timestamp.toLocaleDateString()}): ${e.summary} [${e.outcome}]`)
      .join("\n");
 
    return [
      `Issue ID: ${issueId}`,
      `Current status: ${session.currentStatus}`,
      session.activeTicketId ? `Active ticket: ${session.activeTicketId}` : "",
      "Contact history:",
      eventSummaries,
    ]
      .filter(Boolean)
      .join("\n");
  }
}

buildHandoffContext returns a string you inject at the start of any new conversation for a customer with an open issue. The agent reads it the way a human agent reads a case summary. It knows what happened, what channel it happened on, and what the current state is.

The issue identification step is worth thinking through carefully. You need a way to match an incoming contact to an existing issue. That might be a ticket number the customer provides, an open issue detected by the CRM lookup, or a recent session that looks related based on the customer's first message.

The agent-native stack

These five infrastructure layers don't replace your framework. They wrap it. Your framework handles the conversation loop at the center. The infrastructure layers make that loop reliable, observable, and capable of serving real customers across real time and real channels.

The framework is in the center because it's doing real work. But what you ship to customers is the whole stack. A framework alone is a demo. A framework with these five layers is an agent you can be responsible for.

Build vs. buy

Build the infrastructure when what you're building is the infrastructure. If you're a platform company selling agent tooling, your memory store and tool reliability layer are your product. Build them.

If you're a CX team, your product is the quality of the customer experience your agent delivers. The memory store is not your product. The retry wrapper is not your product. The scenario runner and the quality scoring pipeline are not your product.

The build-it-yourself tax is substantial. Here's a rough accounting of what you're committing to when you decide to build each layer:

That's 12-20 weeks of initial build for five systems you'll maintain indefinitely. For most CX teams, that's time not spent on the agent behavior, the prompts, the tool integrations, and the conversation design that actually differentiate your customer experience.

The heuristic is simple: if the infrastructure you're considering isn't a competitive advantage for your specific business, you're better off buying it. Use the time you save to make the agent itself better.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed