Context engineering for reliable CX agents

The week before we launched our first production voice agent, we thought we'd solved the hard problems. Strong model, tuned prompt, knowledge base with solid vector search. Three days in, customers started getting wrong answers to questions the agent had handled perfectly in testing.

We spent two days checking the model version, the prompt diff, the knowledge base sync. Everything matched our test environment. Then we pulled a production trace and looked at the full context the agent was receiving.

The problem was obvious once we saw it. In testing, we'd used clean, small contexts: a system prompt, a single user message, and a tight knowledge snippet. In production, the agent was receiving the customer's full three-year order history (8,000 tokens), a 60 KB policy document, and a CRM object with 40 fields it never used. The model was drowning in irrelevant information and producing confident-sounding responses that were factually wrong.

That experience is what pushed us to build a formal context engineering practice. This article covers what we learned.

What context engineering actually is

Context engineering is the discipline of systematically deciding what information your AI agent receives, when it receives it, and how it's structured. It's the layer between your data and your model, and it has more leverage over production agent quality than almost any other factor.

It's not the same as prompt engineering. Prompt engineering focuses on the quality of instructions: how to phrase requests so the model responds correctly. Context engineering focuses on information architecture: what data the model has access to, in what order, and in what format.

Both matter. But in CX systems, context engineering has more leverage. A great prompt paired with a bloated, disorganized context will consistently underperform a decent prompt paired with a well-engineered context.

A 2026 analysis of production agent failures found that model limitations accounted for fewer than 20% of errors. Context problems — irrelevant retrieval, context bloat, poor information formatting — accounted for the majority. You can't fix context problems by upgrading the model.

The four strategies that follow each address a specific failure mode in production CX systems.

Write: build a system prompt that grounds the agent

A good system prompt does three things: establishes the agent's role, defines its behavioral constraints, and gives it a mental model of what information it has access to.

Most system prompts do only the first. They describe who the agent is without explaining what it can trust, what tools are available, or how to handle ambiguity. Here's what the difference looks like in practice:

// Weak: identity without grounding
const weakPrompt = `
You are a helpful customer support agent for Acme Corp.
Answer customer questions about orders, billing, and shipping.
Be friendly and professional.
`;
 
// Strong: identity + grounded context model
const strongPrompt = `
You are a customer support agent for Acme Corp.
 
You have access to:
- Account summary: included below, covers the customer's last 90 days
- Order lookup tool: returns full details for a specific order ID
- Policy lookup tool: returns the relevant policy section for a specific topic
- Escalation tool: routes to a human agent with full conversation context
 
Decision rules:
- Never guess about order status, shipping times, or refund eligibility. Use the lookup tools.
- If a customer mentions a specific order, look it up before responding.
- Escalate if you've made 3 tool calls without resolving the core issue.
- If a customer asks about something not covered by your tools, say so clearly.
 
The customer's account summary is provided below.
`;

The stronger version gives the agent a decision tree: when to use tools, when to escalate, what not to guess about. This cuts hallucinations and unnecessary tool calls in a single change.

The Write strategy also covers how you format injected context. When you inject customer history, account data, or retrieved documents, the format matters as much as the content. A 3,000-token JSON dump of order history performs far worse than a 300-token structured summary containing the same essential facts.

// Weak: raw JSON dump
const rawHistory = JSON.stringify(customer.orders); // ~3,200 tokens
 
// Strong: purpose-built summary
const accountSummary = `
Customer: ${customer.name} (since ${customer.memberSince})
Tier: ${customer.tier}
Last 90 days: ${customer.recentOrderCount} orders, $${customer.recentSpend}
Open issues: ${customer.openTickets.map(t => t.summary).join('; ')}
Last contact: ${customer.lastContact.date}, ${customer.lastContact.summary}
`.trim(); // ~80 tokens

That's a 97% reduction in tokens while preserving everything the agent needs for initial triage. Specific order details are available via tool lookup when the customer brings up a specific transaction.

Select: retrieve the right information, not the most similar

Better selection for production CX agents starts with filtering before searching. Instead of embedding the customer's message and scanning your entire knowledge base, filter first by metadata — customer tier, product category, conversation topic — then search semantically within that filtered set. This improves relevance more than tuning k or switching embedding models ever will.

The root problem: semantic similarity doesn't equal relevance. A customer saying "I'm frustrated with my recent purchase" might match 40 documents in your knowledge base. What the agent actually needs is the return policy for the product category the customer bought last month. Those are completely different retrieval targets, and a similarity search with no filters won't reliably find the right one.

Better selection for CX agents starts with metadata filtering. Instead of searching your entire knowledge base, filter first by what you know about the customer and the current topic, then search semantically within that filtered set.

async function retrieveRelevantPolicy(
  query: string,
  customer: CustomerProfile,
  topic: string
): Promise<PolicyChunk[]> {
  return await vectorStore.search({
    query,
    filter: {
      productCategories: customer.recentProductCategories,
      customerTier: customer.tier,
      topicArea: topic,
    },
    topK: 3, // a small k with good filters beats a large k without them
  });
}

The second technique is progressive disclosure. Instead of pre-loading everything that might be relevant, start with a high-level summary and give the agent retrieval tools to fetch details on demand.

const orderDetailsTool = {
  name: "get_order_details",
  description: `
    Retrieve full details for a specific order: line items, shipping status,
    payment method, and any agent notes.
 
    Call when the customer references a specific order ID or needs detailed
    information about a transaction. Do NOT call preemptively — use the
    account summary first and only look up a specific order when needed.
  `.trim(),
  parameters: {
    type: "object",
    properties: {
      order_id: {
        type: "string",
        description: "The order ID referenced by the customer or visible in the account summary."
      }
    },
    required: ["order_id"]
  }
};

The tool description tells the agent when to call it and when not to. This is where tool design and context engineering overlap: a well-written tool description is a form of context that shapes the agent's retrieval behavior.

Memory Search

Semantic recall across sessions

budget approval

Discussed upgrading to Business plan...

92%

Mar 1, 2026

Budget approved at $50k for Q2...

87%

Feb 27, 2026

Follow up scheduled for Tuesday...

74%

Feb 25, 2026

Filters

All AgentsLast 30 daysHigh relevance

For a deeper look at building retrieval agents that make these filtering decisions dynamically, see our guide to agentic RAG for production CX systems.

Compress: represent information efficiently

Compression means representing the same facts in fewer tokens without losing accuracy. For voice agents this is the most direct lever on response latency: every extra 1,000 tokens adds 40 to 100 milliseconds per turn, and a context with 5,000 unnecessary tokens can make a phone conversation feel noticeably halting. Three techniques reduce token footprint in production without reducing information quality.

This is most important for voice agents, but it also matters for chat. Every extra token increases cost, and bloated contexts push important information further from the model's attention window.

Three compression techniques that work well in production:

Rolling summaries for conversation history. As a conversation grows, compress older turns into a summary rather than passing the full transcript.

async function buildConversationContext(
  history: ConversationTurn[],
  model: Model
): Promise<string> {
  if (history.length <= 4) {
    return formatTurns(history);
  }
 
  const older = history.slice(0, -4);
  const recent = history.slice(-4);
 
  const summary = await model.summarize(
    formatTurns(older),
    "Summarize the key facts, customer requests, and decisions from this conversation. Be concise."
  );
 
  return `Earlier: ${summary}\n\n${formatTurns(recent)}`;
}

Passage extraction for documents. When you retrieve a 2,000-token document to answer a specific question, don't inject the whole thing. Extract the relevant sentences.

async function extractRelevantPassage(
  document: string,
  query: string,
  model: Model
): Promise<string> {
  return await model.extract(
    document,
    `Extract only the 2-4 sentences that directly answer: "${query}".
     Return the sentences verbatim without summarizing.`
  );
}

Structured data over prose. Customer attributes, order details, and account metadata should be formatted as structured key-value text, not prose descriptions. The model handles both, but structured text is 2 to 3 times more token-efficient for the same factual content.

Chanl's prompt management tracks context token counts across versions so you can measure whether a new prompt or retrieval strategy actually reduces your token footprint before you deploy it to live traffic.

Isolate: separate concerns so they don't bleed

Isolation means structuring your context with explicit section boundaries so the agent knows what each piece of information is for and when it applies. Without this, information from one part of the context bleeds into unrelated decisions — and the resulting failures are among the hardest to debug because the agent's responses are plausible-sounding even when they're wrong.

The classic failure: inject a customer's complaint history at the top of the context, and the agent routes every conversation toward past grievances even when the current call is about something new. The customer called about a shipping delay. The agent keeps circling back to the billing dispute from four months ago. That's context bleeding, and it can't be fixed by changing the model or the prompt — only by restructuring the context.

function buildIsolatedContext(
  customer: CustomerProfile,
  currentTurn: string,
  retrievedPolicy: string | null
): string {
  return `
## Account Overview
${customer.name}, ${customer.tier} tier, member since ${customer.memberSince}.
 
## Current Conversation Topic
${classifyTopic(currentTurn)}
 
## Recent Activity (last 30 days only)
${customer.recentActivity}
 
## Open Issues
${customer.openTickets.length > 0
  ? customer.openTickets.map(t => `- ${t.summary}`).join('\n')
  : 'None.'}
 
## Policy Reference
${retrievedPolicy ?? 'No policy retrieved. Use the policy lookup tool if needed.'}
`.trim();
}

The section headers give the model explicit anchors. "Recent Activity" doesn't include the full complaint history. "Open Issues" doesn't include resolved tickets. Each section has a defined scope and a defined purpose.

Isolation also applies to tools. If your agent has 12 tools, grouping them by category reduces the decision overhead on each turn:

const toolContext = `
You have three categories of tools:
 
Lookup tools: order_details, customer_profile, policy_lookup
Use these to fetch specific facts before responding.
 
Action tools: create_ticket, update_order, process_refund
Use these when you need to take a concrete action on the customer's behalf.
 
Escalation tools: route_to_human, schedule_callback
Use these when the issue is outside your scope or when the customer requests a human.
`;

This framing means the agent doesn't scan all 12 tool descriptions on every turn. It categorizes the needed action type first, then picks within that category. Less decision overhead, fewer wrong tool calls.

Putting it together: a full context pipeline

Here's how the four strategies combine in a production CX system:

The context assembly step is entirely in your code, before inference. You're not hoping the model figures out what's relevant. You're making those decisions explicitly and measurably.

Chanl's memory and conversation analytics let you track context decisions across every conversation and surface the ones where context bloat correlates with lower resolution rates or higher handle times.

import { Chanl } from '@chanl/sdk';
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
async function handleCustomerMessage(
  customerId: string,
  message: string,
  conversationId: string
): Promise<AgentResponse> {
  const topic = await classifyTopic(message);
  const customer = await getCompressedProfile(customerId);
  const policy = topic ? await retrieveWithFilter(message, topic, customer) : null;
 
  const context = assembleIsolatedContext(customer, message, policy);
 
  await chanl.calls.logContext({
    conversationId,
    tokenCount: countTokens(context),
    topic,
    hasRetrievedPolicy: policy !== null,
  });
 
  return await agent.respond(context, message);
}

Measuring context engineering in production

Context quality is measurable. These four metrics are the most useful ones to track.

Token efficiency ratio. Total context tokens per turn divided by the tokens that appear to have influenced the response. If you're injecting 8,000 tokens and the agent's response reflects 400 of them, your selection strategy is too broad.

Retrieval precision. For each retrieval call, did the retrieved document actually get used in the response? Track this by correlating retrieval events with response content. Precision below 50% means your filters need tightening.

Context-latency curve. Plot context token count against first-token latency for each conversation turn. The relationship is roughly linear. Set a per-turn token budget and alert when you exceed it.

Resolution rate by context size. Compare conversation resolution rates in the bottom and top quartiles of context size. If smaller-context conversations resolve better, you have context bloat that's hurting your customers.

Chanl's conversation monitoring surfaces these metrics in a dashboard tied to real conversation outcomes, so you can see the relationship between context decisions and CSAT scores.

For a broader look at instrumenting your agent pipeline so context problems surface before they reach customers, see the article on the measurement gap in AI agent observability.

Context engineering is a continuous practice

Three weeks after the voice agent launch that opened this article, we had rebuilt the context pipeline with all four strategies in place. The wrong-answer rate dropped by 60 percent. Handle time dropped by 22 percent. We hadn't changed the model or the prompt — only what the model was allowed to see.

The work doesn't end at launch. Customer conversation patterns shift, knowledge bases grow, and new product lines create new context needs. An agent tuned for your current support volume will drift if you stop maintaining its context pipeline. Context engineering is the Build layer of building, connecting, and monitoring AI agents for customer experience — get it right and everything downstream becomes more tractable.

Build these habits into your development cycle:

• Review token counts and retrieval precision weekly
• Re-audit your system prompt each time a major product or policy changes
• Test new context strategies against historical conversations before deploying
• Track the relationship between context size and latency as your knowledge base grows

The article on session context, long-term memory, and knowledge architecture covers how to decide what belongs in short-term conversation context versus long-term memory, which is a key isolation decision for agents that handle repeat customers.

The agents that stay reliable over time are the ones where teams treat context as a first-class engineering concern and measure it the same way they measure latency or error rate.

El briefing de Signal

Un email por semana. Cómo los equipos líderes de CS, ingresos e IA están convirtiendo conversaciones en decisiones. Benchmarks, playbooks y lo que funciona en producción.

Subscribe

500+ líderes de CS e ingresos suscritos