Context engineering: why your agent gets dumber mid-conversation

Your support agent handled the first three turns flawlessly. The customer said their package hadn't arrived. The agent confirmed the order number, pulled up the tracking record, and explained that the shipment was delayed at a regional distribution hub.

Then on turn four, the customer asked a follow-up question. The agent asked them to confirm their order number again.

Nothing broke. Nothing hallucinated. The agent just stopped knowing what it knew four turns ago. The customer, who'd already given that number once, had to give it again. The conversation that had been going well hit a wall -- not because the model failed, but because the information the model needed wasn't there anymore.

This is context rot. It's not a model problem. It's a context engineering failure. And if you build CX agents without deliberately designing your context pipeline, you're hitting it constantly without realizing it.

What context engineering actually is

Context engineering is the discipline of deciding what goes into your agent's context window on each turn, in what order, and what leaves when the window starts to fill. Prompt engineering is a subset of it: what you write in your system instructions. Context engineering is everything else.

The LLM doesn't see "your agent." On every single turn, it sees a block of text -- structured as a conversation -- that appears fresh. It has no memory between requests unless you provide one. It doesn't know what was in the last turn's context unless you include that turn in the current one. The only thing it knows is what's in the current context window right now.

That means every design decision about what appears in that window is a context engineering decision. Which customer history to retrieve. How much conversation to keep verbatim. When to summarize tool outputs rather than include them in full. Whether to track task progress explicitly or rely on the model to reconstruct it from history.

According to a 2026 survey, 82% of AI teams agree that prompt engineering alone is no longer sufficient for production systems at scale. The teams outperforming on agent quality aren't just writing better prompts. They've invested in context pipelines.

Prompt engineering asks: what should I tell the model? Context engineering asks: what should the model be able to see, in what order, to do its job reliably at turn 1 and also at turn 20?

The pipeline runs on every single turn. When it's been designed well, the model sees exactly what it needs and nothing more. When it's been left as an implicit default -- append all turns, include all outputs, hope the model keeps track -- the window grows until critical early information is too diluted to influence outputs.

How context rot works in practice

Context rot is the measurable decline in agent quality as the context window fills up. It's not a cliff where the model suddenly fails when you hit a token limit. It's a gradual degradation that starts much earlier than most developers expect -- often past turn 6 or 7 in a typical support conversation.

LLMs attend to recent tokens more strongly than older ones. This isn't a bug; it's how attention mechanisms in transformers work. Information from turn 1 competes against everything from turns 2 through 15 for the model's attention when generating turn 16. If that early information hasn't been referenced or reinforced, it effectively fades from the model's working consideration.

In a CX context, this creates specific, recognizable failure patterns:

The agent confirms a customer's name in turn 1, then switches to "you" from turn 6 onward because the name has been pushed back far enough that it stops influencing outputs strongly. The agent commits to processing a refund in turn 3, then hedges that commitment in turn 9 because the exchange is buried under six turns of troubleshooting dialogue. The agent runs a CRM lookup in turn 2, then runs the exact same lookup in turn 11 because it's lost track of having already done it.

None of these look like failures on a single-turn eval. The model is generating coherent, on-topic text. But the agent is behaving incoherently across the conversation arc, and customers feel it as a real degradation in service quality.

// Pull turn-level coherence score from conversation trace
const trace = await chanl.calls.getTrace(callId);
const turns = trace.turns;
 
// Check whether confirmed facts persist through the later turns
const confirmedFacts = extractConfirmedFacts(turns[0], turns[1]);
const lateCoherenceScore = measureFactPersistence(confirmedFacts, turns.slice(6));
 
if (lateCoherenceScore < 0.70) {
  console.warn("Context rot detected: confirmed facts not persisting past turn 6");
  // Flag this session for context pipeline review
}

The fix isn't a bigger context window. Context windows in 2026 are large -- Claude 3.5 Sonnet supports 200K tokens, Gemini 1.5 Pro supports 1M. But attention dilution is a function of where information sits in the window relative to how full it is, not just whether it fits. You need to engineer the context so that critical information stays near the front of the window or gets refreshed explicitly, rather than hoping it survives 15 turns of dilution.

The five layers of the context pipeline

The context window your agent assembles on every turn has five distinct layers. Each needs its own management strategy. Most teams have thought carefully about only one of them.

Layer 1: The system prompt. Static instructions, persona definition, tool list, core constraints. This is where almost all prompt engineering effort goes. It should be cached on every call (not reprocessed from scratch), and trimmed as tightly as possible without losing essential instructions. We covered how to audit and compress system prompts in depth in how system prompt token bloat costs you on every agent call.

Layer 2: Retrieved memory. Relevant facts about this specific customer, fetched at the start of each turn. This is not "load all recent interactions." It's "retrieve the subset semantically relevant to what the customer just said." A customer asking about a delayed package gets their recent order interactions retrieved, not their full 3-year interaction history. Recency isn't the signal; relevance is.

Layer 3: Conversation history. The current dialogue turns. This is the primary source of context rot. Most agents append every turn in full, indefinitely. A smarter approach: keep only the last 4-6 turns verbatim, and replace older turns with a running summary that preserves commitments and confirmed facts rather than conversational filler.

Layer 4: Tool outputs. Results from function calls made during the conversation. Verbatim tool outputs are often extremely long -- a full order record, a product catalog response, a customer account summary. Most of those fields aren't relevant to the current question. Summarize tool outputs before they go into the history, keeping only the fields that matter for this turn.

Layer 5: Working memory (task state). An explicit, structured record of what has been established in this conversation: customer identity confirmed, issue diagnosed, commitments made, next action determined. This gets injected fresh at the start of every turn, so the model doesn't have to reconstruct state from history.

The first layer is where most teams spend nearly all their context effort. The last four layers are where most of the context rot actually happens.

Context strategies that actually work

Three strategies have the clearest quality impact for CX agents. I'll walk through them from simplest to implement to most involved.

Strategy 1: Semantic memory retrieval. Instead of loading the most recent N interactions by date, retrieve the interactions most semantically similar to the current message.

A customer asking "what about my return?" should trigger retrieval of return-related interactions, not just the chronologically most recent ones. Vector search over the customer's interaction history finds relevance without the token cost of loading everything. In practice, this means embedding each past interaction at storage time and doing a similarity search against the current message at retrieval time.

Chanl's memory system works this way. Past interactions are embedded when stored, and retrieval at each turn uses semantic similarity rather than recency alone. The context for each turn becomes topically dense -- it contains the interactions most relevant to what the customer is currently asking about, not just the most recent ones.

// Retrieval based on relevance to current message, not just recency
const relevantMemory = await chanl.memory.search({
  customerId: "C123",
  query: currentMessage,   // what the customer just said
  limit: 5,                // top 5 most relevant past interactions
  minSimilarity: 0.72,     // filter out low-relevance matches
});
 
// These 5 entries go into the context window for this turn
// Not the last 5 entries by date -- the most semantically relevant 5

Memory Search

Semantic recall across sessions

budget approval

Discussed upgrading to Business plan...

92%

Mar 1, 2026

Budget approved at $50k for Q2...

87%

Feb 27, 2026

Follow up scheduled for Tuesday...

74%

Feb 25, 2026

Filters

All AgentsLast 30 daysHigh relevance

Strategy 2: Progressive conversation summarization. After N turns, replace the oldest turns with a compressed summary. The summary preserves what was established (confirmed order number, issue type, commitments made) without the full conversational overhead of recreating every exchange.

The critical rule: explicit commitments and confirmed facts must survive in the summary. "We discussed the order" is not a useful summary. "Customer confirmed order #ORD-28847. Agent confirmed delivery delay. Agent offered 10% discount on next order, which customer accepted." is a useful summary. The model needs to know what was decided, not just what was discussed.

async function buildConversationHistory(
  turns: ConversationTurn[],
  llm: LLMClient
): Promise<string[]> {
  if (turns.length <= 6) {
    return turns.map(t => formatTurn(t));
  }
 
  // Summarize everything except the last 4 turns
  const toSummarize = turns.slice(0, -4);
  const recentTurns = turns.slice(-4);
 
  const summary = await llm.complete(`
    Summarize this conversation excerpt. Preserve all of:
    - Confirmed facts (names, order numbers, account details)
    - Explicit commitments made by the agent
    - The current issue type and status
    Do not include conversational filler. Be specific.
    
    ${toSummarize.map(formatTurn).join('\n')}
  `);
 
  return [
    `[Conversation summary: ${summary}]`,
    ...recentTurns.map(t => formatTurn(t)),
  ];
}

Strategy 3: Explicit task state injection. This is the most impactful and the most underused. Rather than expecting the model to track task state by reading conversation history, you maintain a structured state object and inject it at the top of every turn.

interface CXTaskState {
  customerIdentified: boolean;
  customerName?: string;
  orderReferenced?: string;
  issueType?: string;
  issueResolved: boolean;
  commitmentsMade: string[];
  nextAction?: string;
}
 
function buildTaskStateBlock(state: CXTaskState): string {
  return `## Current conversation state
Customer identified: ${state.customerIdentified ? `yes (${state.customerName})` : 'no'}
Order referenced: ${state.orderReferenced ?? 'none yet'}
Issue: ${state.issueType ?? 'not yet determined'}
Resolved: ${state.issueResolved ? 'yes' : 'no'}
Commitments made: ${state.commitmentsMade.length > 0 ? state.commitmentsMade.join('; ') : 'none'}
Next action: ${state.nextAction ?? 'clarify issue'}`;
}
 
// Injected at the top of the context on every single turn
// Small (100-150 tokens) and always current
const messages = [
  { role: "system", content: SYSTEM_PROMPT },
  { role: "user", content: buildTaskStateBlock(taskState) }, // refreshed every turn
  ...compressedHistory,
  { role: "user", content: currentMessage },
];

The task state block is 100-150 tokens. It's always current because you update it after each turn based on what the agent said and what happened. The model doesn't have to reconstruct what it knows from a 12-turn conversation -- it's told the current state directly at the start of each turn. Context rot becomes much harder when the thing most likely to be forgotten is also the thing refreshed most recently.

Putting it together end to end

Let me trace through what a well-engineered context pipeline looks like for a real support conversation, turn by turn.

Turn 1. Customer: "My package hasn't arrived." Pipeline: load system prompt (cached), no prior memory for a new contact, empty task state, single-turn history. Total context: ~3,000 tokens.

Turn 2. Agent asks for the order number. Customer provides #ORD-28847. Pipeline update: task state records orderReferenced: "ORD-28847". Memory retrieves past orders matching this customer. Tool lookup returns the order record -- the pipeline summarizes the 2,400-token record to the 5 relevant fields (status, location, expected delivery, carrier, tracking number). Total context: ~4,200 tokens.

Turn 5. Agent confirms the delay and offers a 10% discount code. Pipeline update: task state records commitmentsMade: ["10% discount DC-7834"]. This commitment now appears in the state block at the top of every subsequent turn. It cannot be forgotten.

Turn 9. Customer asks a tangential question. Progressive summarization has compressed turns 1-5 into a 4-sentence summary. Turns 6-9 remain verbatim. Task state is fully current. The model knows the customer's name, order number, issue, and the commitment made -- all injected fresh this turn.

Turn 14. Customer is satisfied and wrapping up. Task state shows issueResolved: true. The context is lean (summary + 4 recent turns + fresh task state), coherent, and accurate. Total context: ~3,800 tokens -- roughly the same as turn 2.

The customer doesn't repeat themselves. The agent doesn't contradict its earlier commitment. The conversation holds together through 14 turns because the context was engineered, not just appended.

For teams that want to see where their current agent's context goes wrong, Chanl's analytics dashboard shows the full context assembled at each turn -- including what memory was retrieved, what task state looked like, and which turns got compressed. The monitoring view surfaces sessions where coherence scoring degraded across turns, which is where your context pipeline needs work.

You can also find a related deep-dive on persistent memory specifically for voice agent architectures in how voice agents build memory with Pipecat and LiveKit. The memory layer described there works as Layer 2 in the pipeline we've covered here.

Context rot is not a model limitation. It's a context engineering failure. The models are capable of holding coherent long conversations -- if you give them what they need to do it.

El briefing de Signal

Un email por semana. Cómo los equipos líderes de CS, ingresos e IA están convirtiendo conversaciones en decisiones. Benchmarks, playbooks y lo que funciona en producción.

Subscribe

500+ líderes de CS e ingresos suscritos