Build Context Editing Into Your Agent and Cut Token Use 84%

A customer is forty turns into a billing dispute with your support agent. They've explained the charge, read out an order number, corrected the agent once, and uploaded a screenshot the agent looked up against three internal systems. Every one of those lookups is still sitting in the context window, verbatim, being resent to the model on every single turn.

By turn forty the agent is slow. Each response costs more than the last. And somewhere around turn thirty-five it stopped reliably remembering the order number the customer gave at turn four, because that detail is now buried under a wall of stale API responses the model has to read past every time it thinks.

Nothing crashed. The conversation just quietly got expensive and a little dumber, and the customer noticed the second part before you noticed the first.

This is the default behavior of almost every agent in production right now, and it has a fix that's both cheap to implement and startlingly effective. Anthropic's published benchmark on a 100-turn task showed an 84% reduction in token usage from the technique alone. Here's how it works and how to build it.

Why Do Long Agent Conversations Get Expensive and Dumb?

Long conversations get expensive because the entire history is resent to the model on every turn, and most of that history is stale tool output the agent already acted on. Cost grows with each turn even when the useful information doesn't. Worse, the stale content dilutes the model's attention, so accuracy on details from early turns degrades as the window fills.

A stateless LLM has no memory of previous turns. To keep a conversation coherent, your application resends the full transcript every time: system prompt, every user message, every assistant reply, and critically, every tool call and every tool result. That last category is the problem. A single database lookup or a knowledge base query can return thousands of tokens. The agent reads the result, acts on it, and moves on, but the raw result stays pinned in the window forever.

Multiply that across a forty-turn support call with a dozen tool calls and you're resending tens of thousands of tokens of dead weight on every response. You pay for those tokens each turn, and the model spends real attention scanning past them.

The article on why the context window is RAM, not storage makes the mental model concrete: the window is working memory, not an archive. Treating it like an archive is exactly how you run out of room. This article is about the mechanism that keeps it clean.

What Is Context Editing?

Context editing is the automatic removal of stale content from the context window before the model processes a turn. The most common and safest target is old tool-use and tool-result pairs. Once the agent has acted on a lookup, the raw result is dead weight. Editing clears it, keeping the window lean without losing the conversational thread.

Anthropic ships this as a server-side feature. The context editing API clears old tool-use and result pairs and thinking blocks before the request reaches the model, so the cleared tokens never hit your bill and your client keeps the full unmodified history. By default it fires once the prompt passes 100,000 input tokens and preserves the three most recent tool results. In their 100-turn web search evaluation, it cut token consumption by 84% and let agents finish tasks that would otherwise have hit the context ceiling and failed outright.

You don't have to use the managed version to get the benefit. The core idea is simple enough to build yourself, and building a basic version first is the fastest way to understand what the managed one is doing for you.

Context editing is one of four moves in the wider context engineering playbook. LangChain's write, select, compress, isolate split is the common map, and it helps to know where editing sits before you build it.

Context editing is the sharp edge of "compress," and it's the one with the highest payoff for the least effort, because clearing a stale tool result loses nothing you actually needed.

How Do You Build Context Editing Yourself?

You build it by walking the message history, finding tool-result pairs that are older than a threshold, and replacing their content with a short placeholder. The assistant's turn that acted on the result stays, so the conversational logic is intact. Only the raw payload the model no longer needs gets cleared.

Here's a minimal version that clears tool results older than a set number of turns.

interface Message {
  role: 'user' | 'assistant' | 'tool';
  content: string;
  toolResult?: boolean;
  turn: number;
}
 
function editContext(
  messages: Message[],
  currentTurn: number,
  keepToolResultsForTurns = 3
): Message[] {
  return messages.map((msg) => {
    const isStaleToolResult =
      msg.toolResult &&
      currentTurn - msg.turn > keepToolResultsForTurns;
 
    if (isStaleToolResult) {
      return {
        ...msg,
        content: '[tool result cleared to save context]',
      };
    }
 
    return msg;
  });
}

The keepToolResultsForTurns window matters. Set it to zero and you clear results the instant the next turn starts, which is fine for lookups the agent fully consumes in one step. Set it to three and the agent can still refer back to a recent result if a follow-up question depends on it. Three is a reasonable default for customer conversations, where a caller often asks a clarifying question about something the agent just looked up. It's also what Anthropic's managed strategy keeps by default, which is a decent sanity check on the number.

There's a trap here worth naming before you ship it. Clearing content invalidates your prompt cache from the point of the edit onward, and you pay full price to re-encode everything after it on the next request. Clear a few hundred tokens every turn and the cache re-writes will cost more than the editing saves. Anthropic's docs are blunt about this for their own feature, and the managed version compensates by firing rarely and clearing in bulk; there's a clear_at_least parameter for exactly this reason. Copy that behavior: keep the system prompt and earliest turns stable, and clear in batches, not continuously.

Once you're clearing tool noise reliably, the next question is what to do when even the pruned conversation gets long. That's where summarization comes in.

async function compactIfNeeded(
  messages: Message[],
  tokenCount: number,
  threshold = 150_000,
  summarize: (msgs: Message[]) => Promise<string>
): Promise<Message[]> {
  if (tokenCount < threshold) return messages;
 
  const keepRecent = 10;
  const older = messages.slice(0, -keepRecent);
  const recent = messages.slice(-keepRecent);
 
  const summary = await summarize(older);
 
  return [
    { role: 'user', content: `Conversation so far: ${summary}`, turn: 0 },
    ...recent,
  ];
}

Anthropic's compaction API uses a default threshold of 150,000 input tokens with a minimum of 50,000 before it generates a summary of the conversation history. Those are sensible starting numbers. The pattern is the same whether you use the managed version or roll your own: clear tool noise continuously, and summarize conversational history only when the window approaches its ceiling.

When Should You Clear Context Versus Summarize It?

Clear a tool result when the agent has already extracted everything it needs from it, which is most database lookups, API calls, and knowledge base queries. Summarize conversational turns when the raw exchange still carries intent or nuance you can't lose, but don't need word for word. Clearing is cheaper and lossless for tool noise; summarizing is for the human parts of the conversation.

The distinction lines up with where information lives. A tool result is data the agent transformed into an action or an answer. Once that transformation happened, the raw data is redundant with the assistant's turn that used it. Clearing it costs you nothing.

A stretch of back-and-forth where the customer explains what went wrong is different. There's no single downstream turn that captured all of it. If you clear it, you lose the thread. If you keep it verbatim across fifty more turns, you pay for it fifty more times. Summarizing is the middle path: compress the establishing conversation into a few sentences of intent, and keep the recent turns raw so the agent stays responsive.

It's a lossy path, though. The model doing the summarizing decides what mattered, and it's sometimes wrong. When a detail is load-bearing, an order number, a promised refund amount, write it to memory instead of trusting a summary to carry it.

Get the two jobs backwards and you'll either lose the plot or keep paying for lookups nobody will read again.

What Is the Memory Tool and How Is It Different?

The memory tool persists facts to durable storage that survives after the conversation ends, while context editing only manages the live window during one conversation. Editing keeps a single session lean. Memory carries knowledge from one session into the next. A production agent uses both: editing so today's call stays cheap, memory so tomorrow's call starts informed.

Anthropic's memory tool, now generally available on the Messages API, gives the agent a file directory it can read from and write to across sessions. The agent makes tool calls to create, read, update, and delete memory files, and your application executes those operations locally. The window empties when the session ends, but the memory files persist, so the next conversation can start with what the agent already learned.

Here's the shape of the loop. The agent decides what's worth remembering and writes it; on the next session, it reads before it responds.

interface MemoryStore {
  read(path: string): Promise<string | null>;
  write(path: string, content: string): Promise<void>;
}
 
async function handleMemoryCall(
  store: MemoryStore,
  call: { op: 'read' | 'write'; path: string; content?: string }
): Promise<string> {
  if (call.op === 'read') {
    const value = await store.read(call.path);
    return value ?? '[no memory at this path]';
  }
 
  await store.write(call.path, call.content ?? '');
  return '[memory saved]';
}
 
// At the start of a new session, load what the agent knew.
async function primeSession(
  store: MemoryStore,
  customerId: string
): Promise<string> {
  const profile = await store.read(`customers/${customerId}/profile.md`);
  return profile ?? '';
}

The two techniques stack. On Anthropic's 100-turn benchmark, editing alone improved task performance by 29%; adding the memory tool pushed that to 39% over baseline, on top of the 84% token savings. Editing solves cost and window pressure; memory solves continuity. For a support agent, that's the difference between a cheap call and a call that also remembers the customer from last week.

Customer Memory

4 memories recalled

Sarah Chen

Premium

Last call

2 days ago

Prefers

Email follow-up

Session Memory

“Discussed upgrading to Business plan. Budget approved at $50k. Follow up next Tuesday.”

85% relevance

If you want the deeper split between session-scoped context and long-term knowledge, the article on session context versus long-term knowledge breaks down what belongs in each. The short version: the window is for now, memory is for later, and confusing the two is how agents either forget the caller or drown in their own history.

How Does This Play Out in Customer Experience?

Back to the billing dispute from the top of this article. With editing on, the three internal lookups the agent ran get cleared once it acted on them. By turn forty the window holds the conversation, not forty turns of stale JSON, so the agent is as fast on the last turn as the first and still remembers the order number from turn four. The customer never sees the machinery. They just notice the agent kept up.

Now imagine that customer calls back next week. Without memory, they start from zero and re-explain the whole dispute. With the memory tool, the agent primed the session from a profile it wrote last time: the order number, the disputed charge, the resolution offered. The conversation opens with "I see we were working on the charge from the fourteenth" instead of "How can I help you today?"

That continuity is what turns a competent agent into one customers trust. This is squarely the Build and Monitor side of shipping agents for customer experience: give the agent durable memory, then watch what it costs and how well it holds the thread. The context window crisis article covers the token budgeting side in more depth if you're sizing this for a high-volume deployment.

How Do You Know Context Editing Is Actually Working?

Watch one number: input tokens per turn across the length of a conversation. Without editing it's a rising line, because the full history is resent every turn. With editing it stays roughly flat as the conversation grows. The gap between those two curves, priced out over your call volume, is your real savings, not a benchmark from someone else's workload.

Watch it in production, not just in a test. Real customer conversations are longer and messier than anything you'll script, and a regression here doesn't throw an error. It just quietly puts the curve back on a slope.

The instrumentation is not exotic. Every major model API returns a usage block on each response with the input token count, so log that number per turn, tagged with the session ID, and chart it. That's the whole telemetry. Chanl's analytics and monitoring cover the conversation side: an alert when a cohort of sessions starts trending long and expensive is usually the first sign editing regressed or a new tool is returning bloated results.

The memory half has its own production check: when a customer comes back, does the agent actually retrieve what it stored? That's a search against the memory store, scoped to the customer.

import { Chanl } from '@chanl/sdk';
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
// When the customer calls back, prime the session from what
// the agent stored about them last time.
async function recallCustomer(customerId: string): Promise<string[]> {
  const { data } = await chanl.memory.search({
    entityType: 'customer',
    entityId: customerId,
    query: 'open disputes and prior resolutions',
    limit: 5,
  });
 
  return data?.memories.map((m) => m.content) ?? [];
}

The token curve tells you editing is holding cost down. The memory.search call tells you memory is holding continuity up: it's the production companion to the memory tool, retrieving what the agent stored so the next conversation starts informed instead of from zero.

The Lean Agent at Turn Forty

One more run at that dispute. Same customer, same forty turns, same dozen lookups. This time the context editor cleared each tool result once the agent acted on it, and around turn thirty it summarized the opening explanation into two sentences of intent. The window at turn forty is about the size it was at turn ten.

The agent is fast. The response cost is flat. It still knows the order number from turn four, because that fact was never buried under stale JSON, and if it mattered long-term the agent wrote it to memory. When the customer calls back next week, the conversation opens where the last one ended.

Nothing crashed at turn forty this time either. The difference is that nothing quietly rotted on the way there. That's the whole game with long agent conversations: keep the window for what's happening now, put the durable facts where they'll survive, and watch the token curve stay flat while the conversation runs as long as the customer needs.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed