When vector search fails: GraphRAG for complex CX agent queries

A premium customer calls your support agent. They're asking about a discount they heard about for long-term customers in their state. The answer exists in your knowledge base, but it requires connecting three separate facts: the customer's tier, their join date, and the state-specific promotion rules.

Your vector search returns the generic discount policy document. The agent reads it confidently. The customer gets the wrong answer, because the right answer required joining facts across three documents that aren't similar to each other in embedding space.

This is the multi-hop retrieval problem, and it's where standard RAG fails quietly. GraphRAG is the approach that fixes it.

What GraphRAG actually is

GraphRAG replaces or supplements flat vector search with an explicit knowledge graph. Instead of finding documents that look similar to the query, it traverses relationships between entities to retrieve the connected facts needed to answer correctly.

The building process has three steps:

1. An LLM reads your knowledge base and extracts entities (products, policies, customer tiers, locations, dates) and their relationships (applies-to, requires, supersedes, effective-from).
2. The extracted entities and relationships are organized into a graph structure, with community detection identifying clusters of closely related facts.
3. The LLM generates summary text for each community, giving the query layer a summary-search option for broad questions alongside a graph-traversal option for specific ones.

At query time, your agent sends a question to the retrieval layer. A classifier decides whether it's a simple lookup (use vector search), a multi-hop relational query (traverse the graph), or a broad aggregate question (use community summaries). The graph layer returns the connected facts the agent needs, structured so the model can reason about the relationships rather than just reading chunks.

Microsoft Research introduced the technique in 2024 specifically to address the multi-hop failure mode of standard RAG. Since then it's been integrated into production knowledge systems at companies managing complex policy trees, large product catalogs, and tiered customer programs.

The three CX query types that vector search fails on

The three categories where standard RAG consistently fails are multi-condition policy queries, temporal relationship queries, and hierarchical relationship queries. Understanding each failure mode helps you decide whether GraphRAG is worth the investment for your knowledge base.

Multi-condition policy queries. "What's the return window for premium customers who purchased during the holiday promotion?" requires three lookups: the base return policy, the premium customer tier policy, and the holiday promotion override. Vector search finds each document separately, and the model has to synthesize them. Sometimes it gets this right. Often it uses whichever document had the strongest embedding match and ignores the others.

Temporal relationship queries. "Did the free shipping threshold change after the March 2025 update?" requires knowing that there was a March 2025 policy update, what it changed, and how it relates to the current policy. Documents about policy changes aren't semantically similar to documents about current policies. Vector search doesn't naturally connect them.

Hierarchical relationship queries. "Which support tier handles a customer who is enterprise, bought through a reseller, and has an active professional services contract?" requires traversing an org chart or routing matrix that may never appear in the same document as the query terms. The answer might be four hops from any individual document in the knowledge base.

For simple FAQ-style queries, "what is your refund policy?", vector search works fine. The answer is in one document, the query is semantically similar to that document, retrieval succeeds. GraphRAG provides no benefit here. The art is knowing when to use which path.

Building a knowledge graph from your CX knowledge base

Building a knowledge graph takes three steps: run an LLM over each document to extract entities and relationships, load those tuples into a graph database, and run community detection to cluster related facts into searchable groups. You don't need to design the schema manually. The LLM extraction step creates it from your actual content. Here's the extraction pattern:

import Anthropic from "@anthropic-ai/sdk";
 
interface Entity {
  id: string;
  type: string;   // 'policy', 'product', 'customer_tier', 'promotion', 'location'
  name: string;
  attributes: Record<string, string>;
}
 
interface Relationship {
  from: string;   // entity id
  to: string;     // entity id
  type: string;   // 'applies_to', 'requires', 'supersedes', 'effective_from', 'overrides'
  conditions?: string;   // any conditions on this relationship
}
 
interface ExtractionResult {
  entities: Entity[];
  relationships: Relationship[];
  sourceDocumentId: string;
}
 
const client = new Anthropic();
 
async function extractGraphFromDocument(
  documentId: string,
  documentText: string
): Promise<ExtractionResult> {
  const response = await client.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 4096,
    messages: [
      {
        role: "user",
        content: `Extract entities and relationships from this knowledge base document for a customer support knowledge graph.
 
Entity types to extract: policy, product, customer_tier, promotion, location, time_period, procedure
 
Relationship types to extract: applies_to, requires, supersedes, effective_from, overrides, conditional_on, part_of
 
For each relationship, note any conditions (e.g., "only for enterprise customers", "after 2024-01-01").
 
Document:
${documentText}
 
Respond with JSON matching this structure:
{
  "entities": [{"id": "...", "type": "...", "name": "...", "attributes": {...}}],
  "relationships": [{"from": "...", "to": "...", "type": "...", "conditions": "..."}]
}`,
      },
    ],
  });
 
  const content = response.content[0];
  if (content.type !== "text") throw new Error("Unexpected response type");
 
  const parsed = JSON.parse(content.text);
  return {
    ...parsed,
    sourceDocumentId: documentId,
  };
}

Run this over each document in your knowledge base. The result is a set of entity-relationship tuples you can load into a graph database (Neo4j, Amazon Neptune, or even a simple JSON adjacency list for smaller knowledge bases).

The extraction prompt matters. Telling the model what entity and relationship types to look for produces much cleaner graphs than open-ended extraction. For CX knowledge bases, the entity types above cover most real-world scenarios. Add support_tier, contract_type, or subscription_plan if your knowledge base references those.

The hybrid retrieval pattern

The most practical architecture for production CX agents isn't pure GraphRAG or pure vector search. It's a retrieval router that sends each query to the right path.

type QueryType = "simple_lookup" | "multi_hop" | "aggregate";
 
async function classifyQuery(query: string): Promise<QueryType> {
  const response = await client.messages.create({
    model: "claude-haiku-4-5-20251001",   // fast, cheap classification
    max_tokens: 10,
    messages: [
      {
        role: "user",
        content: `Classify this customer support query. Reply with exactly one word.
 
Query: "${query}"
 
Options:
- simple_lookup: answers in one document, direct question
- multi_hop: requires connecting facts from multiple documents or applying multiple conditions
- aggregate: asks about patterns across many interactions
 
Classification:`,
      },
    ],
  });
 
  const text = (response.content[0] as { text: string }).text.trim().toLowerCase();
  if (text.includes("multi")) return "multi_hop";
  if (text.includes("agg")) return "aggregate";
  return "simple_lookup";
}
 
async function retrieve(query: string, customerId: string): Promise<string> {
  const queryType = await classifyQuery(query);
 
  switch (queryType) {
    case "simple_lookup":
      return await vectorSearch(query);
 
    case "multi_hop":
      return await graphTraversal(query, customerId);
 
    case "aggregate":
      return await communitySearch(query);
  }
}

The classification call uses Claude Haiku for speed and cost. For a correctly structured prompt, this classification is accurate enough to route most queries correctly, with multi-hop being the conservative fallback (graph traversal works for simple queries too, it's just slower and more expensive).

The graph traversal function uses the customer's context to anchor the search:

async function graphTraversal(
  query: string,
  customerId: string
): Promise<string> {
  // 1. Load customer's known attributes from your data layer
  const customer = await getCustomerProfile(customerId);
 
  // 2. Find relevant entity nodes based on customer attributes + query
  const startNodes = await findRelevantNodes(query, {
    tier: customer.tier,
    joinDate: customer.joinDate,
    location: customer.state,
  });
 
  // 3. Traverse the graph from each start node up to 3 hops
  const subgraph = await expandNeighborhood(startNodes, maxHops = 3);
 
  // 4. Extract the text content of all reached nodes and relationships
  const context = formatSubgraphAsText(subgraph);
 
  return context;
}

The expandNeighborhood step is where the multi-hop magic happens. Starting from "premium customer tier" as a node, it follows edges to "premium discount policy," then follows that policy's "effective_from" edge to the date constraint, then follows the "location_override" edge to the California-specific rule. It collects all the text along the way and returns it as structured context.

Memory Search

Semantic recall across sessions

budget approval

Discussed upgrading to Business plan...

92%

Mar 1, 2026

Budget approved at $50k for Q2...

87%

Feb 27, 2026

Follow up scheduled for Tuesday...

74%

Feb 25, 2026

Filters

All AgentsLast 30 daysHigh relevance

The CX knowledge base types that benefit most

Knowledge bases with conditional policy hierarchies, product dependency chains, support routing matrices, and historical policy tracking benefit most from GraphRAG. FAQ-style knowledge bases with self-contained entries see minimal benefit over vector search. The investment in time and ongoing cost is worth it for some knowledge bases and clearly not for others.

High benefit:

• Policy hierarchies with conditional logic. If you have policies that apply differently based on customer tier AND purchase date AND location AND channel, the graph explicitly represents those conditions as edges. Multi-hop queries traverse them accurately.
• Product catalogs with dependency chains. Product A requires Service B, which is only available in regions C and D, which have different pricing than region E. A graph represents these dependencies explicitly.
• Support routing matrices. Escalation paths that depend on multiple attributes (account size, contract type, issue category, time since purchase) are naturally graph-structured.
• Historical policy tracking. When policy version 3 supersedes version 2 for orders after date X but version 2 still applies for orders before date X, the temporal relationships in the graph make this query answerable.

Low benefit:

• FAQ-style knowledge bases. If each entry is self-contained (question, answer, done), there are no meaningful relationships between entries for the graph to traverse. Vector search works fine.
• Product documentation. Technical documentation for a single product usually doesn't require multi-hop reasoning. A user asking "how do I reset my password" needs one document, not a graph traversal.
• Script-based support. If your agent follows a fixed script for each issue type, you don't need retrieval at all. The script is the context.

A quick test for your knowledge base: can you write five questions that require connecting facts from three or more different documents? If yes, GraphRAG is worth evaluating. If you struggle to write those questions, stick with vector search and optimize your chunking strategy instead.

Indexing cost and maintenance

The upfront indexing cost is the main operational concern with GraphRAG. You're running an LLM over every document in your knowledge base to extract entities and relationships. For a typical CX knowledge base:

These estimates assume Claude Haiku for extraction (cheaper than Opus with good enough entity extraction for most use cases). The range reflects document complexity. Simple FAQ documents extract cheaply; dense policy documents with many entities and conditions cost more.

Re-indexing happens when documents change. For policies that update quarterly, this is a minor cost. For live product catalogs that change daily, you need incremental indexing: update only the changed documents and splice their entities into the existing graph, rather than re-indexing everything.

One practical tip: run entity extraction in batches with a 2-second delay between documents. LLM APIs have rate limits, and extraction jobs for large knowledge bases will hit them.

Connecting to your agent

Once the graph is built, the agent uses it through the retrieval layer. For agents using Chanl's memory system, the semantic search layer runs vector search by default. For queries flagged as multi-hop by the classifier, route to the graph layer instead. The underlying memory architecture that makes cross-session knowledge persistence work is described in AI agent memory: session context vs. long-term knowledge.

For teams building from scratch, the agent sees the graph output as plain text context, not as graph structure. The graph traversal function collects entity descriptions and relationship text, formats them into a readable context block, and returns that to the LLM. The model doesn't need to know it's reading from a graph.

async function handleCustomerQuery(
  query: string,
  customerId: string,
  conversationHistory: Message[]
): Promise<string> {
  // Hybrid retrieval based on query type
  const context = await retrieve(query, customerId);
 
  const response = await client.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 2048,
    system: `You are a CX agent for Acme Corp. Use the provided context to answer the customer's question accurately. If the context contains conditional policies, apply the conditions that match the customer's situation.`,
    messages: [
      ...conversationHistory,
      {
        role: "user",
        content: `Customer query: ${query}\n\nRelevant context:\n${context}`,
      },
    ],
  });
 
  return (response.content[0] as { text: string }).text;
}

The context block from the graph traversal reads naturally to the model because it's formatted as descriptive text about entities and their relationships, not as raw graph data. Something like: "Premium tier discount policy: 15% off for customers with account age over 24 months. For California customers: the state promotion adds an additional 5% from April through June 2026, applicable to orders over $100."

Building and connecting a GraphRAG layer sits squarely in Chanl's build and monitor pillars: building means designing the retrieval architecture your agent uses; monitoring means measuring whether retrieval is actually working in production. Before going live, test your graph retrieval against the scenarios framework using synthetic conversations that include multi-hop policy questions. These are the exact queries that break in production if your graph is incomplete or your traversal doesn't reach the right nodes.

For a broader look at how retrieval decisions affect agent quality at each step, the agentic RAG decision patterns guide covers the full retrieval architecture. GraphRAG is the right choice for the multi-hop layer; the guide covers how it fits alongside vector search and structured data queries.

When to invest and when to wait

GraphRAG isn't the right first step for most teams. If you're still figuring out your chunking strategy, testing your vector search recall, or haven't yet encountered the multi-hop failure mode in production, the investment is premature.

The right trigger is when you have documented cases of your agent giving wrong answers to questions that require connecting facts from multiple sources. Those cases are your GraphRAG test set. Run both retrieval approaches against them, measure accuracy, and let the data decide.

The teams who've found GraphRAG most valuable in CX are those managing knowledge bases that reflect the real complexity of their business: tiered products, conditional policies, regional variations, historical promotions. The more your policies sound like database joins, the more your retrieval layer benefits from graph structure.

Learn Agentic AI

Weekly. Patterns for shipping agents that work. MCP, scorecards, regression tests, prompts, model comparisons.

Subscribe

500+ builders subscribed