Testing & Evaluation Articles
48 articles · Page 1 of 4
How to Test Agents That Call the Right Tool and Still Get It Wrong
Tool-use benchmarks check whether an agent picks the right tool. They miss agents that call correctly, then mishandle the result. Here's how to test for it.
How to Test Tool Argument Correctness in AI Agents
The most common production agent bug isn't picking the wrong tool. It's picking the right tool and passing the wrong arguments. Here's how to catch it.
LLM judges grade your agents wrong. Here's how to fix them.
LLM judges are the only practical way to evaluate agents at production scale, but they have predictable biases that give you false confidence. Here's how to calibrate them so the scores you see reflect the quality you actually have.
Testing AI agent handoffs before they break in production
Multi-agent CX systems fail at the seam between agents. Here's how to build handoff tests that catch context loss, duplicate actions, and loop failures before your customers feel them.
The trace-to-dataset loop: turning live conversations into eval cases
Your best eval cases are already in your production traces. Here's how to automatically curate the interesting ones into a test suite that gets better every week without manual effort.
Why most AI agents in production are flying blind
57% of organizations have AI agents in production. Only a third are satisfied with their observability. Here's what's wrong with single-call LLM logging and how to build a monitoring stack that actually works.
Build evals before you build the agent
Eval-Driven Development (EDD) means your grader ships before your prompt. Here's the complete methodology for building CX agents with evals as the first deliverable, not an afterthought.
Fault Injection Is the Missing Layer in Agent Testing
ReliabilityBench found rate limiting is the most damaging fault for AI agents in production -- more damaging than wrong answers. Here's how to inject chaos into your test suite before your users do.
Tool receipts: verifiable proof of what your agent actually did
When your AI agent claims it called a tool and got a result, can you verify that? Tool receipts are structured execution proofs that catch fabricated tool outputs before they damage customer trust.
Eval sampling for production agents: the 100/10/1 playbook
Running LLM-as-judge on every production conversation costs more than the agent itself. Here's the three-tier sampling system -- 100% lightweight heuristics, 10% LLM judge, 1% human review -- that keeps quality high without multiplying your eval bill.
Tracing AI agent failures across multi-step tool chains
When a production CX agent returns the wrong answer, the bug rarely lives in the last LLM call. Here's how to trace failures back to their root cause across multi-step tool chains.
Your CX agent hits 95% containment. Why is CSAT tanking?
Containment rate tells you how many conversations your agent handled without escalation. It says nothing about whether they went well. Here's the quality-weighted scorecard that tells the real story.
El briefing de Signal
Un email por semana. Cómo los equipos líderes de CS, ingresos e IA están convirtiendo conversaciones en decisiones. Benchmarks, playbooks y lo que funciona en producción.
