Testing & Evaluation Articles
35 articles · Page 2 of 3
Every Failed Call Is a Test Case You Haven't Written Yet
The gap between staging and production for AI agents is measured in surprise. Here's how to close the loop from live failure to regression gate.
How Much Testing Is Enough for Your AI Agent?
Code coverage doesn't apply to AI agents. Here's a framework for thinking about evaluation coverage: how many scenarios you need, what distribution to target, and how to know when you've tested enough.
Your LLM-as-judge may be highly biased
LLM-as-Judge has 12 documented biases. Here are 6 evaluation methods production teams actually use instead, with code examples and patterns.
Is monitoring your AI agent actually enough?
Research shows 83% of agent teams track capability metrics but only 30% evaluate real outcomes. Here's how to close the gap with multi-turn scenario testing.
Memory bugs don't crash. They just give wrong answers.
Memory bugs don't crash your agent. They just give subtly wrong answers using stale context. Here are 5 test patterns to catch them before customers do.
Online vs. Offline Evals: Close the Production Gap
89% of teams have observability but only 37% run online evals. Here's why that gap is where production failures hide, and how to close it with a practical online eval pipeline.
We open-sourced our AI agent testing engine
chanl-eval is an open-source engine for stress-testing AI agents with simulated conversations, adaptive personas, and per-criteria scorecards. MIT licensed.
Agent Drift: Why Your AI Gets Worse the Longer It Runs
AI agents silently degrade over long conversations. Research quantifies three types of drift and shows why point-in-time evals miss them entirely.
NIST Red-Teamed 13 Frontier Models. All of Them Failed.
NIST ran 250K+ attacks against every frontier model. None survived. Here's what the results mean for teams shipping AI agents to production today.
Your Agent Aced the Benchmark. Production Disagreed.
We scored 92% on GAIA. Production CSAT: 64%. Here's which AI agent benchmarks actually predict deployed performance, why most don't, and what to measure instead.
Zero-Shot or Zero Chance? How AI Agents Handle Calls They've Never Seen Before
When a customer calls with a request your AI agent has never encountered, what actually happens? We break down the mechanics of zero-shot handling, and how to test for it before it fails in production.
Your Agent Passed Every Dev Test. Here's Why It'll Fail in Production
A 4-layer testing framework for AI agents (unit, integration, performance, and chaos testing) so your agent survives real customers, not just controlled demos.
The Signal Briefing
Un email por semana. Cómo los equipos líderes de CS, ingresos e IA están convirtiendo conversaciones en decisiones. Benchmarks, playbooks y lo que funciona en producción.
