Articles tagged “evaluation”
24 articles
How to Test Agents That Call the Right Tool and Still Get It Wrong
Tool-use benchmarks check whether an agent picks the right tool. They miss agents that call correctly, then mishandle the result. Here's how to test for it.
How to Test Tool Argument Correctness in AI Agents
The most common production agent bug isn't picking the wrong tool. It's picking the right tool and passing the wrong arguments. Here's how to catch it.
The trace-to-dataset loop: turning live conversations into eval cases
Your best eval cases are already in your production traces. Here's how to automatically curate the interesting ones into a test suite that gets better every week without manual effort.
Why most AI agents in production are flying blind
57% of organizations have AI agents in production. Only a third are satisfied with their observability. Here's what's wrong with single-call LLM logging and how to build a monitoring stack that actually works.
Build evals before you build the agent
Eval-Driven Development (EDD) means your grader ships before your prompt. Here's the complete methodology for building CX agents with evals as the first deliverable, not an afterthought.
Fault Injection Is the Missing Layer in Agent Testing
ReliabilityBench found rate limiting is the most damaging fault for AI agents in production -- more damaging than wrong answers. Here's how to inject chaos into your test suite before your users do.
Eval sampling for production agents: the 100/10/1 playbook
Running LLM-as-judge on every production conversation costs more than the agent itself. Here's the three-tier sampling system -- 100% lightweight heuristics, 10% LLM judge, 1% human review -- that keeps quality high without multiplying your eval bill.
A quality control plane for your production agent fleet
Quality drift in production AI agents is silent. Build a quality control plane that scores every conversation, enforces SLOs, gates deployments, and alerts on drift.
Why your eval scores don't predict real conversation quality
Your agent passes 91% of your eval suite but customers keep escalating. The problem is almost always the same: you're measuring single-turn quality, and customers experience multi-turn conversations. Here's how to build evaluation that reflects what actually happens.
How to Build a Trajectory Eval for Your AI Agent
Outcome evals check the final answer. Trajectory evals check the path: tools called, data touched, steps taken. Here's how to build one for a CX agent.
The Agent Development Lifecycle: Ship, Observe, Improve
Shipping an AI agent is easy. Keeping it reliable after launch is where most teams struggle. The ADLC gives you a structured approach: Intent, Build, Evaluate, Deploy, Observe -- and then do it again.
Trajectory Eval: Catch Agent Bugs Output Scoring Misses
Final-output scoring misses 20-40% of agent regressions. Trajectory evaluation scores every step an agent takes -- tool calls, reasoning decisions, order of operations -- and catches the bugs that output-only evals can't see.
Your Agent Has Observability. It Doesn't Have Measurement.
89% of AI teams added observability. 52% added evals. But only 31% can say whether their agent is getting better or worse. Here's the difference between watching your agent and actually measuring it.
AI Agent KPIs: What to Measure Before You Ship
Only 31% of teams have a measurement framework for their AI agents. Here's how to define task completion rate, escalation rate, cost per outcome, and CSAT delta before your first production interaction.
How to Eval Agents When There's No Right Answer
Most eval methods assume you know the correct response. CX agents rarely have one. Here's how to score agent quality with criteria-based rubrics and LLM-as-judge, no labeled ground truth required.
Stop Using SWE-Bench to Pick Your CX Model
SWE-Bench scores 85% or 23% depending on the harness, and neither measures customer experience. Why tau-bench, tau2-bench, and pass^k matter for CX agents.
Every Conversation Is an Experiment You Didn't Run
Your agent already ran the A/B test you're scoping. Here's how to read the results in your logs with propensity matching, synthetic control, and diff-in-diff.
Is AI Better Than Your Humans? Score Both on One Rubric
Most teams can't say whether AI beats humans because they score them differently. One rubric, run on both, sliced by segment, gives you an honest answer.
How Much Testing Is Enough for Your AI Agent?
Code coverage doesn't apply to AI agents. Here's a framework for thinking about evaluation coverage: how many scenarios you need, what distribution to target, and how to know when you've tested enough.
Is monitoring your AI agent actually enough?
Research shows 83% of agent teams track capability metrics but only 30% evaluate real outcomes. Here's how to close the gap with multi-turn scenario testing.
We open-sourced our AI agent testing engine
chanl-eval is an open-source engine for stress-testing AI agents with simulated conversations, adaptive personas, and per-criteria scorecards. MIT licensed.
Your Agent Completed the Task. It Also Forgot 87% of What It Knew.
Task completion hides a silent failure: agents forget 87% of stored knowledge under complexity. New research reveals why standard evals miss this entirely.
74% of Production Agents Still Rely on Human Evaluation
A survey of 306 practitioners reveals most production agents are far simpler than expected. The eval gap isn't a tooling problem. It's a trust problem.
Voice AI Tests Pass in the Lab. They Fail on the Call.
Why happy-path test suites pass voice agents through QA that fall apart on the first real call, and the five testing habits that actually catch the failures.
El briefing de Signal
Un email por semana. Cómo los equipos líderes de CS, ingresos e IA están convirtiendo conversaciones en decisiones. Benchmarks, playbooks y lo que funciona en producción.
