How to Test Tool Argument Correctness in AI Agents

A support agent did everything right last month, according to the transcript. A customer asked for a refund on a duplicate charge. The agent confirmed the order, apologized for the double billing, and called issue_refund. The reply back to the customer was warm and correct. The scorecard for tool selection was green: it called the refund tool, which was exactly the right tool.

The refund went to the wrong order. Same customer, different purchase, from three weeks earlier. The agent had two orders in context and passed the order ID of the older one. Nobody caught it until the customer emailed back confused about why their headphones purchase had been refunded instead of the duplicate charge they actually complained about.

Nothing about that call looked broken. The tool was right. The language was right. The JSON was valid. The only thing wrong was a single value inside the arguments, and that single value is where the largest category of production agent bugs lives.

What Argument Correctness Actually Measures

Argument correctness measures whether your agent passed the right inputs into a tool, given that it already picked the right tool. Tool correctness asks "did the agent call the right function?" Argument correctness asks "did it call that function correctly?" These are different questions, and the second one is where real customer impact hides.

Most teams evaluate tool selection and stop there. You look at a trace, confirm the agent called issue_refund instead of escalate_to_human, and mark it correct. That check passes on the refund story above, because the tool selection genuinely was correct. The failure was one level deeper, in the orderId argument.

Here is the split that matters:

Selection and completion are the two checks most eval setups include. Argument correctness is the layer in between, and it's the one that turns a plausible-looking trace into a real-world mistake. The 2026 agent evaluation guides from Confident AI and Future AGI break argument correctness out as its own metric, separate from tool selection, and production failure-mode writeups name tool misuse, which includes exactly this right-tool-wrong-arguments case, as the most common agent-specific failure in production. It's common precisely because it survives the checks that only look at the endpoints.

Why Wrong Arguments Survive Every Casual Review

Wrong arguments survive review because everything around them looks correct. The agent narrates its reasoning fluently, the tool call is valid JSON that matches the schema, and the customer-facing reply is coherent. A reviewer skimming the transcript sees a competent interaction. The error is buried inside a structurally perfect call, and you only notice it if you compare each argument against what it should have been.

Think about the ways a single argument goes wrong even when the tool is right:

• Wrong value from context. Two orders in the conversation, and the agent grabs the wrong ID. Two people named in an email thread, and it addresses the wrong one.
• Format errors. A date the agent parses from "next Tuesday" into the wrong Tuesday, or a phone number without the country code, or an amount in dollars where the API expects cents.
• Unit confusion. 29.99 passed where the tool expects 2999 cents. The refund is off by a factor of 100 and the JSON is still valid. Stripe's cents-based amounts have caused this exact bug in more codebases than anyone will admit.
• Hallucinated arguments. A required reason_code the agent invents rather than selecting from the allowed enum, or a customer_tier it guesses because it wasn't in context.
• Missing optional fields that matter. The tool runs without the idempotency_key, so a retry double-charges.

None of these are visible from tool selection. All of them pass a quick human read. Several of them pass schema validation too, which is the trap I want to spend the most time on.

Schema Validation Is Necessary and Nowhere Near Sufficient

Schema validation confirms an argument is well-formed. It does not confirm the argument is correct. A refund of $500.00 to a valid order ID that belongs to the wrong customer passes every schema check you can write, because each field is the right type, present, and within range. The call is structurally perfect and semantically wrong.

This is the distinction between structured outputs and argument correctness. Structured outputs, which I covered in reliable structured outputs for AI agents, guarantee the shape of a tool call: valid JSON, correct types, required fields present. That solves the malformed-call problem. It does nothing for the wrong-value problem, because a wrong value can be perfectly shaped.

Start with schema validation anyway, because it's cheap and it catches the easy tier of failures for free.

import { z } from 'zod';
 
// The refund tool's argument contract
const issueRefundSchema = z.object({
  orderId: z.string().regex(/^ord_[a-z0-9]{16}$/),
  amountCents: z.number().int().positive().max(1_000_00),
  currency: z.enum(['usd', 'eur', 'gbp']),
  reasonCode: z.enum(['duplicate', 'defective', 'not_received', 'goodwill']),
  idempotencyKey: z.string().uuid(),
});
 
export function validateRefundArgs(rawArgs: unknown) {
  const result = issueRefundSchema.safeParse(rawArgs);
  if (!result.success) {
    return { valid: false, errors: result.error.issues };
  }
  return { valid: true, args: result.data };
}

This catches a hallucinated reasonCode, a non-integer amountCents, an amount over the cap, a malformed orderId, and a missing idempotencyKey. Those are real bugs and worth catching at the boundary before the tool ever runs.

What it does not catch: orderId: "ord_a1b2c3d4e5f6g7h8" when the correct order was ord_z9y8x7w6v5u4t3s2. Both match the regex. Both are the right type. The schema is happy. The customer is not. Semantic argument errors, the ones that pass validation cleanly, are the exact class that reaches production, so the schema check is your floor, not your ceiling.

Three Layers That Together Give You Real Coverage

To actually measure argument correctness you need three checks stacked, because no single one covers the full range. Schema validation handles structural errors. Deterministic assertions handle known-answer cases. An LLM judge handles the open-ended middle where arguments are reasonable-or-not rather than exactly-right-or-wrong. Each layer catches what the one before it misses.

The first layer you already have. The second layer is where testing pays off, because it's deterministic and fast enough to run on every change.

Deterministic Assertions for Known Cases

For any test case where you know the correct answer, assert on the arguments directly rather than on the final reply. This is the highest-signal, lowest-flake check you can run. You set up a scenario, drive the agent to a tool call, and compare the arguments it produced against the arguments it should have produced.

import { runAgentTurn } from './harness';
 
test('refund targets the duplicate charge, not the older order', async () => {
  const trace = await runAgentTurn({
    context: {
      orders: [
        { id: 'ord_headphones071001', amountCents: 8999, date: '2026-07-10' },
        { id: 'ord_duplicate0731a01', amountCents: 4500, date: '2026-07-31' },
        { id: 'ord_duplicate0731a02', amountCents: 4500, date: '2026-07-31' },
      ],
    },
    userMessage: "I got charged twice for the same order on the 31st, please refund one.",
  });
 
  const call = trace.toolCalls.find((c) => c.name === 'issue_refund');
  expect(call).toBeDefined();
 
  // Argument-level assertions: the part most tests skip
  expect(call!.args.orderId).toBe('ord_duplicate0731a02');
  expect(call!.args.amountCents).toBe(4500);
  expect(call!.args.currency).toBe('usd');
  expect(call!.args.reasonCode).toBe('duplicate');
});

Notice the test asserts on call.args, not on trace.finalReply. A reply-only test passes as long as the agent says something sensible about issuing a refund. This test fails the moment the agent refunds the headphones instead of the duplicate charge, which is exactly the bug from the opening story. The QA layer that most CX teams skip, tool call testing, is largely about writing assertions at this level.

Build a suite of these, and deliberately include adversarial cases: two orders with the same amount, a customer with a similar name to another, a date phrased ambiguously, a currency that differs from the account default. Adversarial argument cases are cheap to write and they surface the failure modes that only appear under pressure.

An LLM Judge for the Open-Ended Middle

Some arguments don't have one exact right answer. A summary field, a priority the agent infers, a category chosen from a large taxonomy, a note written for the next agent. For these you can't write a hard-coded assertion, so you score them with a judge that reads the conversation and rates whether the argument is reasonable given what the customer actually said.

import { Chanl } from '@chanl/sdk';
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
// One-time setup: judge criteria live on a scorecard as prompt criteria
export async function addArgumentCriteria(scorecardId: string, categoryId: string) {
  await chanl.scorecard.createCriterion(scorecardId, {
    categoryId,
    name: 'Argument Grounding',
    type: 'prompt',
    settings: {
      description:
        'For each tool call, are the argument values grounded in what the ' +
        'customer actually said or in retrieved context? Fail any value ' +
        'that appears invented, guessed, or copied from the wrong entity.',
      evaluationType: 'boolean',
    },
    threshold: { expectedValue: true },
  });
 
  await chanl.scorecard.createCriterion(scorecardId, {
    categoryId,
    name: 'Unit and Format',
    type: 'prompt',
    settings: {
      description:
        'Check numeric and date arguments for unit or format errors: ' +
        'dollars vs cents, wrong timezone, ambiguous date resolved incorrectly.',
      evaluationType: 'boolean',
    },
    threshold: { expectedValue: true },
  });
}
 
// Score a finished conversation's tool arguments against that scorecard
export function judgeArguments(interactionId: string, scorecardId: string) {
  return chanl.scorecard.evaluate(interactionId, { scorecardId });
}

The judge is the flexible layer, and you keep it flexible on purpose. It won't catch a wrong-but-plausible order ID as reliably as a deterministic assertion, which is why you use assertions wherever a known answer exists. But it covers the arguments assertions can't reach, and it scales to production traffic where you don't have a labeled correct answer for every call. Component-level metrics like argument correctness work best combined with at least one end-to-end task-completion metric, so the judge feeds a per-call argument score that you track alongside whether the customer's goal was actually met. For more on running judges as a production pipeline rather than a one-off, see LLM as a judge for production eval.

Score

Good

0/100

Tone & Empathy

94%

Resolution

88%

Response Time

72%

Compliance

85%

Where Argument Errors Hurt Most, and Where They Don't

Argument errors on write operations with money or identity attached are the ones that actually damage customers. A refund to the wrong account, a cancellation of the wrong subscription, an email to the wrong recipient, an appointment at the wrong time. Read errors are recoverable, because the agent can re-fetch and correct itself mid-conversation. Write errors with wrong arguments create side effects in the real world that are expensive to reverse.

This gives you a natural prioritization for where to spend your testing budget:

• Highest priority: write tools with money or identity. issue_refund, cancel_subscription, update_payment_method, send_email, book_appointment, apply_credit. Every argument on these tools deserves a deterministic assertion suite and a judge check on the fuzzy fields. A wrong argument here is a real financial or trust event.
• Medium priority: write tools without money. add_note, set_tag, update_preference. Wrong arguments create clutter and confusion but rarely a crisis.
• Lower priority: read tools. get_order, search_faq, lookup_customer. A wrong argument produces a wrong lookup the agent usually notices and retries. Still worth checking, but not where the first hour of your testing effort goes.

The failures compound in multi-step chains, which is worth naming explicitly. If an agent reads the wrong order in step one, then passes that wrong order into a refund in step three, the argument error propagated silently through the chain. This is the compound error problem applied to arguments: a small wrong value early becomes a large wrong action later. Asserting on arguments at each step, not just the final action, is how you localize where the chain went wrong.

Putting It in the Pipeline Before It Hits a Customer

The place to catch argument errors is in scenario tests that run on every change, driven by a simulated user, asserting on the arguments the agent produces. This turns argument correctness from something you discover in a customer email into something a failing test tells you about before you ship. The pattern is the same one you'd use for any agent regression: define the scenario, run it, assert on what the agent did, not just what it said.

import { Chanl } from '@chanl/sdk';
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
// Each scenario drives a simulated customer toward a tool call. Its
// scorecard grades the arguments (order ID, amount, currency), not
// how friendly the reply sounded.
const ARGUMENT_SCENARIOS = [
  'scn_refund_duplicate_not_older_order',
  'scn_currency_matches_account',
];
 
const PASS_BAR = 8; // your passing bar, on the scorecard's scale
 
export async function runArgumentSuite(agentId: string) {
  const failures: string[] = [];
 
  for (const scenarioId of ARGUMENT_SCENARIOS) {
    const { data: started } = await chanl.scenarios.run(scenarioId, { agentId });
 
    // Poll until the run finishes in real CI; one fetch shown for brevity
    const { data: execution } = await chanl.scenarios.getExecution(
      started.executionId ?? started.execution.id,
    );
 
    const score = execution.overallScore ?? 0;
    if (execution.status !== 'completed' || score < PASS_BAR) {
      failures.push(scenarioId);
    }
  }
 
  if (failures.length > 0) {
    throw new Error(`Argument correctness regressions: ${failures.join(', ')}`);
  }
}

Wire this into CI the same way you'd wire unit tests. When someone changes the system prompt, swaps the model, or edits a tool description, the argument suite runs and tells you if the change moved argument accuracy. Tool descriptions matter more than most teams expect here, because a vague parameter description is a direct cause of wrong arguments, which is why the tool management layer treats descriptions and argument contracts as first-class. If you want the deeper version of that, function calling accuracy and the tool count decay curve covers how argument accuracy degrades as you add tools, and how function calling actually works covers the mechanics of the call itself.

Run the same scenarios continuously against a sample of production traffic, not just in CI, so that a model update from your provider or a slow drift in argument quality shows up on a monitoring dashboard instead of in a support ticket. Argument correctness isn't a one-time gate. It's a number you watch, the same way you watch latency or containment. The build side of this, defining the tools and their contracts clearly enough that the agent passes correct arguments, connects directly to the monitor side, where you confirm it still does under real traffic.

The refund that went to the wrong order didn't need a smarter model to prevent. It needed one assertion: refund the order the customer complained about, not a different one. That assertion is three lines of test code. The bug was invisible to every check that looked only at which tool got called, and visible immediately to the one check that looked at what got passed into it. That's the whole case for argument correctness. The tool was never the problem. The value inside it was.

El briefing de Signal

Un email por semana. Cómo los equipos líderes de CS, ingresos e IA están convirtiendo conversaciones en decisiones. Benchmarks, playbooks y lo que funciona en producción.

Subscribe

500+ líderes de CS e ingresos suscritos