Parallel tool calls: the 5x speedup in your agent

A customer asks your support agent: "Can you check my order status, confirm my warranty is still active, and tell me if there's a discount I can apply?" Three questions, three independent tool calls. If your agent runs them sequentially (order lookup, then warranty check, then discount query) at 200ms each, the user waits 600ms before the agent can answer. If the agent calls all three at once, it waits 200ms. Same work, one third the time.

Most LLMs can issue multiple tool calls in a single response turn. Most agent runtimes don't take advantage of it because the sequential pattern is the natural one to write. This article explains how parallel tool calling works, when to use it, how to handle errors when one call in a batch fails, and how to verify in production that your agent is actually running calls concurrently.

What parallel tool calls are

When an LLM responds with tool calls in a single turn, it can return a list of them rather than just one. Your runtime is responsible for executing those calls -- sequentially or in parallel is your decision, not the model's. The model says "I need these results." How you fetch them is an implementation choice in your code.

Parallel tool calls execute all the requested calls at the same time using concurrent I/O. All results are collected and returned to the model in a single follow-up message. The model then uses all the results to formulate its response.

This is different from speculative tool calling (where you predict and pre-fetch results before the model asks, covered in speculative tool calling for agent latency). Parallel execution applies when the model has already requested multiple tools in one turn. You're just choosing not to make the user wait for each one serially.

The model doesn't know or care which approach you used. It receives the same set of tool results either way. The only thing that changes is how long the user waits.

When to parallelize versus sequence

You can only parallelize tool calls when there's no data dependency between them -- when the result of call A is not the input to call B.

Safe to parallelize:

Fetching data from independent sources (CRM lookup, inventory check, order history in the same turn). Calling different APIs that don't share state. Running the same tool multiple times with different arguments (look up order 1001 and order 1002 at the same time). Read operations that don't modify shared state.

Must sequence:

Call B needs data from call A's result. Write operations where order matters (create a record, then update it). Tool calls where the second depends on the outcome of the first (check if an item is in stock, then reserve it if it is).

Most LLMs sequence dependent calls naturally. If the model needs to look up a customer ID first and then fetch orders using that ID, it'll split those across two separate turns rather than requesting them simultaneously. But if you're ever unsure whether two calls in a batch are independent, treat them as sequential.

// Safe to parallelize: all lookups are independent
const independentCalls = [
  { name: "get_order_status", args: { orderId: "1001" } },
  { name: "get_customer_tier", args: { customerId: "cust_123" } },
  { name: "get_available_discounts", args: { productSku: "SKU-456" } },
];
 
// Must sequence: second call needs the first call's output
// The model would NOT request these in the same turn
const dependentCalls = [
  { name: "lookup_customer_by_email", args: { email: "user@example.com" } },
  // next turn: { name: "get_order_history", args: { customerId: "<result from above>" } }
];

Implementing parallel execution in TypeScript

Parallel execution comes down to one swap: replace your sequential tool loop with Promise.all. Here's the complete pattern.

import Anthropic from "@anthropic-ai/sdk";
 
type ToolHandler = (args: Record<string, unknown>) => Promise<unknown>;
 
async function runAgentTurn(
  client: Anthropic,
  tools: { name: string; handler: ToolHandler; schema: Anthropic.Tool }[],
  messages: Anthropic.MessageParam[]
): Promise<string> {
  const handlerMap = new Map(tools.map((t) => [t.name, t.handler]));
 
  const response = await client.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 2048,
    tools: tools.map((t) => t.schema),
    messages,
  });
 
  if (response.stop_reason !== "tool_use") {
    const textBlock = response.content.find((b) => b.type === "text");
    return textBlock?.type === "text" ? textBlock.text : "";
  }
 
  // Collect every tool_use block from this response
  const toolCalls = response.content.filter(
    (b): b is Anthropic.ToolUseBlock => b.type === "tool_use"
  );
 
  // Execute ALL of them in parallel -- this is the key line
  const results = await Promise.all(
    toolCalls.map(async (call) => {
      const handler = handlerMap.get(call.name);
      if (!handler) throw new Error(`Unknown tool: ${call.name}`);
 
      const output = await handler(call.input as Record<string, unknown>);
      return {
        type: "tool_result" as const,
        tool_use_id: call.id,
        content: JSON.stringify(output),
      };
    })
  );
 
  // Return all results in a single follow-up message
  const updatedMessages: Anthropic.MessageParam[] = [
    ...messages,
    { role: "assistant", content: response.content },
    { role: "user", content: results },
  ];
 
  return runAgentTurn(client, tools, updatedMessages);
}

The critical difference is await Promise.all(toolCalls.map(...)). Without it -- if you used a for loop or called each tool with await inside a sequential loop -- you'd execute them one at a time. Promise.all fires all the async operations simultaneously and waits for the last one to finish.

When the model returns three tool calls, you're making three concurrent network requests and waiting only as long as the slowest one. When it returns five, you're waiting only as long as the slowest of five. The user never waits for the sum.

Handling errors in a parallel batch

Parallel execution introduces a failure mode that sequential execution doesn't have: partial failure. If five tools run simultaneously and one throws, what do you do?

Promise.all rejects immediately when any promise rejects. This is "fail-fast" behavior -- one bad tool call cancels the whole batch. For many agents, this is correct: if you can't get all the data you need, better to fail cleanly than to continue with incomplete information.

But some CX use cases call for partial success: continue with what you have, even if one call failed, and let the model reason about the gaps.

async function executeWithPartialSuccess(
  toolCalls: Anthropic.ToolUseBlock[],
  handlerMap: Map<string, ToolHandler>
): Promise<Anthropic.ToolResultBlockParam[]> {
  // Promise.allSettled collects both successes and failures
  const settled = await Promise.allSettled(
    toolCalls.map(async (call) => {
      const handler = handlerMap.get(call.name);
      if (!handler) throw new Error(`Unknown tool: ${call.name}`);
      const output = await handler(call.input as Record<string, unknown>);
      return { id: call.id, output };
    })
  );
 
  return settled.map((result, i) => ({
    type: "tool_result" as const,
    tool_use_id: toolCalls[i].id,
    content:
      result.status === "fulfilled"
        ? JSON.stringify(result.value.output)
        : JSON.stringify({
            error: result.reason?.message ?? "Tool call failed",
          }),
    is_error: result.status === "rejected",
  }));
}

Promise.allSettled never rejects, even if some promises do. You get a result for every call, tagged as fulfilled or rejected. Return all of them to the model, including the error messages, and a well-prompted agent will handle partial data gracefully ("I found your order status and your warranty information, but couldn't retrieve available discounts right now -- you may want to check the promotions page directly").

When to use which:

Use Promise.all (fail-fast) for write operations, for cases where all data is required to form a valid response, and for agents where partial state is dangerous.

Use Promise.allSettled (partial success) for read-only lookups where incomplete data is still useful, and for agents that have fallback language for "I couldn't get X."

Testing that parallelism actually happens

Here's a common trap: you add Promise.all and assume your tools now run in parallel. But if your tool handlers internally await sequentially or share a blocking resource (a database connection pool, a rate-limited API client with a mutex), Promise.all starts all the handlers at once but they end up waiting for the same bottleneck. You're still sequential in practice.

A simple test: add a 200ms artificial delay to each tool and measure total execution time.

import { describe, it, expect } from "vitest";
 
const slowTool: ToolHandler = async () => {
  await new Promise((r) => setTimeout(r, 200));
  return { data: "result" };
};
 
describe("parallel tool execution", () => {
  it("should run 3 tools in ~200ms, not ~600ms", async () => {
    const mockCalls: Anthropic.ToolUseBlock[] = [
      { type: "tool_use", id: "1", name: "slow_tool", input: {} },
      { type: "tool_use", id: "2", name: "slow_tool", input: {} },
      { type: "tool_use", id: "3", name: "slow_tool", input: {} },
    ];
 
    const handlerMap = new Map([["slow_tool", slowTool]]);
 
    const start = Date.now();
    await executeWithPartialSuccess(mockCalls, handlerMap);
    const elapsed = Date.now() - start;
 
    // Parallel: ~200ms. Sequential: ~600ms.
    expect(elapsed).toBeLessThan(400);
  });
});

If this test takes 600ms instead of 200ms, your execution path is sequential somewhere. Common causes: a shared async/await bottleneck in the handler, a Prisma or Sequelize connection pool that serializes queries, or a third-party SDK client that queues concurrent calls.

Add this test to your suite now and keep it. It's the canary for "someone added a mutex and broke our parallelism."

Monitoring tool call performance in production

Two metrics tell you whether parallel execution is working and how much it's helping.

Parallelism rate: the percentage of agent turns where the model issued more than one tool call. This tells you how often your agent has the opportunity to benefit from parallel execution. Low rates (below 20%) mean your agent mostly calls one tool per turn and the optimization doesn't apply much. High rates mean parallel execution is doing meaningful work.

Batch latency ratio: for turns with multiple tool calls, compare the total time to get all results against the sum of individual call latencies. A ratio near 1.0 means you're running sequentially (total equals sum). A ratio near the maximum individual latency divided by the sum means you're running in parallel.

import { Chanl } from "@chanl/sdk";
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
const metrics = await chanl.calls.getMetrics({
  timeRange: "24h",
  groupBy: "turn",
  fields: [
    "tool_call_count",
    "tool_batch_latency_ms",
    "tool_sum_latency_ms",
    "parallelism_ratio",
  ],
});
 
const multiToolTurns = metrics.turns.filter((t) => t.tool_call_count > 1);
 
if (multiToolTurns.length > 0) {
  const avgRatio =
    multiToolTurns.reduce((sum, t) => sum + t.parallelism_ratio, 0) /
    multiToolTurns.length;
 
  console.log(`Turns with 2+ tools: ${multiToolTurns.length}`);
  console.log(`Average parallelism ratio: ${avgRatio.toFixed(2)}`);
  // 0.3 or below = good parallelism (capped near the slowest call)
  // Near 1.0 = likely sequential
}

Chanl's Analytics dashboard also shows tool call timing as a per-turn timeline in the conversation trace view. You can see each tool call's start time and duration on a visual timeline, which makes it immediately obvious when tools ran concurrently (overlapping bars) versus back-to-back (bars lined up end to end).

MCP and concurrent tool calls

If your agent calls tools through MCP servers, the same parallel execution model applies. MCP doesn't enforce ordering on concurrent tools/call requests. Your client can fire multiple requests simultaneously and await their responses.

Watch for one edge case: stateful MCP servers that maintain session-level shared state. Two simultaneous calls that read and write the same server state can produce race conditions. This is rare for well-designed MCP servers (the MCP spec recommends stateless tool implementations), but worth testing if you're calling tools on a server you didn't build.

The 2026 MCP stateless core spec makes this cleaner by design: servers that follow the new spec don't maintain protocol-level session state, which means concurrent requests from the same client are safe without coordination.

For a broader look at how MCP fits into agent tool architecture, MCP deep dive: advanced patterns has a section on concurrent call patterns and how to handle servers that mix stateless and stateful tools.

You can audit the MCP servers your agent is connected to -- and see which ones support concurrent calls -- through Chanl's MCP dashboard. If a server is flagged as stateful, it'll show up there so you know to treat its tools as non-parallelizable.

The compounding benefit at scale

Back to the opening scenario: one customer, three independent questions, three tool calls. Parallel execution saves 400ms on that one turn. Now multiply: an agent handling 1,000 conversations a day, with an average of 3 tool calls per turn and 4 turns per conversation, runs 12,000 tool calls daily. If 40% of those turns have 2 or more independent calls, and you save an average of 200ms per parallel turn, that's 960 seconds of cumulative user wait time eliminated every single day. Per agent.

That number grows with the complexity of your agent's tasks. Agents that do more external lookups (CRM, inventory, knowledge base, scheduling) have more parallelizable work. Every new tool you add to your agent is another candidate for concurrent execution.

The implementation is a one-time change to your tool execution loop. Once Promise.all is in place, every future tool the agent gains benefits automatically. It's one of the few optimizations with near-zero ongoing maintenance cost.

The discipline is in verification. Once you make the change, run the timing test. Track the parallelism ratio in production. And when someone adds a new tool handler with a synchronous bottleneck inside, the test will catch it before it quietly turns your 200ms parallel execution back into 600ms sequential.

El briefing de Signal

Un email por semana. Cómo los equipos líderes de CS, ingresos e IA están convirtiendo conversaciones en decisiones. Benchmarks, playbooks y lo que funciona en producción.

Subscribe

500+ líderes de CS e ingresos suscritos