How do parallel tool calls change an agent's latency?

Independent tools run concurrently, so the tool phase takes as long as the slowest call instead of the sum of all calls.

What must be true for prompt caching to help an agent?

The cached prefix (system prompt, tool definitions, stable context) must be identical on every call and come before the parts that change, such as the conversation and dynamic data.

When is it safe to route a step to a smaller model?

When evals show the smaller model meets the quality bar for that specific step, typically classification, extraction or routing, while reasoning-heavy steps stay on the larger model.

Cost and Latency Budgets for Agents — AI Agents and MCP in Production

Scenario

Example company: a travel agent assistant checks flights, hotels and visa rules for each request, using the largest available model for every step.

What you see in production

Median response time is 11 seconds and cost per request is several times the budget. Traces show three independent lookups running one after another, a large model used to decide trivial things ("is this a travel question?"), and the same 6,000-token system prompt and tool definitions billed in full on every call.

The naive code

for tool in ["search_flights", "search_hotels", "visa_rules"]:      # one at a time
    results.append(call_tool(tool, request))
answer = big_model(system=LONG_SYSTEM_PROMPT, tools=ALL_TOOLS, messages=history + results)

Reproduce it

Latency and cost for sequential versus parallel tool calls, with and without routing and caching:

Where the seconds and the money goTry it yourself

tools_ms = {"search_flights": 2400, "search_hotels": 1900, "visa_rules": 700}
big = {"latency_ms": 3000, "price_in": 15.0, "price_out": 75.0}     # $ per million tokens (illustrative)
small = {"latency_ms": 600, "price_in": 1.0, "price_out": 5.0}
PREFIX, VARIABLE, OUT = 6_000, 2_000, 400                             # tokens per call
CACHE_READ = 0.1                                                       # cached prefix billed at 10 %

def cost(model, cached=False):
    prefix = PREFIX * (CACHE_READ if cached else 1)
    return ((prefix + VARIABLE) * model["price_in"] + OUT * model["price_out"]) / 1e6

def scenario(parallel, routed, cached):
    route = small if routed else big
    latency = route["latency_ms"] + (max if parallel else sum)(tools_ms.values()) + big["latency_ms"]
    dollars = cost(route, cached) + cost(big, cached)
    return latency / 1000, dollars

for label, args in [("naive", (False, False, False)), ("parallel tools", (True, False, False)),
                    ("+ small router", (True, True, False)), ("+ prompt caching", (True, True, True))]:
    seconds, dollars = scenario(*args)
    print(f"{label:17} {seconds:4.1f} s   ${dollars:.4f} per request")

tools_ms = {"search_flights": 2400, "search_hotels": 1900, "visa_rules": 700}
big = {"latency_ms": 3000, "price_in": 15.0, "price_out": 75.0}     # $ per million tokens (illustrative)
small = {"latency_ms": 600, "price_in": 1.0, "price_out": 5.0}
PREFIX, VARIABLE, OUT = 6_000, 2_000, 400                             # tokens per call
CACHE_READ = 0.1                                                       # cached prefix billed at 10 %

def cost(model, cached=False):
    prefix = PREFIX * (CACHE_READ if cached else 1)
    return ((prefix + VARIABLE) * model["price_in"] + OUT * model["price_out"]) / 1e6

def scenario(parallel, routed, cached):
    route = small if routed else big
    latency = route["latency_ms"] + (max if parallel else sum)(tools_ms.values()) + big["latency_ms"]
    dollars = cost(route, cached) + cost(big, cached)
    return latency / 1000, dollars

for label, args in [("naive", (False, False, False)), ("parallel tools", (True, False, False)),
                    ("+ small router", (True, True, False)), ("+ prompt caching", (True, True, True))]:
    seconds, dollars = scenario(*args)
    print(f"{label:17} {seconds:4.1f} s   ${dollars:.4f} per request")

Output

naive             11.0 s   $0.3000 per request
parallel tools     8.4 s   $0.3000 per request
+ small router     6.0 s   $0.1600 per request
+ prompt caching   6.0 s   $0.0736 per request

Root cause

Agent latency is the sum of everything on the critical path, and cost is tokens × price for every call. Running independent tools sequentially puts their latencies end to end; using the biggest model for every step pays top price for easy decisions; and resending an unchanged prefix without caching pays full price for the same tokens again and again.

The fix

Parallel tool calls: let the model request several tools in one turn and run them concurrently.
Route by difficulty: a small, fast model for classification, extraction and routing; the large model only where reasoning quality matters.
Prompt caching: keep the system prompt and tool definitions stable and first, so they are cached.
Trim context as covered earlier, and cap output length per step.
Budgets per task type: maximum steps, tokens and wall-clock time, with graceful degradation (a partial answer plus follow-up) when exceeded.
Stream the final answer so users see progress early.

Trade-offs

Small models make more mistakes; measure quality per step with evals before routing traffic to them.
Caching works only if the prefix is byte-identical; dynamic content (dates, user names) belongs after the cached part.
Parallel calls increase peak load on downstream services; respect their rate limits.

Detect it in production

Latency broken down by model time versus tool time, from traces.
Cost per task type and per user, with alarms on budget overruns.
Cache hit rate (cached input tokens / total input tokens).

Key points

Run independent tools in parallel; latency is the critical path, not the sum.
Route easy steps to small models and cache the stable prefix.
Set per-task budgets for steps, tokens and time, and degrade gracefully.

Further reading: Parallel tool use · Prompt caching · Choosing a model