Context Growth in Long-Running Agents

Scenario

Example app: an operations agent investigates incidents. Its tools return log excerpts, metrics and deployment histories. A typical investigation takes 20 to 30 steps.

What you see in production

Investigations that run past about 30 steps fail with a context length error. The ones that finish cost far more than expected: the token bill for a 30-step investigation is roughly ten times that of a 10-step one, not three times.

The naive code

def tool_result(block, output):
    return {"type": "tool_result", "tool_use_id": block.id, "content": output}   # full output, every time

# each step: messages grows by the tool output, and the whole list is sent again

Reproduce it

Count the input tokens sent across a 30-step investigation where each tool returns about 6,000 tokens of logs:

SYSTEM_AND_TASK = 2_000
TOOL_OUTPUT = 6_000          # a big log excerpt per step
MODEL_TURN = 300
WINDOW = 200_000

def simulate(steps, keep_full_outputs, trimmed_size=800, keep_last=3):
    history, total_input = [], 0
    for step in range(1, steps + 1):
        size = SYSTEM_AND_TASK + sum(history)
        if size > WINDOW:
            return f"step {step}: context overflow ({size:,} tokens)"
        total_input += size                             # the whole history is sent on every call
        history.append(MODEL_TURN)
        history.append(TOOL_OUTPUT if keep_full_outputs else trimmed_size)
        if not keep_full_outputs and len(history) > keep_last * 2:
            # older tool results are replaced by a one-line note pointing at the saved copy
            history = [h if h == MODEL_TURN or i >= len(history) - keep_last * 2 else 60
                       for i, h in enumerate(history)]
    return f"{steps} steps, {total_input:,} input tokens in total"

for steps in (10, 30, 40):
    print(f"full  outputs, {steps}:", simulate(steps, True))
    print(f"trimmed outputs, {steps}:", simulate(steps, False))

full  outputs, 10: 10 steps, 303,500 input tokens in total
trimmed outputs, 10: 10 steps, 53,960 input tokens in total
full  outputs, 30: 30 steps, 2,800,500 input tokens in total
trimmed outputs, 30: 30 steps, 278,760 input tokens in total
full  outputs, 40: step 33: context overflow (203,600 tokens)
trimmed outputs, 40: 40 steps, 445,160 input tokens in total

Root cause

Model APIs are stateless: every call sends the whole conversation again. If each step adds a large tool output, the input of step n is proportional to n, so the total over a task grows with n². Eventually one call no longer fits in the context window at all. Most of those tokens are stale logs the model no longer needs.

The fix

Treat context as a budget and manage it actively:

• Trim at the source. Tools return a summary and the relevant lines (for example errors and the lines around them), with a handle such as log_ref: "inc-77/step-4" to fetch more if needed.
• Clear old tool results. Keep the last few results verbatim and replace older ones with a one-line note.
• Summarise milestones. When a phase of the investigation ends, ask the model for a short summary of findings and continue from it.
• Keep notes outside the context. A scratchpad file or memory tool holds findings the agent can re-read.
• Cache the stable prefix. System prompt and tool definitions are identical on every call; prompt caching makes them much cheaper.

Trade-offs

• Trimming can drop the one line that mattered; keep a way to fetch the full output by reference.
• Summaries lose detail and can introduce errors; keep raw data available and summarise conservatively.
• Context editing changes the prompt, which reduces prompt-cache hits for the edited part; keep the cached prefix stable.

Detect it in production

• Input tokens per call and per task, plotted against step number.
• Count of context-length errors and of tasks hitting the step limit.
• Size distribution of tool outputs per tool; outliers need trimming.

Key points

• Every call resends the history, so large tool outputs make cost grow quadratically.
• Return concise, referenced tool outputs and clear old ones.
• Summarise, keep notes outside the context, and cache the stable prefix.

Further reading: Context windows · Prompt caching · Effective context engineering for AI agents