Fit conversation history into a token budget

Approach

Count tokens with client.messages.count_tokens before sending. If the request exceeds your budget, drop the oldest turns (keeping the first user message for context) or replace them with a short summary.

Solution

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-5-5"
BUDGET = 30_000                                   # input tokens you are willing to send per call


def count(system: str, messages: list[dict]) -> int:
    return client.messages.count_tokens(model=MODEL, system=system, messages=messages).input_tokens


def fit_history(system: str, messages: list[dict]) -> list[dict]:
    trimmed = list(messages)
    while count(system, trimmed) > BUDGET and len(trimmed) > 3:
        # drop the oldest user/assistant pair after the first message, keeping turns alternating
        del trimmed[1:3]
    return trimmed


def reply(system: str, history: list[dict]) -> str:
    messages = fit_history(system, history)
    response = client.messages.create(model=MODEL, max_tokens=1000, system=system, messages=messages)
    return response.content[0].text

Pitfalls

• Breaking the user/assistant alternation (or orphaning a tool result) when deleting turns.
• Counting with a characters-divided-by-four estimate for hard limits; use the counting endpoint.
• Trimming the system prompt instead of old turns.

Variations

• Summarise dropped turns with a small model and insert the summary as context.
• Store the full history in your database and send only a window.

Further reading: Token counting