Route easy requests to a smaller model

Approach

Use a small, fast model (or rules) to classify each request's difficulty, then call the matching model. Log the route with every request and evaluate the small model's quality per route before trusting it.

Solution

import anthropic

client = anthropic.Anthropic()
SMALL, LARGE = "claude-haiku-5-5", "claude-sonnet-5-5"
ROUTER_SYSTEM = ("Classify the user's request. SIMPLE: a factual lookup or short rewrite. "
                 "COMPLEX: reasoning, multi-step planning, code, or anything ambiguous. Reply SIMPLE or COMPLEX.")


def route(user_text: str) -> str:
    if len(user_text) > 1500:
        return LARGE                                     # cheap rule first
    r = client.messages.create(model=SMALL, max_tokens=5, system=ROUTER_SYSTEM,
                               messages=[{"role": "user", "content": user_text}])
    return SMALL if r.content[0].text.strip().upper().startswith("SIMPLE") else LARGE


def answer(user_text: str) -> tuple[str, str]:
    model = route(user_text)
    r = client.messages.create(model=model, max_tokens=1000, messages=[{"role": "user", "content": user_text}])
    return model, r.content[0].text

Pitfalls

• Routing without evals: quality drops silently on the small model.
• A router call that costs more latency than it saves for short requests; use rules first.
• Not logging the chosen model, so quality issues can't be traced to the route.

Variations

• Escalate: if the small model's answer fails validation, retry once with the large model.
• Route by feature instead of per request (for example, all classification on the small model).

Further reading: Models overview