Model Choice, max_tokens and Temperature

Model choice

Providers offer families of models: large models for hard reasoning and writing, small, fast models for classification, extraction and routing. Start with a capable model to prove the feature works, then test whether a smaller one meets your quality bar for each step. See Models overview.

max_tokens

The maximum number of output tokens. Too low cuts answers off (stop_reason: max_tokens); there is no charge for unused allowance, but very long outputs take longer to generate.

Temperature

Controls randomness when choosing the next token. Low (near 0) gives more consistent, focused output, good for extraction and classification. Higher gives more varied output, useful for brainstorming. Even at 0, outputs are not guaranteed identical across calls.

import math, random

def sample(scores, temperature, rng):
    if temperature == 0:
        return max(scores, key=scores.get)
    weights = {w: math.exp(s / temperature) for w, s in scores.items()}
    r, total = rng.random() * sum(weights.values()), 0
    for word, w in weights.items():
        total += w
        if r <= total:
            return word

next_word = {"blue": 2.0, "grey": 1.2, "orange": 0.4}     # model scores for "The sky is ..."
for t in (0, 0.5, 1.5):
    rng = random.Random(1)
    picks = [sample(next_word, t, rng) for _ in range(12)]
    print(f"temperature {t}: {' '.join(picks)}")

temperature 0: blue blue blue blue blue blue blue blue blue blue blue blue
temperature 0.5: blue grey blue blue blue blue blue blue blue blue grey blue
temperature 1.5: blue orange grey blue blue blue grey grey blue blue orange blue