AI Engineering Tutorial · Quality and Production · Lesson 22 of 25
Estimating and Controlling Cost
Estimate cost per request and per month, and know the levers: model choice, context size, output length and caching.
The formula
cost = input_tokens × input_price + output_tokens × output_price (prices are per million tokens).
The levers
- Model choice: small models for simple steps (classification, routing).
- Context size: send fewer, better chunks and trim conversation history.
- Output length: ask for concise answers; set a sensible
max_tokens. - Prompt caching: a stable prefix (system prompt, tools, long documents) is billed at a fraction on reuse.
- Batch APIs: non-urgent jobs at a discount.
PRICES = {"large": (3.00, 15.00), "small": (0.80, 4.00)} # illustrative $ per million tokens (in, out)
def monthly(requests, model, input_tokens, output_tokens, cached_share=0.0, cache_price=0.1):
p_in, p_out = PRICES[model]
effective_in = input_tokens * ((1 - cached_share) + cached_share * cache_price)
return requests * (effective_in * p_in + output_tokens * p_out) / 1e6
base = monthly(300_000, "large", 6000, 400)
print(f"large model, 6k input: ${base:,.0f}/month")
print(f"+ trim context to 3k: ${monthly(300_000, 'large', 3000, 400):,.0f}/month")
print(f"+ cache 2k-token stable prefix: ${monthly(300_000, 'large', 3000, 400, cached_share=2/3):,.0f}/month")
print(f"small model for the same calls: ${monthly(300_000, 'small', 3000, 400, cached_share=2/3):,.0f}/month")
Output
large model, 6k input: $7,200/month + trim context to 3k: $4,500/month + cache 2k-token stable prefix: $2,880/month small model for the same calls: $768/month
See Pricing and Prompt caching.
Test yourself
A 4,000-token system prompt is identical on every request. What reduces its cost most directly?
Raising temperaturePrompt cachingStreamingIncreasing max_tokens
Show answer
Answer: Prompt caching
Caching a stable prefix bills those repeated tokens at a fraction of the normal price.