AI Engineering Tutorial · Quality and Production · Lesson 23 of 25
Caching: Prompts and Responses
Two kinds of caching: prompt caching inside the model API, and caching whole responses in your app, with care.
Prompt caching (provider side)
Mark a stable prefix (system prompt, tool definitions, long documents) as cacheable. Later requests that start with exactly the same prefix are processed faster and cheaper. Put anything that changes (the user's question, dates, names) after the cached part.
Response caching (your side)
For very common questions with stable answers ("what are your opening hours?"), you can cache the final answer. Key it by the normalised question plus the content version, never cache personal answers, and invalidate when the underlying content changes.
import hashlib
def cache_key(question, content_version):
normalised = " ".join(question.lower().replace("?", "").split())
return hashlib.sha256(f"{content_version}|{normalised}".encode()).hexdigest()[:12]
cache = {}
for q, version in [("What are your opening hours?", "kb-v7"), ("what are your opening hours", "kb-v7"),
("What are your opening hours?", "kb-v8")]:
key = cache_key(q, version)
status = "hit" if key in cache else "miss"
cache.setdefault(key, "Mon-Fri 9-17")
print(f"{status:4} {key} {version} {q!r}")
Output
miss 5d9b1c7be16f kb-v7 'What are your opening hours?' hit 5d9b1c7be16f kb-v7 'what are your opening hours' miss 87b62dfb49b3 kb-v8 'What are your opening hours?'
See Prompt caching.
Test yourself
Why include the content version in a response-cache key?
To make keys longerSo answers are not reused after the underlying content changesProviders require itTo cache personal data safely
Show answer
Answer: So answers are not reused after the underlying content changes
A new content version produces new keys, so stale answers are never served.