Cache answers to frequent questions safely

Approach

Cache only public, non-personal answers. Key them by a normalised question plus the content version, so a content update invalidates every cached answer at once. Set a TTL as a safety net.

Solution

import hashlib
import json

import redis

r = redis.Redis()
TTL_SECONDS = 24 * 3600


def normalise(question: str) -> str:
    return " ".join(question.lower().replace("?", "").split())


def cache_key(question: str, content_version: str) -> str:
    digest = hashlib.sha256(f"{content_version}|{normalise(question)}".encode()).hexdigest()
    return f"faq-answer:{digest}"


def answer_public_question(question: str, content_version: str, generate) -> dict:
    key = cache_key(question, content_version)
    if (hit := r.get(key)) is not None:
        return {**json.loads(hit), "cached": True}
    result = generate(question)                  # your RAG pipeline: {"answer": ..., "sources": [...]}
    if result.get("sources"):                     # only cache grounded answers
        r.set(key, json.dumps(result), ex=TTL_SECONDS)
    return {**result, "cached": False}

Pitfalls

• Caching answers that depend on the user (orders, account data).
• Forgetting the content version, so policy changes keep showing old answers until the TTL expires.
• Semantic (embedding-similarity) caches with loose thresholds that return answers to different questions.

Variations

• Pre-generate and review answers for the top 50 questions.
• Combine with prompt caching for the uncached questions.

Further reading: Prompt caching