Prompt and Model Changes Without Surprises

Scenario

Example company: an assistant's prompt lives in a database field that product managers edit, and the code calls a model by a moving alias.

What you see in production

On Tuesday, responses suddenly become longer and start using bullet lists that break the mobile app's layout. Nobody deployed anything. Investigation finds that a product manager edited the prompt on Monday evening, and the model alias had started pointing at a newer model snapshot the same week. It takes a day to work out which change caused what.

The naive code

prompt = db.get_setting("assistant_prompt")         # edited live, no history, no review
reply = client.messages.create(model=MODEL_ALIAS, system=prompt, ...)

Reproduce it

A canary rollout compares the new version with the current one on live traffic before switching everyone:

import random
random.seed(9)

def answer_ok(version, request_type):
    base = {"v12": 0.91, "v13": 0.93}[version]
    if version == "v13" and request_type == "mobile":
        base = 0.70                                   # the new prompt's bullet lists break the mobile layout
    return random.random() < base

stats = {"v12": {}, "v13": {}}
for _ in range(4000):
    request_type = random.choice(["web", "web", "mobile"])
    version = "v13" if random.random() < 0.10 else "v12"
    ok_total = stats[version].setdefault(request_type, [0, 0])
    ok_total[0] += answer_ok(version, request_type)
    ok_total[1] += 1

for version, by_type in stats.items():
    summary = {t: f"{ok / n:.0%} of {n}" for t, (ok, n) in sorted(by_type.items())}
    print(version, summary)

v12 {'mobile': '92% of 1238', 'web': '90% of 2329'}
v13 {'mobile': '71% of 147', 'web': '95% of 286'}

Root cause

Behaviour depends on the prompt, the model version, the tools and the retrieval pipeline. When any of these can change without versioning, review or evaluation, changes collide and cannot be attributed. Model aliases that move to new snapshots change behaviour without a deploy.

The fix

• Version prompts in code (or a prompt registry with history and review), with an ID logged on every request.
• Pin model versions to specific snapshots in production; upgrade deliberately.
• Change one thing at a time, and run the eval suite on every change (prompt, model, tools, retrieval).
• Roll out gradually: shadow-run the new version on copied traffic or send a small canary percentage, compare graded quality by segment, then increase.
• Keep a fast rollback: switching back to the previous prompt or model version should be a configuration change.

Trade-offs

• Prompt changes through code review are slower for non-engineers; a registry with approvals and evals keeps them safe without a full deploy.
• Pinned models eventually reach end of life; plan upgrades as projects with evals.
• Canaries need enough traffic per segment to show differences; small products may need longer canaries or shadow evaluation.

Detect it in production

• Prompt version and model ID on every trace, so quality metrics can be split by version.
• Canary dashboards comparing versions by request type.
• Alerts when a model alias or prompt changes outside the release process.

Key points

• Version prompts, pin model snapshots and log both on every request.
• Evaluate every change and change one thing at a time.
• Roll out with shadow traffic or canaries by segment, and keep rollback instant.

Further reading: Models overview: model names and snapshots · Increase output consistency