The Prompt Change That Broke Extraction

Scenario

Example deployment: an FDE runs an LLM-based extraction pipeline that pulls parties, amounts and dates from supplier contracts for a manufacturer's procurement team. A user complains that currency codes are sometimes missing, so the FDE adds a line to the prompt and spot-checks five contracts. All five look right; the change goes live.

What you see in production

A week later the procurement team notices renewal reminders going out for the wrong months. Dates in European format (03/04/2026 meaning 3 April) are now often read as US dates. About 2,400 contracts were processed in that week; the team has to recheck them by hand and stops trusting the system.

The naive code

PROMPT = '''Extract parties, total amount, currency (ISO code, always include it!) and renewal date
from the contract below. Return JSON.'''
# change tested by reading five outputs; no stored test set, no comparison with the previous version

Reproduce it

A golden set turns "looks right" into numbers and catches regressions per field:

golden = [
    {"currency": "EUR", "renewal": "2026-04-03", "total": "12000.00"},
    {"currency": "USD", "renewal": "2026-11-30", "total": "800.00"},
    {"currency": "GBP", "renewal": "2026-02-05", "total": "45000.00"},
    {"currency": "EUR", "renewal": "2027-01-15", "total": "3100.00"},
]
outputs = {
    "v1 (current)": [{"currency": None, "renewal": "2026-04-03", "total": "12000.00"},
                     {"currency": "USD", "renewal": "2026-11-30", "total": "800.00"},
                     {"currency": "GBP", "renewal": "2026-02-05", "total": "45000.00"},
                     {"currency": None, "renewal": "2027-01-15", "total": "3100.00"}],
    "v2 (candidate)": [{"currency": "EUR", "renewal": "2026-03-04", "total": "12000.00"},
                       {"currency": "USD", "renewal": "2026-11-30", "total": "800.00"},
                       {"currency": "GBP", "renewal": "2026-05-02", "total": "45000.00"},
                       {"currency": "EUR", "renewal": "2027-01-15", "total": "3100.00"}],
}
CRITICAL = {"renewal": 1.0}          # renewal dates must never regress

scores = {}
for version, outs in outputs.items():
    scores[version] = {f: sum(o[f] == g[f] for o, g in zip(outs, golden)) / len(golden) for f in golden[0]}
    print(version, {f: f"{s:.0%}" for f, s in scores[version].items()})

old, new = scores["v1 (current)"], scores["v2 (candidate)"]
regressions = [f for f in old if new[f] < old[f]]
blocked = [f for f in regressions if f in CRITICAL]
print("regressions:", regressions, "-> BLOCK release" if blocked else "-> ok")

v1 (current) {'currency': '50%', 'renewal': '100%', 'total': '100%'}
v2 (candidate) {'currency': '100%', 'renewal': '50%', 'total': '100%'}
regressions: ['renewal'] -> BLOCK release

Root cause

Language models respond to the whole prompt; adding an instruction changes behaviour elsewhere in ways that a handful of spot checks won't reveal. There was no evaluation set of real contracts with expected answers, no per-field metrics, and no rule that a change must not regress critical fields. The same risk applies to model upgrades and changes to retrieval or preprocessing.

The fix

1. Build a golden set from real (anonymised) customer documents: 100-300 cases covering formats, languages and edge cases, with expected outputs reviewed by customer experts.
2. Measure per field and per segment (by language, document type, date format), not just an overall score.
3. Gate every change (prompt, model, parameters, preprocessing) in CI: no regression on critical fields, overall above the agreed threshold.
4. Make the format explicit in code where possible: ask the model for dates as written plus the locale, or parse dates deterministically from the extracted text.
5. Monitor in production: sample outputs for human review weekly, add every confirmed error to the golden set.

Trade-offs

• Building the golden set costs expert time; it's the asset that makes every later change safe.
• Model-graded evaluations scale but can share the model's blind spots; anchor them with human-labelled cases.
• Strict gates slow small fixes; that's intended for fields with business impact.

Detect it in production

• Weekly sampled review with accuracy per field, trended.
• User corrections captured in the UI as signals.
• Distribution checks (for example share of dates that fall on day <= 12, which reveals day/month swaps).

Key points

• Any prompt, model or preprocessing change can regress unrelated behaviour.
• A golden set of real cases with per-field metrics catches it before customers do.
• Gate changes in CI and feed production errors back into the set.