Output Guardrails: Validating What the Model Says

Scenario

Example company: an e-commerce site uses a model to generate product descriptions as JSON (title, bullets, price_note) from supplier data, inserted directly into product pages.

What you see in production

Three incidents in a month: a page broke because the model returned JSON wrapped in a code fence; a description included a competitor's discount code copied from supplier notes; and a supplier field containing <script> ended up in a bullet point, rendered unescaped on the product page.

The naive code

data = json.loads(llm(prompt))                 # crashes on fences or trailing text
page.bullets = data["bullets"]                 # rendered as raw HTML

Reproduce it

Validate structure, then content rules, then escape before rendering:

import html, json, re

def parse(raw):
    raw = raw.strip()
    if raw.startswith("```"):
        raw = raw.strip("`").removeprefix("json").strip()
    return json.loads(raw)

def validate(data):
    errors = []
    if not isinstance(data.get("title"), str) or not 5 <= len(data["title"]) <= 80:
        errors.append("title must be 5-80 characters")
    if not isinstance(data.get("bullets"), list) or not 2 <= len(data["bullets"]) <= 5:
        errors.append("bullets must be a list of 2-5 items")
    text = " ".join([data.get("title", "")] + data.get("bullets", []))
    if re.search(r"\b(code|coupon)\s+[A-Z0-9]{5,}\b", text, re.I):
        errors.append("policy: no discount codes")
    return errors

raw = '```json\n{"title": "Trail running shoe", "bullets": ["Grippy sole", "Use code SAVE20NOW at checkout", "<script>alert(1)</script>"]}\n```'
data = parse(raw)
print("validation:", validate(data) or "ok")
data["bullets"] = [b for b in data["bullets"] if "code" not in b.lower()]
print("validation after fix:", validate(data) or "ok")
print("rendered:", [html.escape(b) for b in data["bullets"]])

validation: ['policy: no discount codes']
validation after fix: ok
rendered: ['Grippy sole', '<script>alert(1)</script>']

Root cause

Model output is untrusted input to the rest of your system. It can be malformed, it can copy anything that was in its input (including attacker-controlled text), and it can break policies you never stated. Code that consumes it directly inherits all of these problems: crashes, injection into HTML or SQL, and content you would never publish.

The fix

• Structure: use structured outputs or tool calls with a JSON schema where available; still parse defensively and validate with a schema library (Pydantic, Zod).
• Content rules in code: lengths, allowed values, forbidden patterns (discount codes, URLs, phone numbers), required facts.
• Policy checks: a moderation or classifier step for categories you must not publish.
• Escape for the destination: HTML-escape for pages, parameterised queries for SQL, never eval or shell out with model text.
• On failure: retry once with the validation errors in the prompt, then fall back (template text, human review queue) rather than publishing.

Trade-offs

• Every check adds latency; run cheap code checks first, model-based checks only when needed.
• Retries cost money; cap them and track the failure rate per rule to improve the prompt.
• Overly strict content rules block good output; review rejected samples.

Detect it in production

• Validation failure rate per rule and per prompt version.
• Fallback and human-review queue sizes.
• Security scanning (for example CSP violation reports) on pages that render model output.

Key points

• Treat model output as untrusted input to your system.
• Validate structure and content in code, and check policy before publishing.
• Escape for the destination, retry with errors once, then fall back safely.

Further reading: OWASP LLM05: Improper output handling · Structured outputs · Pydantic