What are the two error types of a safety guardrail?

Allowing harmful requests (false negatives) and refusing legitimate ones (false positives, over-blocking). Tightening one usually increases the other.

Why is tuning a safety filter only on harmful prompts a mistake?

It measures only one error; the threshold drifts until all harmful prompts are blocked, at the cost of refusing many legitimate questions that were never measured.

What is a graded response, and why prefer it to a refusal?

A response matched to the level of risk, for example answering with safety information or recommending a professional, instead of a flat refusal; it keeps legitimate users helped while limiting harm.

Over-Blocking: Measuring False Refusals — LLM Evals and Guardrails

Scenario

Example company: a pharmacy's assistant had an incident where it gave unsafe dosage advice. The team tightened the safety filter until a test set of harmful prompts was fully blocked.

What you see in production

Customer satisfaction drops. The assistant now refuses "Can I take ibuprofen with my blood pressure medication?", "What's the maximum dose of paracetamol for adults?" (answered on the pack itself) and even "What are your opening hours on Sunday?" when the message also mentions a medicine. Users go back to phoning the pharmacy.

The naive code

if risk_score(message) > 0.2:          # tuned until every harmful test prompt was blocked
    return "I'm sorry, I can't help with that."

Reproduce it

Sweep the threshold and measure both harmful requests allowed and legitimate requests refused:

Threshold sweep: harmful allowed versus legitimate refusedTry it yourself

# (risk score from a classifier, is_harmful) for a labelled set of 20 messages
labelled = [(0.95, 1), (0.88, 1), (0.81, 1), (0.74, 1), (0.66, 1), (0.42, 1),
            (0.58, 0), (0.47, 0), (0.39, 0), (0.33, 0), (0.28, 0), (0.25, 0), (0.22, 0),
            (0.18, 0), (0.12, 0), (0.09, 0), (0.07, 0), (0.05, 0), (0.03, 0), (0.02, 0)]
harmful = sum(y for _, y in labelled)
legit = len(labelled) - harmful

for threshold in (0.2, 0.4, 0.5, 0.7):
    allowed_harmful = sum(1 for s, y in labelled if y and s <= threshold)
    refused_legit = sum(1 for s, y in labelled if not y and s > threshold)
    print(f"threshold {threshold}: harmful allowed {allowed_harmful}/{harmful}, "
          f"legitimate refused {refused_legit}/{legit} ({refused_legit / legit:.0%})")

Output

threshold 0.2: harmful allowed 0/6, legitimate refused 7/14 (50%)
threshold 0.4: harmful allowed 0/6, legitimate refused 2/14 (14%)
threshold 0.5: harmful allowed 1/6, legitimate refused 1/14 (7%)
threshold 0.7: harmful allowed 2/6, legitimate refused 0/14 (0%)

Root cause

Every classifier trades two errors: letting harmful requests through and refusing legitimate ones. Tuning only on harmful examples measures one error and ignores the other, so the threshold drifts until the product becomes useless. Many legitimate questions in sensitive domains look superficially like risky ones.

The fix

Label both kinds: an eval set with harmful and legitimate-but-sensitive requests (the questions real customers ask).
Measure both rates for every threshold or prompt change: harmful-allowed and legitimate-refused.
Choose the threshold deliberately from the business's tolerance for each error, and write that decision down.
Use graded responses instead of a binary block: answer with safety information, answer partially and recommend a pharmacist, or hand off to a human for the riskiest cases.
Review refusals in production regularly and add wrongly refused questions to the eval set.

Trade-offs

There is no threshold with zero errors of both kinds; the business must choose which errors it prefers.
Graded responses are harder to build and test than a single refusal message.
Human handoff costs staff time; reserve it for the narrow band of truly risky requests.

Detect it in production

Refusal rate over time and by topic; sudden increases after a change.
User signals after refusals: rephrasing, abandoning, contacting support.
Sampled review of refusals, labelled as correct or over-blocking.

Key points

Guardrails make two kinds of errors; measure both.
Pick thresholds deliberately, using a labelled set of harmful and legitimate requests.
Prefer graded, helpful responses over a flat refusal.

Further reading: Define your success criteria · NIST AI Risk Management Framework