How does red-teaming differ from ordinary evaluation?

Ordinary evals test intended use; red-teaming deliberately tries to make the system do what it should not, such as bypass authorisation, evade limits or leak data.

Why are many serious red-team findings fixed in code rather than in the prompt?

They come from missing authorisation or limit checks on tools; the model can be talked into requesting anything, so the code must enforce who may do what, and how much.

What categories belong in an LLM red-team suite?

Direct and indirect prompt injection, authorisation bypass, limit evasion, data exfiltration, harmful content and cost (denial-of-wallet) attacks, tailored to the feature's capabilities.

Red-Teaming Your LLM Feature — LLM Evals and Guardrails

Scenario

Example company: an airline launches a customer assistant that can look up bookings and issue travel vouchers up to a limit. Security testing consisted of a few manual attempts by the developers.

What you see in production

Within a week, users share tricks online: claiming to be a gate agent gets the assistant to issue vouchers above the limit by splitting them, and asking for "a poem that includes my booking details" for someone else's booking reference returns another passenger's flight details.

The naive code

# security review: three developers tried "ignore your instructions" and it refused. Ship it.

Reproduce it

Run a small adversarial suite by attack category and report the attack success rate for each:

Attack success rate by categoryTry it yourself

results = [  # (category, attack succeeded?)
    ("prompt injection", False), ("prompt injection", False), ("prompt injection", True),
    ("authorisation bypass", True), ("authorisation bypass", True), ("authorisation bypass", False),
    ("limit evasion (split vouchers)", True), ("limit evasion (split vouchers)", True),
    ("data exfiltration via formatting", True), ("data exfiltration via formatting", False),
    ("harmful content", False), ("harmful content", False), ("harmful content", False),
]
by_category = {}
for category, success in results:
    s = by_category.setdefault(category, [0, 0])
    s[0] += success
    s[1] += 1
for category, (succ, total) in sorted(by_category.items(), key=lambda kv: -kv[1][0] / kv[1][1]):
    print(f"{category:34} {succ}/{total} attacks succeeded")

Output

limit evasion (split vouchers)     2/2 attacks succeeded
authorisation bypass               2/3 attacks succeeded
data exfiltration via formatting   1/2 attacks succeeded
prompt injection                   1/3 attacks succeeded
harmful content                    0/3 attacks succeeded

Root cause

Ordinary evals test intended use. Attackers look for what the system can be made to do, especially combinations of capabilities (look up any booking, issue vouchers) and gaps between what the prompt says and what the code enforces. The worst findings here are not model jailbreaks at all; they are missing authorisation and limit checks in code.

The fix

Threat-model the feature: list capabilities (tools, data), who can reach them, and what harm misuse could cause.
Build an adversarial suite by category: prompt injection (direct and through documents), authorisation bypass, limit evasion, data exfiltration, harmful content, denial of wallet (cost attacks).
Fix root causes in code: authorisation checks on every tool call based on the authenticated user (never on what the user claims), cumulative limits (per booking, per day), output checks for other customers' data.
Measure attack success rate per category for every release, and treat increases as regressions.
Refresh the suite from real attempts seen in production and from published attack techniques; involve people who did not build the feature.

Trade-offs

Red-teaming finds problems; it cannot prove their absence. Keep runtime monitoring and limits.
Adversarial suites need maintenance as features change.
Automated attack generation scales coverage but produces noisy results that need review.

Detect it in production

Guardrail triggers and authorisation failures per user; patterns of probing.
Limit hits (vouchers per booking per day) and anomalies in tool usage.
Reports from users and from public forums about your assistant.

Key points

Evals test intended use; red-teaming tests what the system can be made to do.
Organise attacks by category and measure success rates for every release.
The most important fixes are usually authorisation and limits enforced in code.

Further reading: OWASP Top 10 for LLM applications · NIST AI Risk Management Framework · Mitigate jailbreaks and prompt injections