LLM Evals and Guardrails · Capstone
Capstone: An Eval and Guardrail Plan for a Banking Assistant
Design the evaluation, guardrails and monitoring for an assistant that answers account questions and can move money between a customer's own accounts.
Scenario
Example company: a retail bank is launching an assistant in its mobile app. It answers product and account questions (balances, transactions, fees) and can transfer money between the customer's own accounts up to 5,000 per day. It runs on a hosted model with tools for account data and transfers.
Requirements
- Factual answers about fees and products must match current policy documents.
- Transfers are only between the authenticated customer's own accounts, within limits, with confirmation.
- The bank must show regulators how quality and safety are measured and monitored.
Constraints
- Customers write in English and Spanish; many messages contain account numbers and personal details.
- Policy documents change monthly.
- The model provider releases new model versions several times a year.
Deliverables
- The eval set: sources, types and success criteria.
- Graders: which criteria use code, model graders or humans, and how graders are calibrated.
- Guardrails: input, tool-level and output controls, including personal data.
- Release process: CI gates, prompt and model changes, rollout.
- Production monitoring and red-teaming.
Hints
- Separate "answered correctly" from "acted safely".
- Some checks belong in the tools, not around the model.
- Regulators will ask how you know, not just what you did.
Draft each deliverable below before revealing the model answer and rubric.
Further reading: NIST AI Risk Management Framework · Demystifying evals for AI agents
Deliverables
Describe the eval set.
Model answer
- 400 cases from anonymised pilot conversations and support tickets, stratified by intent (fees, products, balances, transactions, transfers, out of scope), language (English, Spanish) and risk.
- Each case has success criteria: key facts with their source document, forbidden content, expected tool calls or refusal, and for transfers the expected final account state.
- Adversarial cases: other customers' accounts, limit evasion (several small transfers), injection in transaction descriptions.
- Policy-dependent cases are tagged with the document version and refreshed when policies change.
Rubric
Look for real-traffic sources, stratification by intent, language and risk, explicit success criteria, adversarial cases and a plan for policy changes.
Describe the graders.
Model answer
- Code graders: tool calls and arguments, final account state for transfer tasks, limit checks, no personal data of others in output, refusal on out-of-scope requests.
- Model graders with rubrics: factual correctness against cited policy text, faithfulness, tone; one criterion per prompt, PASS or FAIL.
- Humans: compliance staff label 100 cases each quarter; model-grader agreement (kappa) must stay above an agreed level, otherwise rubrics are revised.
Rubric
Strong answers prefer code graders for safety-relevant checks and calibrate model graders against human labels with a stated agreement threshold.
Describe the guardrails.
Model answer
- Tools enforce authorisation from the authenticated session (never from the message): only the customer's own accounts, daily cumulative limit of 5,000, explicit in-app confirmation for every transfer.
- Input: an injection and abuse screen; personal data redacted before logging and tracing; account numbers looked up by tools rather than pasted.
- Output: schema validation, a check that no other customer's data or internal notes appear, escaping for the app's renderer, and graded responses rather than flat refusals for sensitive questions.
Rubric
Look for authorisation and limits enforced in tools, confirmation for money movement, redaction at boundaries and output checks.
Describe the release process.
Model answer
- Prompts versioned in the repository; model snapshots pinned; every change runs the full eval suite in CI.
- Gates: zero regressions in transfer safety and authorisation cases; no drop beyond the confidence interval in factual accuracy by language.
- Rollout: shadow evaluation on copied traffic, then a 5 % canary compared by segment, then full rollout; one-step rollback.
- Model upgrades are planned projects with the same gates.
Rubric
Strong answers combine pinned versions, per-type gates with strict rules for safety cases, and gradual rollout with fast rollback.
Describe monitoring and red-teaming.
Model answer
- Daily graded samples of live conversations by intent and language against a baseline; user feedback, escalation and refusal rates; alerts on drops.
- Monitoring of transfer volumes, limit hits and authorisation failures per user.
- Quarterly red-team exercises by people outside the team across injection, authorisation, limit evasion and data exfiltration; attack success rates reported per release.
- A quarterly report for compliance with eval results, grader calibration, incidents and changes.
Rubric
Look for continuous quality measurement on live traffic, safety-specific monitoring, independent red-teaming and documentation regulators can review.
Open in the app to mark lessons complete, take notes and track your progress.