Forward Deployed Engineering in Practice · Capstone Case Studies
Capstone: Deploying an AI Document Assistant at a Bank
A bank wants an assistant that drafts answers to customer complaints from policies and case history, in 12 weeks. Plan the deployment end to end.
Scenario
Example customer: a retail bank receives 9,000 written complaints a month. Case handlers spend about 40 minutes per complaint finding the relevant policies, account history and previous decisions, then drafting a response that compliance reviews. The bank wants an AI assistant that drafts responses with citations for handlers to edit and approve.
Constraints: data must stay in the bank's cloud tenancy; SSO via the bank's identity provider; outbound traffic through a TLS-inspecting proxy; model use approved by the model-risk team; complaint data includes personal and financial information; regulators expect explainability and audit trails.
Requirements
- Cut handling time per complaint by at least 30% without increasing compliance rework.
- Every draft cites the policies and records it used.
- No personal data leaves the bank's environment except to approved model endpoints in-region.
- Pilot with 20 handlers in week 8; decision on rollout in week 12.
Deliverables
- Discovery questions and success criteria.
- Architecture and deployment model.
- Integrations and data handling.
- Evaluation and guardrails.
- A 12-week plan with critical path and risks.
Hints
- The security, model-risk and data-access approvals are on the critical path.
- Retrieval quality and permissions matter more than the model choice.
- Handlers' edits are the best evaluation data you'll get.
Draft each deliverable before revealing the model answer and rubric.
Further reading: NIST AI Risk Management Framework · OWASP Top 10 for LLM applications
Deliverables
Discovery questions and success criteria.
Model answer
Discovery: shadow handlers on 10 real complaints; which sources they use (policy library, CRM case history, core banking transactions, previous decisions); what compliance rejects drafts for; complaint categories and volumes; languages; who owns each data source; what model and data approvals exist.
Success criteria: median handling time down at least 30% for pilot users; compliance rework rate not higher than baseline; at least 90% of drafts with correct citations on the evaluation set; zero access-control violations; handler satisfaction above an agreed score.
Rubric
Full marks: grounded in observed workflow, includes measurable outcome, quality and safety criteria with baselines.
Architecture and deployment model.
Model answer
Deploy into the bank's cloud tenancy (BYOC) via Terraform/Helm: ingestion workers, a vector + keyword index, an orchestration service, and a web UI embedded in the case-handling tool. Model calls go to an approved in-region model endpoint through private connectivity or the bank's proxy (custom CA configured). SSO via the bank's IdP (OIDC/SAML) with SCIM groups mapped to roles; secrets in the bank's vault; logs stay in the bank's logging platform with redaction.
Rubric
Full marks: respects data-residency and network constraints, identity integration, and keeps telemetry inside the bank.
Integrations and data handling.
Model answer
- Policy library: nightly incremental sync with metadata (effective date, status, product, jurisdiction); superseded policies excluded.
- CRM case history and decisions: read-only API with a service account scoped to complaint-related objects; permission-aware retrieval mirrors case access rules.
- Transactions: fetched on demand for the case being worked, never bulk-indexed.
- Personal data minimised in prompts (only fields needed), redacted from logs; retention aligned with the bank's policy; full audit trail of sources, drafts, edits and approvals.
Rubric
Full marks: metadata-aware sync, least-privilege access, data minimisation, permission-aware retrieval and auditability.
Evaluation and guardrails.
Model answer
Golden set of 200 historical complaints with approved responses and cited policies, labelled with the compliance team; metrics: citation correctness, policy accuracy, tone/completeness rubric, and handler edit distance. Gate prompt, model and retrieval changes on it. Guardrails: drafts only (human approval always), required citations, refusal when no current policy is found, PII checks on output, prompt-injection checks on retrieved content, and weekly sampled compliance review in the pilot. Handler edits feed back into the eval set.
Rubric
Full marks: domain-expert golden set, multiple metrics, CI gating, human-in-the-loop and specific guardrails.
12-week plan, critical path and risks.
Model answer
- Weeks 1-2: discovery and shadowing, success criteria signed, security/model-risk/data-access requests submitted (critical path).
- Weeks 3-5: infrastructure in the bank tenancy, policy sync with metadata, CRM read integration, first eval set.
- Weeks 6-7: drafting workflow in the case tool, guardrails, eval gate in CI, SSO and roles.
- Week 8: pilot with 20 handlers; daily feedback, weekly compliance sampling.
- Weeks 9-11: iterate on retrieval and prompts against the eval set; measure handling time and rework.
- Week 12: results versus criteria, rollout recommendation, handover and product feedback.
Risks: approval delays (start day 1, escalate early), poor policy metadata (content-owner workshop), proxy/TLS issues (connectivity self-check), handler adoption (involve them from week 1).
Rubric
Full marks: approvals on the critical path, value early, evaluation built before the pilot, explicit risks with mitigations.
Open in the app to mark lessons complete, take notes and track your progress.