RAG in Production · Capstone
Capstone: A Support Knowledge Assistant for 40,000 Articles
Design a multi-tenant RAG assistant over help-centre articles and customer-specific documents, with evaluation and safety.
Scenario
Example company: a B2B software company has 40,000 public help articles in 6 languages and lets each customer upload their own internal runbooks (about 2 million chunks across 1,500 customers). Support wants an assistant inside the product that answers from both, with citations.
Requirements
- Answers cite their sources, and say so when they cannot find the answer.
- A customer must never see another customer's documents.
- Article edits and deletions are reflected within 15 minutes.
- p95 time to first token under 2.5 seconds.
Constraints
- Questions include error codes, product names and configuration flags, as well as natural-language how-to questions.
- Customer documents may contain anything, including text written to manipulate the assistant.
- Budget: an average cost per answer agreed with finance, and the team cannot manually review more than 50 answers a week.
Deliverables
- Ingestion and indexing: chunking, metadata, freshness and deletions.
- Retrieval: search method, filters and ranking.
- Generation and grounding: prompt structure, citations and "not found".
- Security: tenant isolation and injection defences.
- Evaluation and operations: evaluation set, metrics, cost and latency plan.
Hints
- Two corpora (public and tenant) with different trust levels.
- Identifiers need exact matching.
- You can review only 50 answers a week, so choose them wisely.
Draft each deliverable below before revealing the model answer and rubric.
Further reading: Introducing contextual retrieval · OWASP LLM08: Vector and embedding weaknesses
Deliverables
Describe ingestion and indexing.
Model answer
- Structure-aware chunking by headings and steps; tables and code kept whole; each chunk prefixed with article title, heading path, product, version and language, plus a short contextual description.
- Metadata on every chunk: corpus (public or tenant), tenant ID, document ID, language, version, updated_at, URL.
- CMS webhooks trigger re-indexing of changed articles (delete by document ID, re-insert changed chunks by content hash); a nightly reconciliation removes orphans. Tenant uploads are indexed on upload and on delete.
Rubric
Look for structure-aware chunks with context, rich metadata including tenant and version, and incremental sync with deletes plus reconciliation to meet the 15-minute freshness goal.
Describe retrieval.
Model answer
- Rewrite follow-up questions into standalone queries with a small model; detect the language.
- Hybrid search (BM25 with identifier-preserving tokenisation plus vectors) fused with reciprocal rank fusion, filtered inside the query to public content in the user's language plus the user's own tenant.
- Rerank about 50 candidates to at most 8, with a relevance threshold so irrelevant results are dropped.
Rubric
Strong answers combine hybrid search for identifiers, in-query filters for tenant and language, and a reranking stage with a threshold.
Describe generation and grounding.
Model answer
- Documents first, labelled with ID, title, version and trust level; then instructions and the question.
- The model must answer only from documents, cite each claim with document ID and exact quote, and say when it cannot find the answer.
- Citations are verified in code; unverified answers are regenerated once, then fall back to "I couldn't find this" with links to search results.
Rubric
Look for an explicit 'not found' path and code-level citation verification, not just a prompt instruction.
Describe security.
Model answer
- Tenant isolation enforced below the application: tenant chunks in per-tenant namespaces or protected by row-level security; an automated cross-tenant test runs on every deploy.
- Tenant documents are untrusted: hidden text stripped at extraction, instruction-like content flagged, retrieved text wrapped as untrusted data.
- The answering model has no tools; links in answers are limited to allowlisted domains and cited documents.
Rubric
Strong answers enforce isolation below the app, treat tenant content as untrusted, and limit capabilities so injection has little impact.
Describe evaluation and operations.
Model answer
- An evaluation set of 300 real questions across languages and question types with gold passages and key facts; recall@k, MRR, faithfulness and citation validity run in CI on every change.
- The 50 weekly human reviews are chosen from negative feedback, unverified-citation fallbacks and a random sample, and failures are added to the evaluation set.
- Latency and cost per stage from traces; prompt caching for the stable prefix, streaming, fewer reranked chunks, and a smaller model for simple factual questions once evals confirm quality.
Rubric
Look for a labelled evaluation set gating changes, a targeted human review loop, and per-stage cost and latency measurement driving the optimisations.
Open in the app to mark lessons complete, take notes and track your progress.