Which stage usually dominates the cost and latency of a RAG answer?

The final model call: its input size (context chunks plus prompt) drives cost, and generating the full answer drives latency.

Name three changes that cut RAG latency without changing the model.

Pass fewer, reranked chunks; cache the stable prompt prefix; and stream the answer so the first token arrives early (also skip unnecessary steps such as rewriting first messages).

What makes answer caching safe in a RAG system?

Caching only stable, non-personal content, keying by normalised question plus content version, and invalidating when the underlying documents change.

Cost and Latency of a RAG Pipeline — RAG in Production

Scenario

Example company: a public docs assistant serves 200,000 questions a month. Finance flags the bill, and users complain that answers take 6 seconds to start appearing.

What you see in production

Tracing shows the time and money spread across stages: a query-rewrite call, an embedding call, a vector search, a reranker, and a final call with 30 chunks of context and a long system prompt. About a third of questions are near-repeats of popular ones ("how do I reset my password?").

The naive code

query = rewrite(history, message)                     # always, even for first messages
vector = embed(query)
chunks = rerank(query, search(vector, k=100))[:30]    # 30 chunks into the prompt
answer = llm(SYSTEM_PROMPT + format(chunks) + message)  # nothing cached, nothing streamed

Reproduce it

A per-stage budget for one question, before and after changes:

Where the time goes, stage by stageTry it yourself

naive = {"rewrite": 450, "embed": 120, "search": 60, "rerank 100": 380, "llm (30 chunks, no cache)": 4800}
tuned = {"rewrite (skipped on first turn)": 0, "embed (cached query)": 5, "search": 60,
         "rerank 40": 160, "llm (8 chunks, cached prefix, streamed first token)": 900}

for name, stages in [("naive", naive), ("tuned", tuned)]:
    total = sum(stages.values())
    slowest = max(stages, key=stages.get)
    print(f"{name}: {total} ms to first token; slowest stage: {slowest} ({stages[slowest]} ms)")

Output

naive: 5810 ms to first token; slowest stage: llm (30 chunks, no cache) (4800 ms)
tuned: 1125 ms to first token; slowest stage: llm (8 chunks, cached prefix, streamed first token) (900 ms)

Root cause

Each stage adds latency and cost, and the final model call dominates both: its input size (context chunks plus system prompt) drives cost, and waiting for the full answer drives perceived latency. Without per-stage measurement, teams optimise the wrong stage, for example switching vector databases to save 20 ms.

The fix

Measure each stage in traces (latency, tokens, cost) and fix the largest first.
Shrink the context: rerank to fewer, better chunks (often 5 to 10) instead of passing 30.
Cache the stable prefix (system prompt, instructions) with prompt caching.
Stream the answer so the first words appear quickly.
Skip unnecessary steps: no query rewrite on first turns; no reranker for exact-match lookups.
Cache repeated work: query embeddings for frequent queries; for very common questions, cache answers with care (keyed by normalised question and content version, invalidated when sources change).
Route by difficulty: a smaller model for simple factual answers.

Trade-offs

Answer caching risks serving stale or wrong answers; scope it to stable, public content and include the content version in the key.
Fewer chunks reduce recall if reranking is weak; check recall@k on your evaluation set when reducing k.
Smaller models may lower faithfulness; evaluate before routing.

Detect it in production

p50 and p95 time to first token, and cost per answer, by stage.
Cache hit rates (prompt cache, embedding cache, answer cache).
Context tokens per answer over time.

Key points

Measure cost and latency per stage; the final model call usually dominates.
Fewer, better chunks, a cached prefix and streaming give the biggest wins.
Cache repeated work carefully and skip steps that add nothing.

Further reading: Prompt caching · Streaming messages · Reducing latency