Why doesn't increasing k fix poor retrieval ordering?

More chunks add noise and similar-but-wrong passages that distract the model, and they cost context and latency. The problem is ordering precision, which a reranking stage addresses.

How does a cross-encoder reranker differ from first-stage retrieval?

It reads the question and each passage together and scores their relevance directly, which is more accurate but too slow for millions of items, so it is applied only to a candidate set.

Should a RAG system always pass k chunks to the model?

No. Dropping candidates below a relevance threshold lets the system answer 'I couldn't find this' instead of answering from irrelevant text; calibrate the threshold on evaluation data.

Reranking: Top-k Is Not Top Quality — RAG in Production

Scenario

Example company: a legal-tech assistant retrieves the top 5 chunks from 2 million contract clauses and asks the model to answer from them.

What you see in production

Lawyers report answers that are nearly right: the cited clause is about the right topic but not the one that applies. Inspecting retrieval shows the correct clause is usually there, at rank 10 to 20, just below the cut-off. Increasing k to 30 makes answers worse and slower: the model is distracted by many similar clauses.

The naive code

chunks = search(question, k=5)                  # first-stage scores decide everything
answer = llm(question, chunks)

Reproduce it

First-stage scores versus a reranker that reads the question and passage together:

Retrieve 20, rerank to 3Try it yourself

question = "Can the customer terminate early if the supplier misses the delivery date?"
candidates = [
    ("c1", "Termination for convenience by either party with 90 days notice.", 0.86),
    ("c2", "Delivery dates are estimates and may change.", 0.85),
    ("c3", "The term of this agreement is 24 months.", 0.84),
    ("c4", "Late payment interest accrues at 1% per month.", 0.83),
    ("c5", "Either party may assign this agreement with consent.", 0.82),
] + [(f"c{i}", f"Unrelated clause {i}.", 0.80 - i / 1000) for i in range(6, 14)] + [
    ("c14", "If the supplier fails to deliver within 30 days of the agreed date, the customer may terminate "
            "immediately and receive a refund of prepaid fees.", 0.74),
]

def first_stage(k):
    return [cid for cid, _, s in sorted(candidates, key=lambda c: -c[2])[:k]]

def rerank(ids, top_n):
    # stands in for a cross-encoder or model reranker: scores how well the passage answers the question
    need = {"terminate", "supplier", "deliver", "date"}
    text = {cid: t.lower() for cid, t, _ in candidates}
    score = {cid: sum(w in text[cid] for w in need) for cid in ids}
    return sorted(ids, key=lambda cid: -score[cid])[:top_n]

print("top 5 by first-stage score:", first_stage(5))
print("retrieve 20, rerank to 3:  ", rerank(first_stage(20), 3))

question = "Can the customer terminate early if the supplier misses the delivery date?"
candidates = [
    ("c1", "Termination for convenience by either party with 90 days notice.", 0.86),
    ("c2", "Delivery dates are estimates and may change.", 0.85),
    ("c3", "The term of this agreement is 24 months.", 0.84),
    ("c4", "Late payment interest accrues at 1% per month.", 0.83),
    ("c5", "Either party may assign this agreement with consent.", 0.82),
] + [(f"c{i}", f"Unrelated clause {i}.", 0.80 - i / 1000) for i in range(6, 14)] + [
    ("c14", "If the supplier fails to deliver within 30 days of the agreed date, the customer may terminate "
            "immediately and receive a refund of prepaid fees.", 0.74),
]

def first_stage(k):
    return [cid for cid, _, s in sorted(candidates, key=lambda c: -c[2])[:k]]

def rerank(ids, top_n):
    # stands in for a cross-encoder or model reranker: scores how well the passage answers the question
    need = {"terminate", "supplier", "deliver", "date"}
    text = {cid: t.lower() for cid, t, _ in candidates}
    score = {cid: sum(w in text[cid] for w in need) for cid in ids}
    return sorted(ids, key=lambda cid: -score[cid])[:top_n]

print("top 5 by first-stage score:", first_stage(5))
print("retrieve 20, rerank to 3:  ", rerank(first_stage(20), 3))

Output

top 5 by first-stage score: ['c1', 'c2', 'c3', 'c4', 'c5']
retrieve 20, rerank to 3:   ['c14', 'c2', 'c1']

Root cause

First-stage retrieval (vectors, BM25) is built for speed over millions of items: it scores the question and each passage separately. That is good at finding candidates but poor at fine-grained relevance. Raising k gives the model more noise; what is missing is a precise second stage.

The fix

Two stages: retrieve wide, then rerank narrow.

candidates = hybrid_search(question, k=50)            # fast, high recall
ranked = reranker.rank(question, candidates)           # slower, reads question and passage together
context = ranked[:5]

Use a cross-encoder reranker or a reranking API; a small LLM asked to score relevance also works for lower volumes.
Tune the two numbers separately: candidates (recall) and final k (precision and context size).
Drop candidates below a relevance threshold rather than always passing k chunks; "nothing relevant" is a valid outcome.

Trade-offs

Reranking adds latency (tens to hundreds of milliseconds for 50 candidates) and cost per query.
A reranker trained on general web data may need evaluation on your domain (legal, medical, code).
A threshold that is too strict causes "I don't know" answers; calibrate it on an evaluation set.

Detect it in production

Rank of the correct passage in first-stage results, on an evaluation set: if it is often between 5 and 50, reranking will help.
Answer quality and latency before and after enabling reranking (A/B or offline evaluation).

Key points

First-stage retrieval finds candidates; it does not order them precisely.
Retrieve many, rerank to few, and allow "nothing relevant".
Tune candidate count and final k separately using evaluation data.

Further reading: Introducing contextual retrieval (with reranking results) · Lost in the middle: how language models use long contexts