Why can adding more retrieved chunks make answers worse?

Duplicates and conflicting versions waste the context and confuse the model, and relevant passages in the middle of a long context are used less reliably.

What should each chunk in the assembled context be labelled with?

Its document title or ID, section, version and date, so the model can distinguish sources and cite them.

Where should long documents go relative to the question in a prompt?

Documents first and the question and instructions at the end, which tends to improve answers over long contexts.

Lost in the Middle: Assembling the Context — RAG in Production

Scenario

Example company: a product-docs assistant retrieves 40 chunks per question "to be safe" and pastes them into the prompt in the order the vector store returned them.

What you see in production

Answers ignore the most relevant passage surprisingly often, especially when it lands in the middle of the context. Cost per question is high, and many chunks are near-duplicates: the same paragraph from the docs for versions 3.1, 3.2 and 3.3.

The naive code

chunks = search(question, k=40)
context = "\n\n".join(c.text for c in chunks)               # duplicates, no labels, arbitrary order

Reproduce it

Deduplicate, keep only the current version, label each chunk and fit a token budget:

From 40 noisy chunks to a compact, labelled contextTry it yourself

chunks = []
for v in ("3.1", "3.2", "3.3"):
    for i in range(12):
        chunks.append({"doc": f"guide-{i}", "version": v, "score": 0.9 - i * 0.02,
                       "text": f"Section {i}: configuring retries for the client " + "detail " * 40})
chunks += [{"doc": "release-notes", "version": "3.3", "score": 0.95,
            "text": "In 3.3 the default retry limit changed from 3 to 5. " + "detail " * 10}] * 4

def tokens(text):
    return len(text.split())

def assemble(chunks, current="3.3", budget=600):
    seen, picked, used = set(), [], 0
    for c in sorted(chunks, key=lambda c: -c["score"]):
        key = (c["doc"], c["text"][:60])
        if c["version"] != current or key in seen:
            continue
        seen.add(key)
        if used + tokens(c["text"]) > budget:
            break
        picked.append(c)
        used += tokens(c["text"])
    return picked, used

naive_tokens = sum(tokens(c["text"]) for c in chunks)
picked, used = assemble(chunks)
print(f"naive: {len(chunks)} chunks, {naive_tokens} tokens")
print(f"assembled: {len(picked)} chunks, {used} tokens; first: [{picked[0]['doc']} v{picked[0]['version']}]")

chunks = []
for v in ("3.1", "3.2", "3.3"):
    for i in range(12):
        chunks.append({"doc": f"guide-{i}", "version": v, "score": 0.9 - i * 0.02,
                       "text": f"Section {i}: configuring retries for the client " + "detail " * 40})
chunks += [{"doc": "release-notes", "version": "3.3", "score": 0.95,
            "text": "In 3.3 the default retry limit changed from 3 to 5. " + "detail " * 10}] * 4

def tokens(text):
    return len(text.split())

def assemble(chunks, current="3.3", budget=600):
    seen, picked, used = set(), [], 0
    for c in sorted(chunks, key=lambda c: -c["score"]):
        key = (c["doc"], c["text"][:60])
        if c["version"] != current or key in seen:
            continue
        seen.add(key)
        if used + tokens(c["text"]) > budget:
            break
        picked.append(c)
        used += tokens(c["text"])
    return picked, used

naive_tokens = sum(tokens(c["text"]) for c in chunks)
picked, used = assemble(chunks)
print(f"naive: {len(chunks)} chunks, {naive_tokens} tokens")
print(f"assembled: {len(picked)} chunks, {used} tokens; first: [{picked[0]['doc']} v{picked[0]['version']}]")

Output

naive: 40 chunks, 1776 tokens
assembled: 13 chunks, 585 tokens; first: [release-notes v3.3]

Root cause

More context is not more information when it is repetitive. Duplicates waste the budget, conflicting versions confuse the model, and research shows models use information at the start and end of a long context more reliably than information in the middle. Without labels, the model also cannot say which document a passage came from.

The fix

Deduplicate near-identical chunks (same document and text hash, or high similarity).
Filter to the right version, locale and product using metadata before assembling.
Order deliberately: most relevant first; keep chunks from the same document together in reading order.
Label each chunk with title, section, version and date, for example in XML-style tags.
Budget tokens for context and stop adding chunks when the budget is reached.
Put long documents before the question and instructions, which tends to work better for long contexts.

<document id="release-notes" version="3.3" section="Retries">
In 3.3 the default retry limit changed from 3 to 5.
</document>
...
Question: What is the default retry limit?

Trade-offs

Aggressive deduplication can drop a passage that differs in one important word; compare normalised text, not just document IDs.
A tight budget can cut a needed chunk; reranking before assembly makes the budget go to the best passages.
Version filters require reliable metadata at indexing time.

Detect it in production

Context tokens per question and the share of duplicate chunks.
Answers that mention the wrong version or conflicting facts.
Evaluation results by position of the gold passage in the context.

Key points

Deduplicate, filter by version, label and order the retrieved passages.
Give the context a token budget instead of a large fixed k.
Put documents first, then instructions and the question.

Further reading: Lost in the middle: how language models use long contexts · Long context prompting tips