Why does fixed-size chunking hurt RAG answers?

It splits text regardless of structure, so the facts a question needs (a heading and its values, a table header and its rows) land in different chunks and no single retrieved chunk contains the full answer.

What context should each chunk carry?

The document title and heading path, and optionally a short generated description of how the chunk fits into the document, so it can be understood and matched on its own.

How do you choose a chunking strategy?

Try a few structure-aware strategies and sizes, and compare retrieval recall on an evaluation set of real questions with known source passages.

Chunking: Why the Answer Was Split in Half — RAG in Production

Scenario

Example company: an HR assistant answers questions from a 300-page employee handbook. The first version splits the handbook into 500-character chunks, embeds them and retrieves the top 3 for each question.

What you see in production

Employees asking "How many months of parental leave do I get in Germany?" are told "16 weeks", the number for France. The right answer is in the handbook, but the chunk boundary falls just after the heading "Parental leave: Germany", so the best-matching chunk is the France section with the German heading's first words attached. The chunk with the German numbers no longer says "Germany" near the top.

The naive code

def chunk(text, size=500):
    return [text[i:i + size] for i in range(0, len(text), size)]     # cuts anywhere, even mid-row

Reproduce it

A small handbook, chunked blindly and by section, then searched for the question's words:

Fixed-size chunks lose the heading; section chunks keep itTry it yourself

handbook = (
    "## Parental leave: France\nEmployees in France receive 16 weeks of paid leave. "
    "Partners receive 25 days.\n"
    "## Parental leave: Germany\nEmployees in Germany receive up to 14 months of parental allowance, "
    "shared between parents. Partners can take 14 days of paid leave at birth.\n"
    "## Holidays\nAll employees receive 25 vacation days per year.\n")

def fixed_chunks(text, size=140):
    return [text[i:i + size] for i in range(0, len(text), size)]

def section_chunks(text):
    parts = text.split("## ")[1:]
    return ["## " + p.strip() for p in parts]

def best(chunks, question):
    words = {w.strip("?.,").lower() for w in question.split() if len(w) > 3}
    return max(chunks, key=lambda c: sum(w in c.lower() for w in words))

q = "How many months of parental leave do employees in Germany receive?"
for name, chunks in [("fixed", fixed_chunks(handbook)), ("section", section_chunks(handbook))]:
    top = best(chunks, q)
    print(f"{name:7} -> has the German answer: {'14 months' in top} | {top[:70]!r}")

handbook = (
    "## Parental leave: France\nEmployees in France receive 16 weeks of paid leave. "
    "Partners receive 25 days.\n"
    "## Parental leave: Germany\nEmployees in Germany receive up to 14 months of parental allowance, "
    "shared between parents. Partners can take 14 days of paid leave at birth.\n"
    "## Holidays\nAll employees receive 25 vacation days per year.\n")

def fixed_chunks(text, size=140):
    return [text[i:i + size] for i in range(0, len(text), size)]

def section_chunks(text):
    parts = text.split("## ")[1:]
    return ["## " + p.strip() for p in parts]

def best(chunks, question):
    words = {w.strip("?.,").lower() for w in question.split() if len(w) > 3}
    return max(chunks, key=lambda c: sum(w in c.lower() for w in words))

q = "How many months of parental leave do employees in Germany receive?"
for name, chunks in [("fixed", fixed_chunks(handbook)), ("section", section_chunks(handbook))]:
    top = best(chunks, q)
    print(f"{name:7} -> has the German answer: {'14 months' in top} | {top[:70]!r}")

Output

fixed   -> has the German answer: False | '## Parental leave: France\nEmployees in France receive 16 weeks of paid'
section -> has the German answer: True | '## Parental leave: Germany\nEmployees in Germany receive up to 14 month'

Root cause

Fixed-size chunking ignores the document's structure. The facts a question needs (the country and the number) end up in different chunks, and neither chunk alone answers the question. Retrieval then ranks chunks by partial matches, and the model answers from whichever fragment it gets, sometimes confidently and wrongly.

The fix

Chunk along the structure of the document and give every chunk enough context to stand alone:

Split by headings, paragraphs and list items first; only split further when a section is too long, with a small overlap.
Keep tables and code blocks whole, or repeat the header row in each piece.
Prefix each chunk with its context: document title and heading path, for example Employee handbook > Parental leave > Germany.
Contextual retrieval: generate a one-sentence description of how the chunk fits into the document and index it with the chunk.
Store the chunk's parent ID, so you can return the whole section when a chunk matches.

Trade-offs

Larger chunks carry more context but dilute similarity scores and consume more of the context window.
Contextual descriptions cost one model call per chunk at indexing time; prompt caching of the document makes this cheaper.
There is no single right size: measure retrieval quality on your own questions for two or three chunking strategies.

Detect it in production

Review failed answers: is the right text in the index but split across chunks?
Track chunk length distribution and the share of chunks without a heading.
Retrieval recall on an evaluation set, per chunking strategy (see the evaluation lesson).

Key points

Chunk by document structure, not by character count.
Keep tables, lists and code whole, and prefix chunks with their title and heading path.
Measure retrieval quality for each chunking strategy on real questions.

Further reading: Introducing contextual retrieval · RAG basics in Integrating LLM APIs