Decision guide
RAG Chunking Strategies Compared
How should you split documents into chunks for retrieval-augmented generation?
Side by side
| Criterion | Fixed-size with overlap | Structure-aware | Semantic chunking | Contextual retrieval | Small-to-big (parent-child) |
|---|---|---|---|---|---|
| Retrieval precision | Medium | High on structured docs | Medium-high | High | High |
| Context given to the model | Fragments | Coherent sections | Coherent passages | Chunks plus context | Full parent sections |
| Indexing cost | Lowest | Low | Medium (embeds sentences) | Highest (a model call per chunk) | Low-medium |
| Implementation effort | Trivial | Parser per format | Medium | Medium | Medium (two levels) |
| Best for | Baselines | Docs and wikis | Transcripts | Ambiguous chunks | Answers needing surrounding context |
When to choose each
Fixed-size with overlap
Split every N tokens with some overlap between neighbouring chunks.
Choose when
- Unstructured text with no reliable headings
- A quick baseline to measure against
Avoid when
- Documents with headings, tables or code, which get cut mid-way
Structure-aware
Split along headings, paragraphs, list items and code blocks; prefix each chunk with its heading path.
Choose when
- Docs, handbooks, wikis and other structured content
- Most production RAG over business documents
Avoid when
- Content with no structure (transcripts, chat logs)
Semantic chunking
Embed sentences and split where meaning shifts (similarity between neighbours drops).
Choose when
- Long unstructured text such as transcripts
- Topics change without headings
Avoid when
- Large corpora where the extra embedding cost matters
Contextual retrieval
Before embedding, add a short generated description of how each chunk fits in its document.
Choose when
- Chunks that lose meaning out of context ("revenue grew 3%" of what?)
- When retrieval failures trace back to missing context
Avoid when
- Very large or fast-changing corpora where per-chunk generation is too costly
Small-to-big (parent-child)
Index small chunks for precise matching, but return their larger parent section to the model.
Choose when
- Precise matching but answers that need surrounding context
- Long sections with specific facts
Avoid when
- Tight context budgets
Verdict
Use structure-aware chunking with heading prefixes as the default for business documents, keep fixed-size chunks as a baseline, and add contextual retrieval or small-to-big where evaluation shows chunks lack context. Use semantic chunking for unstructured transcripts. Decide with a retrieval evaluation set (recall@k on real questions), not intuition.
Learn more
- Integrating LLM APIs
- RAG in Production
- ETL and Data Pipelines
- Build hybrid RAG search on PostgreSQL with pgvector (AI & LLM APIs)
- Answer with citations that point at the source text (AI & LLM APIs)