AI Engineering Tutorial · Retrieval (RAG) · Lesson 17 of 25
Chunking Documents
Split documents into chunks that each make sense on their own, keeping headings so every chunk has context.
Why chunk
Documents are too long to embed and retrieve whole, so you split them into chunks (often a few hundred tokens). Retrieval then returns the chunks most relevant to a question.
Chunk well
- Split along structure (headings, paragraphs), not every N characters, so facts stay together.
- Prefix each chunk with its title and heading, so "Germany: 14 months" doesn't become "14 months" of who knows what.
- Keep tables and code blocks whole.
- Store the source (document ID, URL, updated date) with every chunk.
doc = """# Leave policy
## Germany
Parents receive up to 14 months of parental allowance.
## France
Parents receive 16 weeks of paid leave."""
def chunk_by_heading(text, title):
chunks, heading = [], None
for line in text.splitlines():
line = line.strip()
if line.startswith("## "):
heading = line[3:]
elif line and not line.startswith("# ") and heading:
chunks.append(f"{title} > {heading}: {line}")
return chunks
for c in chunk_by_heading(doc, "Leave policy"):
print(c)
Output
Leave policy > Germany: Parents receive up to 14 months of parental allowance. Leave policy > France: Parents receive 16 weeks of paid leave.
More in RAG in Production: chunking.
Test yourself
Why prefix chunks with their document title and heading?
To make chunks longerSo each chunk makes sense and matches searches on its ownEmbedding APIs require itTo reduce storage
Show answer
Answer: So each chunk makes sense and matches searches on its own
A chunk may be retrieved alone; its heading supplies the missing context.