If someone stopped me in a corridor and asked, “What’s the single most important thing to get right in a RAG system?” — I’d give them an unexpected answer. Not the choice of language model. Not the fancy vector database. Not even the prompt. The answer is chunking: how you slice up your documents before you turn them into embeddings.
This is genuinely the highest-leverage decision in RAG, and it’s the one people most consistently under-invest in. Get chunking right and mediocre models produce excellent answers. Get it wrong and no amount of expensive tooling downstream can save you. This article is about why chunking matters so much, and the strategies that reliably work.
Why we chunk at all
First, a fair question — why do we split documents at all? Why not just embed the whole PDF and be done with it?
There are two reasons. First, the practical one: embedding models have a limit to how much text they’ll accept in one go, and even if you could squash a whole book into one embedding, the result would be so averaged-out that it wouldn’t be about anything specific — like taking one photo that summarises an entire city. Second, and more importantly: when retrieval finds a match, we want to send the model just the relevant passage, not the whole document. Sending everything is wasteful, dilutes the model’s focus, and often won’t fit in the context window anyway.
So we split documents into chunks: pieces small enough to be meaningfully specific, big enough to carry a complete thought. That deceptively simple split turns out to determine almost everything about how well your RAG works.
The Goldilocks problem
The tricky part is that chunk size has a genuinely tight sweet spot, and both sides of it hurt. Figure 1 shows the trade-off.
Figure 1: Too small and each chunk lacks the context to be meaningful. Too big and each chunk covers too many topics, so the embedding becomes a blurred average. The sweet spot is a chunk that carries one focused idea, with enough surrounding words to make sense.
Chunks too small — say, single sentences — lose their context. A sentence like “The exception is documented in appendix B” is useless on its own; the model has no idea what exception, or in which document. Retrieve it and the model can’t do anything with it. Small chunks also multiply: you’ll have tens of thousands of them, and important context sits scattered across many.
Chunks too big — say, whole chapters — become vague. A single embedding for a ten-page chapter tries to be “about” everything in the chapter, so it’s specifically about nothing. When a question comes in, the chapter will match a lot of things averagely rather than one thing well, and even when it’s retrieved, the model has to hunt through a lot of irrelevant text to find the useful bit.
The Goldilocks zone is somewhere in between: chunks that carry roughly one focused idea, with enough surrounding words to be understood on their own. In practical terms, that usually lands somewhere in the range of a few hundred tokens — a handful of paragraphs. But the right size depends on your documents, and it’s very much worth experimenting with.
The single most useful trick: overlap
Even a well-sized chunk has a hidden problem: what if a really important idea straddles the boundary you happened to cut on? A definition on one page, its explanation on the next. Cut cleanly at the page break and you’ve split a single thought in two, and neither half is any good on its own.
The lovely fix for this is overlap: let each chunk share a little text with its neighbours, so the boundary itself is never a hard cut. If your chunks are, say, 500 tokens long, you might overlap them by 50 tokens — meaning every idea has a strong chance of appearing intact somewhere. It’s a tiny bit of duplication for a large gain in reliability. In pseudocode:
def chunk_text(text, chunk_size=500, overlap=50):
chunks = []
start = 0
while start < len(text):
chunks.append(text[start : start + chunk_size])
start += chunk_size - overlap # step back a little for overlap
return chunksOverlap is one of those small, unglamorous changes that quietly makes a meaningful difference to real-world RAG quality. It’s rarely the wrong decision.
Chunking with the shape of the document
Here’s the next level up, and it’s often where the biggest quality wins come from. Fixed-size chunks — 500 tokens, 500 tokens, 500 tokens — treat your document like an anonymous blob of text. But real documents have structure — headings, sections, paragraphs, bullet points, code blocks. Ignoring that structure means routinely slicing straight through the middle of a sentence or across the boundary between two unrelated sections.
Structure-aware chunking respects the natural boundaries of the document instead. It tries to split on paragraphs, sections, or headings first, and only falls back to fixed-size splitting when a natural piece is too long. The result is chunks that feel like coherent units — because they are coherent units. Figure 2 shows the difference.
Figure 2: Fixed-size chunking splits blindly and often cuts sentences and sections in half. Structure-aware chunking respects paragraph and heading boundaries, producing chunks that are self-contained and meaningful.
Two extra tricks work beautifully alongside structure-aware chunking. First, prepend the heading or section title to every chunk taken from that section — so even a chunk pulled out of “Section 4.2 — Refund policy” carries that title with it as context, and both retrieval and the model benefit from knowing where the passage came from. Second, keep useful metadata on each chunk: source file, page number, section title. This metadata is gold at retrieval time, both for filtering (“only search the refund policy”) and for citing sources in the final answer.
A note on special content
Some content deserves a moment of extra thought.
- Tables and lists. A table cut through the middle is nonsense. Try to keep tables intact, or convert them to a text form (like a small structured list) that carries the row-column meaning.
- Code and technical snippets. Split on function or class boundaries where you can, not in the middle of a definition.
- Very long documents. For books or long reports, layered chunking sometimes helps — a chunk per section for high-level questions, and finer chunks within sections for detail questions.
You don’t need to solve all of this on day one. Start with structure-aware chunking, a sensible size, and a bit of overlap. That combination alone puts you far ahead of most systems.
The one habit that turns chunking into a superpower
The single most valuable habit I can recommend: look at what’s actually being retrieved. Ask your RAG system a question, then peek at the chunks it fetched. Are they the right chunks? Are they complete — do they contain everything needed to answer? Do they stand alone — could a stranger reading just those chunks answer the question? If the answer to any of these is no, chunking is almost always where you fix it.
This inspect-what-was-retrieved habit is how good RAG builders find their next improvement, and it’s how you’ll debug practically every quality problem you’ll ever have. We’ll formalise it into a proper debugging discipline in the “why your RAG keeps hallucinating” article a few steps ahead.
The pattern taking shape
Look at what we’ve built now. We know what RAG is, how semantic search finds passages by meaning, where those passages live (a vector database), and how to prepare them well (chunking). We’ve got the ingredients. The next article is about how they actually come together — the end-to-end pipeline that turns a heap of documents into a working question-answering system.
Next in the series: Building Your First RAG Pipeline — putting the pieces together into a working system.