Chunking splits large documents into smaller pieces before embedding. It is one of the most impactful engineering decisions in a RAG system — retrieval quality depends directly on how well your chunks are structured.
First, embedding models have token limits (512–8192 tokens). Second, even within the limit, a 5000-token document embedded as one vector averages everything together — it won't score highly for queries about any specific part.
Fixed-size — Split every N tokens. Simple and fast. Risk: may split mid-sentence, breaking a thought at the boundary.
Sentence-level — Split at sentence boundaries. Preserves complete sentences. Requires a sentence tokenizer. Variable-length chunks.
Paragraph-level — Each paragraph becomes a chunk. Works well when documents have clear paragraph structure and each paragraph covers one topic.
Semantic chunking — Detect topic shifts and create boundaries there. Most coherent chunks. Requires a two-pass process and more compute.
Overlap includes the last N tokens of the previous chunk at the start of the next chunk. This prevents relevant information from being lost at boundaries.
Without overlap, a key sentence that spans the boundary between chunk 3 and chunk 4 is split across both — neither chunk has it whole. With overlap, both chunks contain the sentence, and retrieval is much more likely to find it.
Typical overlap: 20–200 tokens, or 10–20% of chunk size. More overlap = better boundary coverage; more storage and compute.
More precise retrieval. Each chunk covers a narrow topic. Better for dense technical Q&A where you need an exact passage.
More context preserved. Better when the answer requires surrounding paragraphs to make sense. Risk of diluting relevance across multiple topics.
Always evaluate empirically on real queries. No universal optimal chunk size exists — it depends on your documents and your users' questions.
Ask the AI assistant about chunking strategies, chunk size selection, or overlap configuration.