RAG Chunking Strategies That Actually Work in Production

A retrieval-augmented generation system has three moving parts: how you split documents, how you embed them, and how you retrieve. The LLM step is largely fixed — the model reads what you feed it. Everything upstream is where quality is won or lost, and empirically the biggest lever is chunking. Swapping text-embedding-3-small for a larger model rarely moves nDCG@10 more than a few points; changing chunk boundaries can move it by 20+ points on the same corpus.

The intuition: an embedding is an average of the semantics in the input. If you concatenate three unrelated paragraphs into one chunk, the vector lands in a meaningless region of latent space and no query will ever find it cleanly. Conversely, if you split a coherent argument into six tiny fragments, none of them contain enough context to answer a real question. The retriever will happily return the right chunk with the wrong text.

Every chunking strategy in production reduces to one of four families. Pick based on the shape of your source documents, not the flavor of the month on Twitter.

• Fixed-size + overlap. Split every 500-1000 tokens with 10-20% overlap. Cheap, deterministic, easy to reindex. The right default for FAQs, blog posts, and short markdown.
• Semantic (breakpoint) chunking. Embed sentences, compute cosine distance between neighbors, cut where distance spikes. Good for long unstructured prose (transcripts, meeting notes, wikis).
• Hierarchical / parent-child. Store small chunks for retrieval, but return the larger parent block to the LLM. This is the pattern for structured long-form: contracts, policies, manuals, textbooks.
• Late chunking. Embed the whole document with a long-context embedder (Jina v3, Voyage-3), then pool per-token embeddings into chunk vectors. Preserves cross-chunk context. Expensive; use when accuracy dominates cost.

Most teams get fixed-size chunking wrong in two ways: they split on characters instead of tokens, and they use no overlap. Both silently destroy recall.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from tiktoken import get_encoding

enc = get_encoding('cl100k_base')
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name='cl100k_base',
    chunk_size=800,          # tokens, not chars
    chunk_overlap=120,       # ~15% — the sweet spot
    separators=['\n\n', '\n', '. ', ' ', ''],  # respect natural boundaries first
)

chunks = splitter.split_text(document)

The separators list is doing more work than the sizes. It tells the splitter to prefer paragraph breaks, then sentences, then words, and only fall back to character cuts as a last resort. On a corpus of 10k engineering docs I measured a 14-point nDCG@10 jump moving from naïve character splitting to token-aware recursive splitting — no embedding changes.

Semantic chunking sounds like the obvious upgrade. In practice it wins on some corpora and loses on others.

The algorithm: embed each sentence, compute the cosine distance to the next sentence, and cut where distance exceeds a threshold (usually the 90th percentile). Sentences within a semantic 'block' end up in the same chunk regardless of length.

Where it wins: podcast transcripts, meeting notes, long informal wiki pages — anywhere the writer wanders and paragraphs don't map to topics. I've seen +9 nDCG points on transcripts vs fixed-size.

Where it loses: highly structured documents (API references, policy handbooks, legal contracts). The section headers already encode topic boundaries, and the semantic splitter often merges sections that a user considers distinct. On our internal policy corpus semantic chunking underperformed fixed-size by 6 points.

This is my default for long structured documents. The idea: index small chunks (100-200 tokens) so the retriever has precise anchors, but at generation time swap each hit for its larger parent block (500-2000 tokens) so the LLM has context.

1. Split each document into parent blocks (usually one per H2 or per ~1000 tokens).
2. Split each parent into children of 100-200 tokens with 30% overlap.
3. Embed children; store parent_id as metadata.
4. At query time, retrieve top-K children, deduplicate by parent_id, fetch parent text.
5. Rerank parents (optional but recommended for K > 3).

Result: retrieval accuracy of the small chunk, context breadth of the large chunk. On a 2M-token engineering handbook this pattern beat every other approach by 8-12 points on both nDCG and end-to-end answer quality (LLM-graded).

Late chunking flips the pipeline. Instead of splitting first and embedding each chunk in isolation, you feed the whole document (or a very long window) to a long-context embedding model like Jina embeddings v3 or Voyage-3, then pool the per-token vectors into chunk vectors afterward.

The benefit: every chunk vector 'knows' the surrounding document context. A chunk containing 'he approved it' will have a different embedding depending on who 'he' is upstream. On coreference-heavy corpora (legal, medical, long reports) late chunking beats naïve chunking by 10-15 points.

The cost: embedding a 32k-token document is 20-50x more expensive than embedding it in 800-token pieces. Only justify late chunking when accuracy is the constraint — customer-facing legal, medical Q&A, high-value RAG.

Every claim above should be re-measured on your corpus. Build a tiny eval set before you ship any RAG system:

1. Sample 50-100 real user questions (from logs, support tickets, or by asking colleagues).
2. For each, mark the correct passage(s) in the source corpus. This is annoying and irreplaceable.
3. Score each chunking strategy on Recall@5, MRR, and nDCG@10.
4. Only then run the full pipeline through an LLM judge for answer quality.

Without this loop you will burn weeks tuning things that don't matter and shipping regressions you can't see.