Text Chunking

Text chunking is the practice of dividing a document into shorter segments — chunks — so that each segment can be converted into an embedding vector and retrieved on its own. It is a standard step in retrieval-augmented generation and semantic search pipelines, where answer quality is bounded by how well the stored chunks line up with the questions users actually ask.^[1]

Overview

Embedding models compress a passage into a single vector, and each model accepts only a finite input window. Feeding an entire policy manual, contract or codebase into one embedding flattens its detail: the resulting vector describes the document on average, not any particular paragraph. Chunking avoids this by indexing many small pieces instead, so a query can be matched to the exact passage that answers it.

The trade-off runs in both directions. Chunks that are too large dilute the signal — a single vector must represent several unrelated topics — and leave less room in the model's context window for the retrieved passages themselves. Chunks that are too small arrive without the surrounding context needed to make sense of them, and multiply the number of vectors the database must search. Almost every strategy therefore adds overlap: a small duplicated window across each boundary so a sentence split in half still carries its neighbour into both chunks.^[2]

Common Strategies

• Fixed-size chunking splits text every N tokens with a fixed overlap. It is trivial to implement and is the recommended starting point for most projects.^[1]
• Recursive (content-aware) chunking tries a fallback hierarchy of separators — paragraph breaks, then headings, then sentences, then words — only cutting deeper when the piece is still too large. A common default is 400 to 512 tokens with 10 to 20 per cent overlap.^[3]
• Structure-aware chunking splits on the document's own grammar: markdown sections, table rows, code blocks, PDF pages or slide boundaries, keeping units intact.
• Semantic chunking embeds each sentence, groups neighbouring sentences, and starts a new chunk wherever the semantic distance to the previous group spikes. It tracks topic shifts rather than character counts, and some benchmarks report recall gains of up to about nine per cent — at the cost of an extra embedding pass over the corpus.^[3][4]
• Contextual approaches, sometimes called late chunking or contextual retrieval, attach a short document-level summary to each chunk before embedding, repairing the context that splitting removed.

A related design is parent-document retrieval: small chunks are matched against the query, but the larger section containing them is what gets passed to the language model.

Applications or Impact

Chunking sits underneath almost every production knowledge assistant — customer-support bots over product documentation, policy Q&A for employees, legal and contract review, and code search across large repositories. Teams rebuilding a retrieval system routinely find that adjusting chunk size and boundaries changes answer quality more than swapping the vector database or the embedding model, which is why chunking is treated as the first hyperparameter to tune.^[4]

>Key Takeaways

• Chunking usually determines retrieval quality more than the choice of vector database or embedding model.
• Start with recursive splitting at 400 to 512 tokens and 10 to 20 per cent overlap, then evaluate before adding complexity.
• Chunk size, overlap, structure and metadata all matter; semantic chunking is an upgrade path, not a default.
• In multilingual Malaysian corpora, size chunks by tokens and validate retrieval per language.

See Also

• Retrieval-Augmented Generation
• Embedding
• Vector Database
• Semantic Search
• Hybrid Search
• Reranking