# RAG chunking strategies: chunk size, overlap, contextual retrieval and hybrid search
> Chunking decides what a RAG system can find. Start with a recursive splitter at a few hundred tokens with some overlap, split by headings where the document has them and keep source metadata on every chunk. Then raise retrieval quality one step at a time: add context to each chunk, add keyword search and rerank, and keep a step only if it helps on your own questions.
Mindket editors · Published 2026-11-04 · Updated 2026-11-05 · https://mindket.com/learn/rag/rag-chunking
## In short
- Pick a chunk size and overlap and know what each setting trades away.
- Split Markdown by headings, keep the headings as metadata and handle Chinese text.
- Add context to chunks with Anthropic's Contextual Retrieval prompt.
- Fuse vector and keyword results with reciprocal rank fusion, then rerank the best candidates.
## Before you start
This lesson builds on [lesson 1](https://mindket.com/learn/rag/rag-tutorial), where the splitter used a fixed size, and on the `chunks` table from [lesson 2](https://mindket.com/learn/rag/embeddings-explained). You need Python 3 and `pip install -qU langchain-text-splitters`.
The context step calls the Claude API with your key, the rerank step uses a Voyage AI key as in lesson 1, and the hybrid query runs on Postgres with pgvector.
## What chunking decides
A chunk is the unit your system embeds, retrieves and shows to the model. Cut it too large and one vector has to stand for several topics. Cut it too small and the chunk loses the words that say what it is about. [LlamaIndex](https://developers.llamaindex.ai/python/framework/optimizing/basic_strategies/basic_strategies/) states the trade-off directly: a smaller chunk size means more precise embeddings, and a larger one means more general embeddings that can miss fine-grained details. Anthropic's post describes chunks as usually no more than a few hundred tokens.
Changing the size has a second effect. LlamaIndex advises raising `similarity_top_k` when you shrink chunks, and its example doubles it from 2 to 4 after halving the size.
> **Our view on a first setting**
>
> We'd start from the OpenAI file search default in the table below, then try one size smaller and one larger and compare retrieval on your own questions with the metrics from lesson 4. None of these docs names a size that wins on every corpus.
| Source | Chunk size | Overlap | Counted in |
| --- | --- | --- | --- |
| LangChain tutorial, RecursiveCharacterTextSplitter | 1000 | 200 | Characters |
| LlamaIndex defaults | 1024 | 20 | Not stated on the page |
| OpenAI file search defaults | 800 | 400 | Tokens |
## Split by structure, then by size
[LangChain](https://docs.langchain.com/oss/python/integrations/splitters) says to start with `RecursiveCharacterTextSplitter` for most cases. It tries a list of separators in order, `["\n\n", "\n", " ", ""]` by default, so it keeps paragraphs together as long as it can, then sentences, then words.
A document with its own structure can do better. A Markdown file is organized by headers, and `MarkdownHeaderTextSplitter` splits on the ones you name and writes their text into each chunk's metadata. LangChain also lists splitters for HTML, JSON and code, and [LlamaIndex](https://developers.llamaindex.ai/python/framework/module_guides/loading/node_parsers/modules/) has node parsers for the same formats.
````python
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
with open("refund-policy.md", encoding="utf-8") as f:
markdown_document = f.read()
# 1. Split on headings. The heading text lands in each section's metadata.
headers_to_split_on = [
("#", "Header 1"),
("##", "Header 2"),
("###", "Header 3"),
]
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on, strip_headers=False)
sections = markdown_splitter.split_text(markdown_document)
# 2. Cut the sections that are still too long. split_documents keeps the metadata.
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = text_splitter.split_documents(sections)
for chunk in chunks:
print(chunk.metadata, chunk.page_content[:60])
````
- Call `split_documents`, not `split_text`, on the sections. Only then does each chunk keep its header metadata.
- Overlap appears only when a single section is longer than `chunk_size`, and it never crosses a section boundary.
- `strip_headers` is on by default, so the header line leaves the chunk text and stays in the metadata. If a header turns into a tiny chunk of its own, turn it back on.
- The default splitter strips white space and new lines. LangChain points to `ExperimentalMarkdownSyntaxTextSplitter` when you need the original formatting.
- Keep the empty string `""` at the end of `separators`, so text with no newlines or spaces can still be split.
## Chinese, Japanese and Thai
These writing systems have no word boundaries. LangChain warns that the default separators can then split a word between two chunks. Its fix is to extend the list with the punctuation of those scripts: full stops, commas and the zero-width space.
The size is still counted in characters, because the recursive splitter measures chunk size by number of characters.
````python
from langchain_text_splitters import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
separators=[
"\n\n",
"\n",
" ",
".",
",",
"", # Zero-width space
"，", # Fullwidth comma
"、", # Ideographic comma
"．", # Fullwidth full stop
"。", # Ideographic full stop
"",
],
chunk_size=1000,
chunk_overlap=200,
)
````
## Keep metadata on every chunk
In LangChain, a `Document` holds the text and a `metadata` dict that can capture its source and its relation to other documents. Setting `add_start_index=True` adds the character offset where each chunk began. That metadata does real work later:
- Citations. Lesson 1 passes each chunk's `source` and `title` to Claude's `search_result` blocks, so the answer can point back to the passage.
- Filtering. LlamaIndex notes that metadata can filter data before the top-k similarity search. OpenAI file search takes `attributes` for the same purpose, with up to 16 keys of 256 characters each.
- Isolation. LlamaIndex's multi-tenancy note says metadata filters let each user search only their own indexed documents. On an approximate pgvector index the filter runs after the scan, as [lesson 2](https://mindket.com/learn/rag/embeddings-explained) explains.
## Add context to each chunk
Anthropic's [Contextual Retrieval post](https://www.anthropic.com/research/contextual-retrieval) starts from a chunk that reads "The company's revenue grew by 3% over the previous quarter." Alone it names neither the company nor the period, so retrieval cannot match it to a question about ACME Corp in Q2 2023. The fix is to prepend a short explanation to each chunk before embedding it, which the post calls Contextual Embeddings, and before building the BM25 index, which it calls Contextual BM25.
A model writes the explanation. Anthropic's prompt hands it the whole document and the chunk and asks for a short, succinct context that situates the chunk for search. The result is usually 50 to 100 tokens. The post used Claude 3 Haiku and pairs the method with prompt caching, so the document loads into the cache once instead of once per chunk.
````python
import anthropic
claude = anthropic.Anthropic() # reads ANTHROPIC_API_KEY
CONTEXT_PROMPT = """
{whole_document}
Here is the chunk we want to situate within the whole document
{chunk_content}
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else."""
def contextualize(whole_document: str, chunk_content: str) -> str:
response = claude.messages.create(
model="claude-haiku-5-5",
max_tokens=200,
messages=[
{
"role": "user",
"content": CONTEXT_PROMPT.format(
whole_document=whole_document, chunk_content=chunk_content
),
}
],
)
return next(block.text for block in response.content if block.type == "text")
for chunk in chunks:
context = contextualize(document_text, chunk.page_content) # document_text: the full text this chunk came from
chunk.metadata["context"] = context
text_to_index = f"{context}\n\n{chunk.page_content}" # embed this, and build the BM25 index from it
````
1. Split the document into chunks as before.
2. For each chunk, send the whole document and the chunk to the model with the prompt above.
3. Store the returned context beside the chunk and prepend it to the chunk text.
4. Embed the contextualized text, and build the BM25 index from the same text.
5. When you generate the answer, pass the contextualized chunk and mark which part is context and which is the chunk. The post says this can improve the response.
## What the context gains, and what to check
Anthropic measured failed retrievals as 1 minus recall@20, the share of relevant documents missing from the top twenty chunks. The chart shows the averages across codebases, fiction, ArXiv papers and science papers with Gemini Text 004 embeddings. The reranked run used the Cohere reranker.
Two other ideas gave little in the same experiments. Generic document summaries added to chunks gave limited gains, and hypothetical document embeddings and summary-based indexing performed poorly.
- Chunk boundaries still matter. Chunk size, boundary and overlap all affect retrieval.
- The embedding model matters. Every model improved, and Anthropic found Gemini and Voyage embeddings particularly effective.
- A prompt written for your domain may do better, for example one that includes a glossary of terms defined only in other documents.
- Voyage AI sells a shortcut: `voyage-context-4` returns chunk vectors that capture the full document context without manual augmentation. The [Voyage page](https://www.mongodb.com/docs/voyageai/models/contextualized-chunk-embeddings) claims higher retrieval accuracy and less sensitivity to your chunking strategy. Anthropic's post did not test it.
- Always run evals. Lesson 4 shows how.
**Failed retrievals in the top 20 chunks** Lower is better. Anthropic reports the failure rate as 1 minus recall@20.
- Embeddings alone: 5.7%
- Contextual Embeddings: 3.7%
- Contextual Embeddings and BM25: 2.9%
- Plus reranking: **1.9%**
Source: [Anthropic: Contextual Retrieval](https://www.anthropic.com/research/contextual-retrieval) (read 2026-11-04)
## Hybrid search with rank fusion
Embeddings capture meaning and can miss an exact string. The post's example is a query for "Error code TS-999" in a support database. An embedding model may find content about error codes in general and miss the exact match, while BM25 looks for that string. BM25 is a lexical ranking function built on TF-IDF. It adds document length and a saturation function on term frequency, so common words do not dominate.
Anthropic's recipe is to get the top chunks from each method, then combine and deduplicate them with rank fusion. Reciprocal Rank Fusion gives each chunk the sum of 1 / (k + rank) over the lists it appears in. The pgvector-python example below uses k = 60, one list from vector search and one from Postgres full-text search, which the [pgvector README](https://github.com/pgvector/pgvector) names as its hybrid pairing.
One difference from the post: Postgres ranks with `ts_rank_cd`, a different function from BM25, and the example uses the `english` text search configuration. Check how your language is tokenized before you trust the keyword half. A vector database with built-in hybrid search is the other route [LlamaIndex](https://developers.llamaindex.ai/python/framework/optimizing/basic_strategies/basic_strategies/) lists, and OpenAI file search exposes `ranking_options.hybrid_search` with `embedding_weight` and `text_weight` for the same fusion.
````sql
-- the index that serves the keyword half
CREATE INDEX ON chunks USING GIN (to_tsvector('english', content));
-- %(embedding)s is the query vector, %(query)s the question text and %(k)s the constant (60 in the example)
WITH semantic_search AS (
SELECT id, RANK () OVER (ORDER BY embedding <=> %(embedding)s) AS rank
FROM chunks
ORDER BY embedding <=> %(embedding)s
LIMIT 20
),
keyword_search AS (
SELECT id, RANK () OVER (ORDER BY ts_rank_cd(to_tsvector('english', content), query) DESC) AS rank
FROM chunks, plainto_tsquery('english', %(query)s) query
WHERE to_tsvector('english', content) @@ query
ORDER BY ts_rank_cd(to_tsvector('english', content), query) DESC
LIMIT 20
)
SELECT
COALESCE(semantic_search.id, keyword_search.id) AS id,
COALESCE(1.0 / (%(k)s + semantic_search.rank), 0.0) +
COALESCE(1.0 / (%(k)s + keyword_search.rank), 0.0) AS score
FROM semantic_search
FULL OUTER JOIN keyword_search ON semantic_search.id = keyword_search.id
ORDER BY score DESC
LIMIT 5
````
**Vector, keyword or both**
| | Embeddings | BM25 | Both, fused (Our pick) |
| --- | --- | --- | --- |
| Finds a paraphrase of the question | ✓ Yes | ◐ Partly | ✓ Yes |
| Finds an exact identifier such as TS-999 | ◐ Partly | ✓ Yes | ✓ Yes |
| Needs one kind of index only | ✓ Yes | ✓ Yes | ✗ No |
**Our pick: Both, fused.** Anthropic's experiments found embeddings plus BM25 better than embeddings alone, at the cost of a second index and a fusion step.
Sources: [Anthropic: Contextual Retrieval](https://www.anthropic.com/research/contextual-retrieval) (read 2026-11-04), [LlamaIndex: Basic strategies](https://developers.llamaindex.ai/python/framework/optimizing/basic_strategies/basic_strategies/) (read 2026-11-04)
## Rerank the candidates
Hybrid search still returns chunks of mixed quality. A reranker scores them again. Voyage AI describes its rerankers as cross-encoders: embedding models encode the query and each document separately, while a reranker processes a query and document pair jointly, which allows a more accurate relevance score. You apply one to the top candidates from embedding search or from lexical search such as BM25.
Anthropic's pipeline takes the top 150 chunks from the first stage, reranks them and keeps the best twenty for the prompt. Reranking adds a step at run time and so a little latency, and the post tells you to tune how many chunks you rerank against cost and speed. The [Voyage page](https://www.mongodb.com/docs/voyageai/models/rerankers) lists `rerank-3` as the highest-accuracy model for most applications and `rerank-3-lite` for latency-sensitive ones, both with a 32,000-token context.
````python
import voyageai
vo = voyageai.Client() # reads VOYAGE_API_KEY
# candidates: the text of each chunk the hybrid query returned (Anthropic's post started from 150)
reranked = vo.rerank(query, candidates, model="rerank-3", top_k=20)
for result in reranked.results:
print(f"{result.relevance_score:.4f}", result.document[:60])
context_chunks = [result.document for result in reranked.results] # these go into the prompt
````
- LlamaIndex calls reranking one of the biggest levers for RAG quality. Its recipe pairs `similarity_top_k=10` with a local cross-encoder, `cross-encoder/ms-marco-MiniLM-L6-v2`, and `top_n=3`.
- An LLM-based reranker can beat a cross-encoder when latency is not critical, at the cost of extra LLM calls, per the same page.
- Anthropic tested the Cohere reranker. The post says it did not have time to test Voyage's.
## Jo Bergum on where embeddings fail, LlamaIndex on what context costs
Jo Kristian Bergum runs Hornet.dev, a retrieval engine built for agents, by his X bio. On 30 August 2025 he wrote [on X](https://x.com/jobergum/status/1961663599195144642) that some people know him as "the BM25 guy", and shared [a paper](https://arxiv.org/abs/2508.21038) by researchers at Google DeepMind and Johns Hopkins. It proves that an embedding's dimension caps how many different top-k result sets it can return. On the paper's deliberately simple LIMIT test, BM25 found 93.6% of the relevant documents in its top hundred results. The best single-vector model found 18.9%.
The paper, revised in March 2026, also tests the reverse. With every item swapped for a synonym, BM25 lost nearly 90% of its score and fell behind most embedding models. Keep both halves of the fused query, then. Your test questions should include some that reuse your documents' exact words and some that paraphrase them.
LlamaIndex, an AI research lab for document parsing and extraction by its X bio, pointed out [on X](https://x.com/llama_index/status/1841210062167294287) on 1 October 2024 what Contextual Retrieval costs. Every chunk takes one model call, and every call carries the whole document as a prefix. The `contextualize()` loop above has that shape. Its prompt opens with the document, so only the chunk changes from call to call, which is what a prompt cache needs. Send one document's chunks one after another rather than mixing documents.
> This requires N LLM calls where each call has the entire document stuffed in as a prefix, so this depends heavily on prompt caching for token cost reduction.
>
> LlamaIndex, AI research lab for document parsing and extraction, by its X bio, [X](https://x.com/llama_index/status/1841210062167294287) (read 2026-11-05)
[](https://x.com/jobergum/status/1961663599195144642)
Jo Kristian Bergum on X, 30 August 2025
[](https://x.com/llama_index/status/1841210062167294287)
LlamaIndex on X, 1 October 2024
[The 5 Levels Of Text Splitting For Retrieval](https://www.youtube.com/watch?v=8OJC21T2SL4)
Greg Kamradt on YouTube, on five levels of text splitting for retrieval, as its title says. A notebook of the same name on GitHub lists them, from character splitting to agentic chunking.
> Screenshots of X's own embeds of the posts, taken on 5 November 2026, with the time in UTC. X cuts LlamaIndex's post short with Show more. Select one to open the post on X.
## Common mistakes
- Mixing units. LangChain counts characters, OpenAI counts tokens, and a limit written in one is wrong in the other.
- Changing the chunk size and leaving the number of retrieved chunks alone. LlamaIndex raises `similarity_top_k` when it shrinks the size.
- Setting an OpenAI overlap above half the chunk size. The docs say it should not exceed `max_chunk_size_tokens / 2`.
- Using the English default separators on Chinese text, which can cut words in two.
- Showing the model the bare chunk when you indexed the contextualized one. Pass the context too, and label it.
- Borrowing a number from someone else's corpus. A chunk count, size or overlap that won for Anthropic or OpenAI says little about your documents.
## Try it yourself
- Split one of your Markdown files with both splitters and print each chunk with its metadata. Which headings survive?
- Run the same ten questions at three chunk sizes and note which chunk the right answer came from each time.
- Write context for five chunks with the Anthropic prompt and read them. Would you find each chunk from its new text alone?
- Search for an exact identifier from your documents with vectors only, then with the fused query.
## Next
[Lesson 4](https://mindket.com/learn/rag/evaluate-rag) shows how to measure retrieval and answers, so each choice above rests on numbers from your own questions. To feed an index from files, databases or the web, the [MCP servers](https://mindket.com/mcp-servers) on Mindket connect an agent to those sources.
## Check yourself
1. You split a Markdown file by headers, then cut long sections down to size. Which call keeps each section's header metadata on the chunks?
A. RecursiveCharacterTextSplitter.split_text(sections)
B. MarkdownHeaderTextSplitter with strip_headers=True
C. RecursiveCharacterTextSplitter.split_documents(sections)
D. Setting chunk_overlap to 0
Answer: C. LangChain says to use split_documents rather than split_text after the header split, so the per-section metadata survives.
2. Your Chinese documents are being cut in the middle of words. What does LangChain suggest?
A. Add the ideographic full stop and comma to the separators list
B. Set chunk_overlap to 0
C. Switch to token counting
D. Remove all punctuation before splitting
Answer: A. The default separators assume spaces. LangChain extends the list with full stops, commas and the zero-width space for scripts without word boundaries.
3. In Anthropic's Contextual Retrieval, what gets prepended to each chunk before embedding and BM25 indexing?
A. A summary of the whole document
B. The chunk's embedding vector
C. The user's question
D. A short model-written note that places the chunk within its document
Answer: D. The prompt asks for a short context that situates the chunk in the overall document. Generic document summaries gave limited gains in the same experiments.
4. What does the rank fusion step do in hybrid search?
A. Re-scores every chunk with a cross-encoder
B. Merges the vector and keyword result lists by rank into one list
C. Splits chunks at sentence boundaries
D. Shortens vectors to fewer dimensions
Answer: B. Reciprocal Rank Fusion adds 1 / (k + rank) for each list a chunk appears in, then sorts by the total.
## Frequently asked questions
### What is the best chunk size for RAG?
No source names one best size. Defaults differ: OpenAI file search uses 800 tokens, LlamaIndex 1024 and LangChain's tutorial 1000 characters. Start with one and compare retrieval on your own questions, because smaller chunks give more precise embeddings and larger ones can miss fine detail.
### How much overlap should chunks have?
Treat it as a setting to tune. LangChain says overlap mitigates loss of information when context is divided between chunks. OpenAI's default is 400 tokens on an 800-token chunk and it allows at most half the chunk size, while LlamaIndex defaults to a much smaller overlap.
### Is semantic chunking worth it?
Measure it first. LlamaIndex's semantic splitter picks breakpoints between sentences by embedding similarity, so each chunk holds related sentences. Its page warns that the sentence regex mainly works for English and that you may have to tune the breakpoint percentile threshold. The pages we read report no gain over a recursive splitter.
### How do I chunk Chinese or Japanese text?
Extend the separator list of RecursiveCharacterTextSplitter with the ideographic full stop, the fullwidth full stop, commas and the zero-width space, as LangChain's guide shows.
### What is contextual retrieval, and is it worth it?
It adds a short, model-written note to each chunk before indexing. In Anthropic's experiments that cut failed retrievals by 35% with embeddings alone and 49% with BM25 added, at the price of one model call per chunk. Prompt caching reduces that cost.
### Is hybrid search better than vector search alone?
Anthropic found embeddings plus BM25 better than embeddings alone. LlamaIndex says embeddings can fail to return chunks with matching keywords and names hybrid search as the usual fix. It costs a second index and a fusion step.
### How many chunks should I pass to the model?
Anthropic tried three counts and found the largest, twenty chunks, the most performant, with the warning that extra information can distract the model. Retrieve more than you keep, rerank, then test your own number.
## Sources
- [LangChain Docs: Text splitter integrations](https://docs.langchain.com/oss/python/integrations/splitters) (read 2026-11-04)
- [LangChain Docs: Splitting recursively](https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter) (read 2026-11-05)
- [LangChain Docs: Split markdown](https://docs.langchain.com/oss/python/integrations/splitters/markdown_header_metadata_splitter) (read 2026-11-04)
- [LlamaIndex Docs: Basic strategies](https://developers.llamaindex.ai/python/framework/optimizing/basic_strategies/basic_strategies/) (read 2026-11-05)
- [LlamaIndex Docs: Node parser modules](https://developers.llamaindex.ai/python/framework/module_guides/loading/node_parsers/modules/) (read 2026-11-04)
- [OpenAI Docs: Retrieval](https://developers.openai.com/api/docs/guides/retrieval) (read 2026-11-05)
- [Anthropic: Contextual Retrieval](https://www.anthropic.com/research/contextual-retrieval) (read 2026-11-04)
- [Voyage AI Docs: Rerankers](https://www.mongodb.com/docs/voyageai/models/rerankers) (read 2026-11-05)
- [Voyage AI Docs: Contextualized Chunk Embeddings](https://www.mongodb.com/docs/voyageai/models/contextualized-chunk-embeddings) (read 2026-11-05)
- [Voyage AI Docs: Quickstart](https://www.mongodb.com/docs/voyageai/quickstart) (read 2026-11-04)
- [pgvector README](https://github.com/pgvector/pgvector) (read 2026-11-04)
- [pgvector-python: hybrid search example](https://github.com/pgvector/pgvector-python) (read 2026-11-04)
- [Jo Kristian Bergum on X: On the Theoretical Limitations of Embedding-Based Retrieval](https://x.com/jobergum/status/1961663599195144642) (read 2026-11-05)
- [arXiv: On the Theoretical Limitations of Embedding-Based Retrieval (v2, March 2026)](https://arxiv.org/abs/2508.21038) (read 2026-11-05)
- [LlamaIndex on X: Implementing Contextual Retrieval RAG](https://x.com/llama_index/status/1841210062167294287) (read 2026-11-05)
- [Greg Kamradt on YouTube: The 5 Levels Of Text Splitting For Retrieval](https://www.youtube.com/watch?v=8OJC21T2SL4) (read 2026-11-05)
- [GitHub: 5 Levels Of Text Splitting notebook](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb) (read 2026-11-05)
[Browse MCP servers](https://mindket.com/mcp-servers)