RAG chunking strategies: chunk size, overlap, contextual retrieval and hybrid search

Before you start

This lesson builds on lesson 1, where the splitter used a fixed size, and on the chunks table from lesson 2. You need Python 3 and pip install -qU langchain-text-splitters.

The context step calls the Claude API with your key, the rerank step uses a Voyage AI key as in lesson 1, and the hybrid query runs on Postgres with pgvector.

What chunking decides

A chunk is the unit your system embeds, retrieves and shows to the model. Cut it too large and one vector has to stand for several topics. Cut it too small and the chunk loses the words that say what it is about. LlamaIndex states the trade-off directly: a smaller chunk size means more precise embeddings, and a larger one means more general embeddings that can miss fine-grained details. Anthropic's post describes chunks as usually no more than a few hundred tokens.

Changing the size has a second effect. LlamaIndex advises raising similarity_top_k when you shrink chunks, and its example doubles it from 2 to 4 after halving the size.

Split by structure, then by size

LangChain says to start with RecursiveCharacterTextSplitter for most cases. It tries a list of separators in order, ["\n\n", "\n", " ", ""] by default, so it keeps paragraphs together as long as it can, then sentences, then words.

A document with its own structure can do better. A Markdown file is organized by headers, and MarkdownHeaderTextSplitter splits on the ones you name and writes their text into each chunk's metadata. LangChain also lists splitters for HTML, JSON and code, and LlamaIndex has node parsers for the same formats.

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

with open("refund-policy.md", encoding="utf-8") as f:
    markdown_document = f.read()

# 1. Split on headings. The heading text lands in each section's metadata.
headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on, strip_headers=False)
sections = markdown_splitter.split_text(markdown_document)

# 2. Cut the sections that are still too long. split_documents keeps the metadata.
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = text_splitter.split_documents(sections)

for chunk in chunks:
    print(chunk.metadata, chunk.page_content[:60])

• Call split_documents, not split_text, on the sections. Only then does each chunk keep its header metadata.
• Overlap appears only when a single section is longer than chunk_size, and it never crosses a section boundary.
• strip_headers is on by default, so the header line leaves the chunk text and stays in the metadata. If a header turns into a tiny chunk of its own, turn it back on.
• The default splitter strips white space and new lines. LangChain points to ExperimentalMarkdownSyntaxTextSplitter when you need the original formatting.
• Keep the empty string "" at the end of separators, so text with no newlines or spaces can still be split.

Chinese, Japanese and Thai

These writing systems have no word boundaries. LangChain warns that the default separators can then split a word between two chunks. Its fix is to extend the list with the punctuation of those scripts: full stops, commas and the zero-width space.

The size is still counted in characters, because the recursive splitter measures chunk size by number of characters.

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    separators=[
        "\n\n",
        "\n",
        " ",
        ".",
        ",",
        "​",  # Zero-width space
        "，",  # Fullwidth comma
        "、",  # Ideographic comma
        "．",  # Fullwidth full stop
        "。",  # Ideographic full stop
        "",
    ],
    chunk_size=1000,
    chunk_overlap=200,
)

Keep metadata on every chunk

In LangChain, a Document holds the text and a metadata dict that can capture its source and its relation to other documents. Setting add_start_index=True adds the character offset where each chunk began. That metadata does real work later:

• Citations. Lesson 1 passes each chunk's source and title to Claude's search_result blocks, so the answer can point back to the passage.
• Filtering. LlamaIndex notes that metadata can filter data before the top-k similarity search. OpenAI file search takes attributes for the same purpose, with up to 16 keys of 256 characters each.
• Isolation. LlamaIndex's multi-tenancy note says metadata filters let each user search only their own indexed documents. On an approximate pgvector index the filter runs after the scan, as lesson 2 explains.

Add context to each chunk

Anthropic's Contextual Retrieval post starts from a chunk that reads "The company's revenue grew by 3% over the previous quarter." Alone it names neither the company nor the period, so retrieval cannot match it to a question about ACME Corp in Q2 2023. The fix is to prepend a short explanation to each chunk before embedding it, which the post calls Contextual Embeddings, and before building the BM25 index, which it calls Contextual BM25.

A model writes the explanation. Anthropic's prompt hands it the whole document and the chunk and asks for a short, succinct context that situates the chunk for search. The result is usually 50 to 100 tokens. The post used Claude 3 Haiku and pairs the method with prompt caching, so the document loads into the cache once instead of once per chunk.

import anthropic

claude = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

CONTEXT_PROMPT = """<document>
{whole_document}
</document>
Here is the chunk we want to situate within the whole document
<chunk>
{chunk_content}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else."""


def contextualize(whole_document: str, chunk_content: str) -> str:
    response = claude.messages.create(
        model="claude-haiku-5-5",
        max_tokens=200,
        messages=[
            {
                "role": "user",
                "content": CONTEXT_PROMPT.format(
                    whole_document=whole_document, chunk_content=chunk_content
                ),
            }
        ],
    )
    return next(block.text for block in response.content if block.type == "text")


for chunk in chunks:
    context = contextualize(document_text, chunk.page_content)  # document_text: the full text this chunk came from
    chunk.metadata["context"] = context
    text_to_index = f"{context}\n\n{chunk.page_content}"  # embed this, and build the BM25 index from it

1. Split the document into chunks as before.
2. For each chunk, send the whole document and the chunk to the model with the prompt above.
3. Store the returned context beside the chunk and prepend it to the chunk text.
4. Embed the contextualized text, and build the BM25 index from the same text.
5. When you generate the answer, pass the contextualized chunk and mark which part is context and which is the chunk. The post says this can improve the response.

What the context gains, and what to check

Anthropic measured failed retrievals as 1 minus recall@20, the share of relevant documents missing from the top twenty chunks. The chart shows the averages across codebases, fiction, ArXiv papers and science papers with Gemini Text 004 embeddings. The reranked run used the Cohere reranker.

Two other ideas gave little in the same experiments. Generic document summaries added to chunks gave limited gains, and hypothetical document embeddings and summary-based indexing performed poorly.

• Chunk boundaries still matter. Chunk size, boundary and overlap all affect retrieval.
• The embedding model matters. Every model improved, and Anthropic found Gemini and Voyage embeddings particularly effective.
• A prompt written for your domain may do better, for example one that includes a glossary of terms defined only in other documents.
• Voyage AI sells a shortcut: voyage-context-4 returns chunk vectors that capture the full document context without manual augmentation. The Voyage page claims higher retrieval accuracy and less sensitivity to your chunking strategy. Anthropic's post did not test it.
• Always run evals. Lesson 4 shows how.

Hybrid search with rank fusion

Embeddings capture meaning and can miss an exact string. The post's example is a query for "Error code TS-999" in a support database. An embedding model may find content about error codes in general and miss the exact match, while BM25 looks for that string. BM25 is a lexical ranking function built on TF-IDF. It adds document length and a saturation function on term frequency, so common words do not dominate.

Anthropic's recipe is to get the top chunks from each method, then combine and deduplicate them with rank fusion. Reciprocal Rank Fusion gives each chunk the sum of 1 / (k + rank) over the lists it appears in. The pgvector-python example below uses k = 60, one list from vector search and one from Postgres full-text search, which the pgvector README names as its hybrid pairing.

One difference from the post: Postgres ranks with ts_rank_cd, a different function from BM25, and the example uses the english text search configuration. Check how your language is tokenized before you trust the keyword half. A vector database with built-in hybrid search is the other route LlamaIndex lists, and OpenAI file search exposes ranking_options.hybrid_search with embedding_weight and text_weight for the same fusion.

-- the index that serves the keyword half
CREATE INDEX ON chunks USING GIN (to_tsvector('english', content));

-- %(embedding)s is the query vector, %(query)s the question text and %(k)s the constant (60 in the example)
WITH semantic_search AS (
    SELECT id, RANK () OVER (ORDER BY embedding <=> %(embedding)s) AS rank
    FROM chunks
    ORDER BY embedding <=> %(embedding)s
    LIMIT 20
),
keyword_search AS (
    SELECT id, RANK () OVER (ORDER BY ts_rank_cd(to_tsvector('english', content), query) DESC) AS rank
    FROM chunks, plainto_tsquery('english', %(query)s) query
    WHERE to_tsvector('english', content) @@ query
    ORDER BY ts_rank_cd(to_tsvector('english', content), query) DESC
    LIMIT 20
)
SELECT
    COALESCE(semantic_search.id, keyword_search.id) AS id,
    COALESCE(1.0 / (%(k)s + semantic_search.rank), 0.0) +
    COALESCE(1.0 / (%(k)s + keyword_search.rank), 0.0) AS score
FROM semantic_search
FULL OUTER JOIN keyword_search ON semantic_search.id = keyword_search.id
ORDER BY score DESC
LIMIT 5

Rerank the candidates

Hybrid search still returns chunks of mixed quality. A reranker scores them again. Voyage AI describes its rerankers as cross-encoders: embedding models encode the query and each document separately, while a reranker processes a query and document pair jointly, which allows a more accurate relevance score. You apply one to the top candidates from embedding search or from lexical search such as BM25.

Anthropic's pipeline takes the top 150 chunks from the first stage, reranks them and keeps the best twenty for the prompt. Reranking adds a step at run time and so a little latency, and the post tells you to tune how many chunks you rerank against cost and speed. The Voyage page lists rerank-3 as the highest-accuracy model for most applications and rerank-3-lite for latency-sensitive ones, both with a 32,000-token context.

import voyageai

vo = voyageai.Client()  # reads VOYAGE_API_KEY

# candidates: the text of each chunk the hybrid query returned (Anthropic's post started from 150)
reranked = vo.rerank(query, candidates, model="rerank-3", top_k=20)

for result in reranked.results:
    print(f"{result.relevance_score:.4f}", result.document[:60])

context_chunks = [result.document for result in reranked.results]  # these go into the prompt

• LlamaIndex calls reranking one of the biggest levers for RAG quality. Its recipe pairs similarity_top_k=10 with a local cross-encoder, cross-encoder/ms-marco-MiniLM-L6-v2, and top_n=3.
• An LLM-based reranker can beat a cross-encoder when latency is not critical, at the cost of extra LLM calls, per the same page.
• Anthropic tested the Cohere reranker. The post says it did not have time to test Voyage's.

Jo Bergum on where embeddings fail, LlamaIndex on what context costs

Jo Kristian Bergum runs Hornet.dev, a retrieval engine built for agents, by his X bio. On 30 August 2025 he wrote on X that some people know him as "the BM25 guy", and shared a paper by researchers at Google DeepMind and Johns Hopkins. It proves that an embedding's dimension caps how many different top-k result sets it can return. On the paper's deliberately simple LIMIT test, BM25 found 93.6% of the relevant documents in its top hundred results. The best single-vector model found 18.9%.

The paper, revised in March 2026, also tests the reverse. With every item swapped for a synonym, BM25 lost nearly 90% of its score and fell behind most embedding models. Keep both halves of the fused query, then. Your test questions should include some that reuse your documents' exact words and some that paraphrase them.

LlamaIndex, an AI research lab for document parsing and extraction by its X bio, pointed out on X on 1 October 2024 what Contextual Retrieval costs. Every chunk takes one model call, and every call carries the whole document as a prefix. The contextualize() loop above has that shape. Its prompt opens with the document, so only the chunk changes from call to call, which is what a prompt cache needs. Send one document's chunks one after another rather than mixing documents.

Screenshots of X's own embeds of the posts, taken on 5 November 2026, with the time in UTC. X cuts LlamaIndex's post short with Show more. Select one to open the post on X.

Common mistakes

• Mixing units. LangChain counts characters, OpenAI counts tokens, and a limit written in one is wrong in the other.
• Changing the chunk size and leaving the number of retrieved chunks alone. LlamaIndex raises similarity_top_k when it shrinks the size.
• Setting an OpenAI overlap above half the chunk size. The docs say it should not exceed max_chunk_size_tokens / 2.
• Using the English default separators on Chinese text, which can cut words in two.
• Showing the model the bare chunk when you indexed the contextualized one. Pass the context too, and label it.
• Borrowing a number from someone else's corpus. A chunk count, size or overlap that won for Anthropic or OpenAI says little about your documents.

Try it yourself

• Split one of your Markdown files with both splitters and print each chunk with its metadata. Which headings survive?
• Run the same ten questions at three chunk sizes and note which chunk the right answer came from each time.
• Write context for five chunks with the Anthropic prompt and read them. Would you find each chunk from its new text alone?
• Search for an exact identifier from your documents with vectors only, then with the fused query.

Next

Lesson 4 shows how to measure retrieval and answers, so each choice above rests on numbers from your own questions. To feed an index from files, databases or the web, the MCP servers on Mindket connect an agent to those sources.

Frequently asked questions

Sources

1. LangChain Docs: Text splitter integrations (read November 4, 2026)https://docs.langchain.com/oss/python/integrations/splitters
2. LangChain Docs: Splitting recursively (read November 5, 2026)https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter
3. LangChain Docs: Split markdown (read November 4, 2026)https://docs.langchain.com/oss/python/integrations/splitters/markdown_header_metadata_splitter
4. LlamaIndex Docs: Basic strategies (read November 5, 2026)https://developers.llamaindex.ai/python/framework/optimizing/basic_strategies/basic_strategies/
5. LlamaIndex Docs: Node parser modules (read November 4, 2026)https://developers.llamaindex.ai/python/framework/module_guides/loading/node_parsers/modules/
6. OpenAI Docs: Retrieval (read November 5, 2026)https://developers.openai.com/api/docs/guides/retrieval
7. Anthropic: Contextual Retrieval (read November 4, 2026)https://www.anthropic.com/research/contextual-retrieval
8. Voyage AI Docs: Rerankers (read November 5, 2026)https://www.mongodb.com/docs/voyageai/models/rerankers
9. Voyage AI Docs: Contextualized Chunk Embeddings (read November 5, 2026)https://www.mongodb.com/docs/voyageai/models/contextualized-chunk-embeddings
10. Voyage AI Docs: Quickstart (read November 4, 2026)https://www.mongodb.com/docs/voyageai/quickstart
11. pgvector README (read November 4, 2026)https://github.com/pgvector/pgvector
12. pgvector-python: hybrid search example (read November 4, 2026)https://github.com/pgvector/pgvector-python
13. Jo Kristian Bergum on X: On the Theoretical Limitations of Embedding-Based Retrieval (read November 5, 2026)https://x.com/jobergum/status/1961663599195144642
14. arXiv: On the Theoretical Limitations of Embedding-Based Retrieval (v2, March 2026) (read November 5, 2026)https://arxiv.org/abs/2508.21038
15. LlamaIndex on X: Implementing Contextual Retrieval RAG (read November 5, 2026)https://x.com/llama_index/status/1841210062167294287
16. Greg Kamradt on YouTube: The 5 Levels Of Text Splitting For Retrieval (read November 5, 2026)https://www.youtube.com/watch?v=8OJC21T2SL4
17. GitHub: 5 Levels Of Text Splitting notebook (read November 5, 2026)https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb