Chunking strategies for retrieval: what actually changes results

Chunking is the part of a retrieval pipeline people tune last and should tune first. It is also the part where the popular advice — pick a chunk size, add some overlap — optimises the least important variable.

Size matters. Boundaries matter more.

Why fixed-size splitting underperforms

Splitting every 1,000 characters is easy, which is why it is everywhere. It is also indifferent to meaning, so it reliably produces two failure modes.

It splits mid-thought. A claim ends up in one chunk and its qualification in the next. Retrieve either alone and you get something that reads as confident and is incomplete — the worst possible output, because nothing signals that context is missing.

It merges unrelated things. The end of one section and the start of the next land in one chunk. The embedding is now an average of two topics and sits near neither, so it retrieves for both weakly and wins on neither.

Overlap is the usual patch. It helps — a claim severed at a boundary appears whole in the neighbouring chunk — but it is a mitigation for a boundary you chose badly, and it multiplies your index size to do it.

Split where the author already split

Documents come with boundaries. Headings mark where the writer decided the subject changed, which is a far better signal than a character count, and it is free.

This is the practical reason to keep Markdown rather than flattening to plain text during content extraction. Strip the structure and you have thrown away the only boundary information the document contained.

function chunkByStructure(markdown: string, maxChars = 4000): string[] {
  const sections = markdown.split(/\n(?=## )/)
  const chunks: string[] = []

  for (const section of sections) {
    if (section.length <= maxChars) {
      chunks.push(section)
      continue
    }
    // Too long: fall back to paragraphs, which are still author-chosen.
    let buffer = ""
    for (const para of section.split(/\n\n+/)) {
      if (buffer.length + para.length > maxChars) {
        chunks.push(buffer)
        buffer = ""
      }
      buffer += para + "\n\n"
    }
    if (buffer.trim()) chunks.push(buffer)
  }
  return chunks
}

function chunkByStructure(markdown, maxChars = 4000) {
  const sections = markdown.split(/\n(?=## )/)
  const chunks = []

  for (const section of sections) {
    if (section.length <= maxChars) {
      chunks.push(section)
      continue
    }
    // Too long: fall back to paragraphs, which are still author-chosen.
    let buffer = ""
    for (const para of section.split(/\n\n+/)) {
      if (buffer.length + para.length > maxChars) {
        chunks.push(buffer)
        buffer = ""
      }
      buffer += para + "\n\n"
    }
    if (buffer.trim()) chunks.push(buffer)
  }
  return chunks
}

import re

def chunk_by_structure(markdown: str, max_chars: int = 4000) -> list[str]:
    sections = re.split(r"\n(?=## )", markdown)
    chunks: list[str] = []

    for section in sections:
        if len(section) <= max_chars:
            chunks.append(section)
            continue
        # Too long: fall back to paragraphs, which are still author-chosen.
        buffer = ""
        for para in re.split(r"\n\n+", section):
            if len(buffer) + len(para) > max_chars:
                chunks.append(buffer)
                buffer = ""
            buffer += para + "\n\n"
        if buffer.strip():
            chunks.append(buffer)
    return chunks

The fallback ladder is the point: headings, then paragraphs, then sentences, and only then a character count. Every rung down loses information, so take the highest one that fits.

Attach context to every chunk

A chunk that begins "It grew faster than usage did" is close to meaningless in isolation. The embedding has no idea what "it" is, and neither will the model that retrieves it.

Prepend the document title, and the section heading where you have one:

const enriched = chunks.map((body, i) => ({
  text: doc.title + "\n" + headingFor(i) + "\n\n" + body,
  metadata: { url: doc.url, title: doc.title, publishedAt: doc.publishedAt },
}))

const enriched = chunks.map((body, i) => ({
  text: doc.title + "\n" + headingFor(i) + "\n\n" + body,
  metadata: { url: doc.url, title: doc.title, publishedAt: doc.publishedAt },
}))

enriched = [
    {
        "text": doc["title"] + "\n" + heading_for(i) + "\n\n" + body,
        "metadata": {
            "url": doc["url"],
            "title": doc["title"],
            "publishedAt": doc["publishedAt"],
        },
    }
    for i, body in enumerate(chunks)
]

This is the cheapest quality improvement available in a retrieval pipeline and it is routinely skipped. It costs a handful of tokens per chunk and it resolves every pronoun the chunk opens with.

Size, now that boundaries are handled

With structural boundaries, size becomes a soft ceiling rather than the rule:

• 2,000–4,000 characters suits most prose. Big enough for a complete thought, small enough that retrieval is precise.
• Smaller for reference material — API docs, FAQs, definitions — where each entry is self-contained and precision matters more than context.
• Larger for narrative or argumentative text, where the reasoning spans paragraphs and splitting it costs more than the imprecision.

Check what you are actually producing rather than assuming, since character count and token count diverge sharply on code and non-English text:

const { inputTokens } = await priceChunk(chunk)

const { inputTokens } = await priceChunk(chunk)

input_tokens = price_chunk(chunk)["inputTokens"]

What to measure

Chunking changes are easy to make and hard to evaluate by feel. Build a small evaluation set — twenty real questions with the passage that should answer each — before you tune anything. Then measure whether the right passage appears in the top k.

Without that, you are changing a number and trusting a vibe. Most chunking "improvements" reported in blog posts are exactly that.

Two things that are usually not the problem

The embedding model. Teams swap models to fix retrieval far more often than the model was at fault. If your chunks contain navigation, no embedding model will save you.

The vector database. At the scale most applications operate at, they perform equivalently on quality. Choose on operations, not recall.

The problem is almost always what went into the index.

The short version

Extract before you chunk. Split on headings, then paragraphs, then sentences, and only then on length. Prepend the title and heading to every chunk. Keep the URL and date. Build twenty test questions before tuning anything.

Boundaries first, size second, and the model last.

Chunking is half the job; retrieval is the other half

Good chunks retrieved badly still produce bad answers, and the two failures look identical from the outside. Three things at the retrieval step interact directly with how you chunked.

Retrieve more than you send. Fetch a generous number of candidates, then narrow. Vector similarity is good at finding the neighbourhood and mediocre at ordering within it, so the passage that actually answers the question is frequently in the top twenty and not the top three. Widening the net costs almost nothing; a reranking pass over those candidates is where most of the accuracy people attribute to better embeddings actually comes from.

Deduplicate before you send. If you index the same content from several URLs — a canonical page and a syndicated copy, or a doc versioned across releases — near-identical chunks will occupy several of your top slots and crowd out the passage that would have completed the answer. Dropping chunks above a similarity threshold to one already selected is a few lines and it recovers real slots.

Give the model the source. Send the URL and the date alongside each chunk, and ask for citations. This is not only for the reader: a model that must attribute a claim to a passage is measurably less inclined to assert things the passages do not support, and you get a trace you can check when it does.

Where a chunk overflows the budget you have, that is a chunking signal rather than a retrieval one — reduce the ceiling and reindex rather than truncating at query time, because truncation cuts at an arbitrary point, which is the failure structural splitting existed to avoid.

What you can build with this

A retrieval evaluation harness you actually keep. Twenty questions, the passage that should answer each, and a script that reports how often the right one lands in the top k. It takes an afternoon, and it turns every future chunking, embedding or reranking change from an argument into a number. Most people skip this and then cannot tell whether anything they did helped.

An "answer with sources" bot for a body of docs you own. Extract, chunk on structure, prepend titles, retrieve widely, rerank, cite. The citation requirement is what makes it trustworthy enough to give to other people, and it is the difference between a demo and something a team uses daily.

A chunk inspector. A page that shows you your own chunks, with token counts and boundaries highlighted. Unglamorous, and it will immediately show you two or three things about your pipeline that no amount of reasoning about it would have.

Start with the evaluation harness even though it is the least fun. Everything after it gets easier to judge.

Related reading

• Preparing web content for a RAG pipeline — the step before this one, and the reason to keep Markdown rather than plain text.
• How to control LLM API costs — retrieved context is input you pay for on every call.
• Web Extract and Token Counter — extraction and sizing for the pipeline above.