How to prepare web content for a RAG pipeline without wasting tokens

The first version of almost every retrieval pipeline fetches a URL, passes the HTML to a model, and works. It also costs several times what it should and retrieves badly, and neither problem announces itself.

Where the tokens actually go

Fetch an average article page and look at what you have. A modern page is largely navigation, inline scripts, style blocks, cookie notices, related-article widgets, comment scaffolding and footers. The article, the reason anyone visited, is frequently under a fifth of the bytes.

Two consequences follow, and only one is obvious.

The obvious one is cost: you pay for every token, and most of them are markup. The subtler one is quality. If you embed a chunk containing navigation, the resulting vector describes the navigation. Since every page on a site shares that navigation, those chunks look alike and retrieve for queries they cannot answer. A pipeline that returns the footer for three unrelated questions usually has exactly this cause.

Extract before you do anything else

Strip the page to its content first.

curl "https://twotic.dev/v1/web-extract/extract?url=https://example.com/post&format=markdown" \
  -H "Authorization: Bearer $TWOTIC_KEY"

You get the article as Markdown, with the metadata the page declared:

{
  "url": "https://example.com/post",
  "title": "How we cut inference cost by 80%",
  "publishedAt": "2026-06-14T09:00:00Z",
  "wordCount": 1840,
  "content": "## The problem\n\nOur inference bill grew faster than usage did…"
}

Markdown rather than plain text is deliberate. Headings mark where topics change, which is the single most useful signal for deciding where to split — and stripping to plain text throws it away.

Measure the difference

Do not take the saving on faith. Price both versions:

curl -X POST "https://twotic.dev/v1/token-counter/count"   -H "Authorization: Bearer $TWOTIC_KEY"   -H "Content-Type: application/json"   -d '{"text": "<your text>", "model": "claude-sonnet-4"}'

# inputTokens comes back on the response; compare raw HTML against extracted
# text by running it twice.

const priceIt = (text: string) =>
  fetch("https://twotic.dev/v1/token-counter/count", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.TWOTIC_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ text, model: "claude-sonnet-4" }),
  }).then((r) => r.json() as Promise<{ inputTokens: number }>)

const rawHtml = await (await fetch(url)).text()
const { content } = await extract(url)

console.log((await priceIt(rawHtml)).inputTokens)   // e.g. 24,900
console.log((await priceIt(content)).inputTokens)   // e.g. 2,400

const priceIt = (text) =>
  fetch("https://twotic.dev/v1/token-counter/count", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.TWOTIC_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ text, model: "claude-sonnet-4" }),
  }).then((r) => r.json())

const rawHtml = await (await fetch(url)).text()
const { content } = await extract(url)

console.log((await priceIt(rawHtml)).inputTokens)   // e.g. 24,900
console.log((await priceIt(content)).inputTokens)   // e.g. 2,400

import os
import requests

def price_it(text: str) -> dict:
    return requests.post(
        "https://twotic.dev/v1/token-counter/count",
        headers={"Authorization": f"Bearer {os.environ['TWOTIC_KEY']}"},
        json={"text": text, "model": "claude-sonnet-4"},
    ).json()

raw_html = requests.get(url).text
content = extract(url)["content"]

print(price_it(raw_html)["inputTokens"])   # e.g. 24,900
print(price_it(content)["inputTokens"])    # e.g. 2,400

A ten-to-one reduction is common on article pages. At that ratio the extraction call pays for itself on the first request, and every one after it is saving.

Check it fits before you send it

A context-window overflow is a failed call you still waited for. The same estimate tells you in advance:

const { fitsInContext, contextUsedPercent } = await priceIt(content)

if (!fitsInContext) {
  // Chunk, or summarise section by section, rather than sending and failing.
}

const { fitsInContext, contextUsedPercent } = await priceIt(content)

if (!fitsInContext) {
  // Chunk, or summarise section by section, rather than sending and failing.
}

result = price_it(content)

if not result["fitsInContext"]:
    # Chunk, or summarise section by section, rather than sending and failing.
    ...

Chunk on headings, not on character counts

Fixed-size chunking is popular because it is easy, and it splits sentences and separates a claim from its qualification.

Because you kept the Markdown, you can split where the document itself changes subject:

function chunkByHeading(markdown: string, maxChars = 4000): string[] {
  const sections = markdown.split(/\n(?=## )/)
  const chunks: string[] = []

  for (const section of sections) {
    if (section.length <= maxChars) {
      chunks.push(section)
      continue
    }
    // A section longer than the limit is split on paragraphs, which is still
    // a boundary the author chose.
    let buffer = ""
    for (const para of section.split(/\n\n+/)) {
      if (buffer.length + para.length > maxChars) {
        chunks.push(buffer)
        buffer = ""
      }
      buffer += para + "\n\n"
    }
    if (buffer.trim()) chunks.push(buffer)
  }
  return chunks
}

function chunkByHeading(markdown, maxChars = 4000) {
  const sections = markdown.split(/\n(?=## )/)
  const chunks = []

  for (const section of sections) {
    if (section.length <= maxChars) {
      chunks.push(section)
      continue
    }
    // A section longer than the limit is split on paragraphs, which is still
    // a boundary the author chose.
    let buffer = ""
    for (const para of section.split(/\n\n+/)) {
      if (buffer.length + para.length > maxChars) {
        chunks.push(buffer)
        buffer = ""
      }
      buffer += para + "\n\n"
    }
    if (buffer.trim()) chunks.push(buffer)
  }
  return chunks
}

import re

def chunk_by_heading(markdown: str, max_chars: int = 4000) -> list[str]:
    sections = re.split(r"\n(?=## )", markdown)
    chunks: list[str] = []

    for section in sections:
        if len(section) <= max_chars:
            chunks.append(section)
            continue
        # A section longer than the limit is split on paragraphs, which is
        # still a boundary the author chose.
        buffer = ""
        for para in re.split(r"\n\n+", section):
            if len(buffer) + len(para) > max_chars:
                chunks.append(buffer)
                buffer = ""
            buffer += para + "\n\n"
        if buffer.strip():
            chunks.append(buffer)
    return chunks

Prepend the page title to each chunk before embedding. A chunk that begins "It grew faster than usage did" is nearly meaningless alone; with the title attached it is retrievable.

Store the metadata alongside

Keep title, url and publishedAt with every chunk. Publication date lets you prefer recent sources, and the URL lets you cite one — an answer that cites its source is verifiable, and one that cannot is a claim the user has to take on trust.

Let an agent do it directly

If an agent is doing the reading rather than a batch job, connect the API over MCP instead of writing an integration:

{
  "mcpServers": {
    "web-extract": {
      "type": "http",
      "url": "https://twotic.dev/api/mcp/web-extract",
      "headers": { "Authorization": "Bearer tk_live_your_key" }
    }
  }
}

The agent gets a tool that reads any URL and returns clean Markdown, billed identically to the HTTP call.

What this adds up to

Extract, measure, check the fit, chunk on headings, keep the metadata. It is a short list, and it is most of the difference between a retrieval pipeline that is cheap and accurate and one that is expensive and vague.

What extraction cannot fix

Extraction is a text problem, and some pages are not a text problem. Knowing which is which saves you from debugging a pipeline that was never going to work.

Pages that render client-side. If the article is assembled by JavaScript after load, the HTML you fetch contains a loading shell and nothing else. No extractor can recover text that was not in the response. You need a headless browser, or — far better where it exists — the site's own feed or API, which will be faster, cheaper and more stable than scraping the render.

Pages behind a paywall or a login. The server returned a teaser and an invitation to subscribe. That is genuinely the whole document, and an extractor faithfully reporting 200 words is not failing.

Documents that are not HTML. A PDF is a layout format, not a text format: reading order is a property of how it was generated, and a two-column academic paper interleaves both columns line by line unless something reconstructs the flow. Route PDFs to a PDF parser rather than through an HTML extractor.

Pages you were asked not to fetch. Check robots.txt, honour it, and identify yourself with a real user agent and a way to be contacted. This is not only etiquette — sites that see well-behaved crawlers block them far less often, and an IP that gets blocked takes your whole pipeline with it.

What you can build with this

Three things that are a weekend each, and all of them are more useful than they sound.

A "read this later, ask it anything" inbox. Save a URL, extract it on the way in, index it. You end up with a searchable archive of everything you meant to read, and asking it a question beats trying to remember which article had the bit you wanted. The trick that makes it good rather than a demo is storing the extraction, not the URL, so the answer survives the page being edited or taken down.

A competitor and release-notes watcher. Extract a set of pages on a schedule, keep the previous Markdown, and diff. Because you are diffing extracted content rather than HTML, a redesign or a new banner does not fire; a changed price or a new paragraph does. Send the diff to yourself weekly.

Documentation search for a tool that has none worth using. Plenty of good projects have docs you cannot search. Extract the pages, chunk them, index them, and put a box on top. This one is genuinely worth publishing: it helps everyone who uses that tool, and it costs you a weekend.

Each of these is the same three steps in a different order. Build one and the next two are mostly configuration.

Related reading

• Chunking strategies for retrieval — what to do with the clean Markdown once you have it, and why boundaries matter more than chunk size.
• How to control LLM API costs — extraction is usually the largest single saving, and this covers the other three.
• Web Extract and Token Counter — the two endpoints used throughout this guide.