Chunking: splitting a document that does not fit, without cutting a sentence in half

Why chunk at all

A model has a context window, and a document is often larger than it. Even when the document fits, sending forty pages to answer a question about one paragraph costs forty pages of tokens per question. And an embedding — module 8's vector — represents one piece of text; a vector for an entire book averages away everything specific. So documents get split into chunks: small enough to embed meaningfully and to send cheaply, large enough to carry a complete thought.

The naive version, and its failure:

chunks = [text[i:i + 1000] for i in range(0, len(text), 1000)]

This cuts at character 1,000 wherever that falls — mid-word, mid-sentence, mid-number. A fact at the boundary is split between two chunks, and neither chunk contains it. The search finds nothing, or finds half, and the answer is confidently wrong. Almost every complaint that "the search does not find things I know are there" traces to this.

Split on structure first

Text has boundaries that mean something: paragraphs, then sentences, then words. Split at the largest boundary that keeps chunks under the limit, and only fall back to smaller ones when a single unit is too big:

import re

def split_paragraphs(text):
    return [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]

def split_sentences(text):
    return [s.strip() for s in re.split(r"(?<=[.!?।])\s+", text) if s.strip()]

The sentence pattern splits after ., !, ? and the Devanagari full stop ।, followed by whitespace. It is deliberately simple. It will split "Dr. Sharma" in the wrong place and it does not know that "3.5" is a number. For most documents that is acceptable; when it is not, the free nltk sentence tokeniser or spacy handle abbreviations and more languages, at the cost of a download.

Assemble to a token budget

def chunk(text, max_tokens, count):
    chunks, current = [], []
    for para in split_paragraphs(text):
        units = [para] if count(para) <= max_tokens else split_sentences(para)
        for unit in units:
            if count(" ".join(current + [unit])) > max_tokens and current:
                chunks.append(" ".join(current))
                current = []
            current.append(unit)
    if current:
        chunks.append(" ".join(current))
    return chunks

Greedy: add units until the next would overflow, then start a new chunk. count is a token counter from module 6, because a chunk limit in characters is off by a factor of two or three for Hindi or code, and the model's limit is in tokens. A paragraph that fits goes in whole; one that does not is split into sentences first. A sentence larger than the budget on its own — a pasted log, a long table row — still passes through oversized; cut it by characters as a last resort, and log that you did.

Overlap

Even with sentence boundaries, a fact can span two chunks: the question is set up at the end of one and answered at the start of the next. Overlap repeats the tail of each chunk at the head of the next:

def with_overlap(chunks, overlap_units=1):
    out = []
    for i, c in enumerate(chunks):
        if i > 0:
            tail = split_sentences(chunks[i - 1])[-overlap_units:]
            c = " ".join(tail) + " " + c
        out.append(c)
    return out

One or two sentences of overlap is typical. It costs a little storage and duplicates a little text in results, and it is the difference between finding boundary facts and not.

Keep the origin

A chunk that cannot say where it came from is a search result nobody can check:

from dataclasses import dataclass

@dataclass(frozen=True)
class Chunk:
    doc_id: str
    index: int
    text: str
    start_char: int

With doc_id and start_char, a result can link back to the page and highlight the passage. Store these alongside the vector; module 8's argsort gives you an index, and this is what the index points to.

Unicode edges

Slicing a string by characters can split a grapheme — a Devanagari consonant from its vowel sign, an emoji from its skin-tone modifier — producing a chunk that ends in a broken character and renders as a box. Splitting on whitespace and punctuation, as above, avoids it. If you must cut by count, regex (the third-party module, free) offers \X to step by grapheme, and module 5's normalisation should happen before any of this.

Choosing the size

There is no right number. Smaller chunks — 100 to 300 tokens — embed precisely and return exact passages, but lose context; a chunk saying "it rose 12 per cent" without the sentence saying what "it" is. Larger chunks — 500 to 1,000 — carry context and cost more per result. Most people start around 300 to 500 with a sentence of overlap and adjust after looking at what the search returns for twenty real questions, which is the evaluation habit from that course applied here. Measure with your own questions before believing anyone's default, including this one.

Try this now

Chunk a long document — a README, a chapter, a policy — with max_tokens=300 and print each chunk's first and last sentence. Find a fact near a boundary and confirm it appears whole in at least one chunk. Then do the same with the naive character splitter and find the fact it cut.