RAG I — Chunking Strategies & Indexing

Why this session exists

An index can retrieve only what ingestion preserved, but RAG failures also arise from parsing, permissions, stale versions, retrieval, evidence selection and generation. Chunking is one important stage—not a measured 80% of every system's performance and not the explanation for every wrong answer.

This lesson develops chunk boundaries, context budgets, metadata and incremental ingestion as a single contract. The lab builds a local lexical-vector index over inline Markdown, with source offsets, versioned IDs, deletion reconciliation, atomic snapshot replacement and authorization filters. It is not MiniLM inference, a Chroma integration test or a neural retrieval benchmark.

Outcomes: compare fixed/recursive/structural/semantic splitting; tune size and overlap; retain provenance and access metadata; distinguish ingestion/retrieval/generation errors; and build an idempotent, incremental, rebuildable index that removes stale chunks.

Prerequisites

Session 106 covers tokenization; a character count is not a model token count. Session 116 covers evidence-only prompting and untrusted data. Know the difference between embedding a chunk, ranking it and verifying that it supports a final claim.

(a) Intuition · 5 min

A librarian with a limited reading budget needs coherent excerpts and catalog labels. A torn sentence can omit its referent; an oversized excerpt can mix relevant and irrelevant material. Yet chunk quality is not the entire answer process: the librarian may select the wrong edition, lack permission, misunderstand the passage or invent a claim despite seeing it.

Where RAG came from

Lewis et al.'s 2020 RAG paper combines a pretrained sequence-to-sequence model with a dense retriever and a Wikipedia index [1]. Modern application pipelines often retrieve text into a prompt without jointly training that architecture. The shared idea is combining parametric generation with retrieved nonparametric evidence, not that every framework implements the original paper unchanged. Longer-context models expand one resource budget; they do not remove source freshness or access-control requirements.

The RAG pipeline (chunking + indexing is the left half)

Record source bytes before normalization and retain a mapping from normalized text to source/page offsets when extraction changes it. PDF reading order, OCR mistakes, lost table headers and code parsing can break retrieval before chunk size matters. The handoff is an active, versioned manifest of vectors and metadata, not a pile of untracked upserts.

The four splitting strategies

• Fixed token windows: simple, deterministic and bounded under the specified tokenizer. They may cut logical units but are a useful baseline, including on poorly structured text. Fixed characters are a different unit.
• Recursive splitting: try stronger separators before weaker ones, then enforce a size bound. In the cited langchain-text-splitters==0.3.11 implementation, default separators are paragraph, newline, space and empty string [2]. Sentence boundaries are not automatically guaranteed; language-specific punctuation and fallback behavior matter.
• Structural splitting: use headings, HTML structure, code ASTs, table groups or speaker turns. Preserve breadcrumbs and source ranges. Large functions/tables can exceed the model window: choose a documented secondary split with repeated context, summarize separately or reject for review. “Never split any structure” is impossible under every finite budget.
• Semantic splitting: compare neighboring sentence representations or another topic-change score. It can be deterministic with fixed model bytes, tokenizer, precision and threshold; changes across model versions are version drift, not intrinsic randomness. Extra embedding work and threshold tuning must earn their cost on evaluation.

For Markdown, headers inside fenced code are not section boundaries. For Python, a real AST/Tree-sitter parser provides syntax nodes; LangChain's language-aware recursive separator list is a heuristic, not automatically an AST parser. Keep imports, class context and source line ranges when splitting code. For tables, repeat necessary column headers with split row groups and maintain cell provenance.

The chunk-size × overlap heatmap (intuition)

Compare candidate sizes such as 128,256,512,1024 tokens only when the chosen embedding model accepts them including prefixes and special tokens. Short units can improve focus but omit context; long units can contain an answer but dilute ranking or waste generator budget. Neither trend is strictly monotonic on all corpora, and no unique interior optimum is mathematically guaranteed.

The original MiniLM example called 512×4 characters “512 tokens.” That is an English heuristic, not a bound. The all-MiniLM-L6-v2 model card says input beyond 256 word pieces is truncated by default [3]. A 512-token splitter therefore cannot be assumed compatible with that model. Count using its actual tokenizer/configuration, reserve room for title/breadcrumb/special tokens and detect truncation. Hindi, Telugu, emoji, code and identifiers make characters-per-token estimates particularly unreliable.

Overlap: tune boundary coverage against duplication cost

For fixed windows of size s and overlap o with 0≤o<s, stride is s−o. Ignoring end effects, the number of indexed tokens grows by a factor s/(s−o)=1/(1−o/s). At 15% overlap that is about 1.17647, or 17.65% extra—not exactly 15%. At 50% it is about 2×. With 512/80 the factor is 512/432≈1.18519.

These are zero-based half-open intervals, each length 512. The earlier one-based endpoints mixed counts. Overlap reduces the chance of losing a boundary-spanning fact but does not guarantee every supporting span fits one chunk. A span longer than s cannot be contained, and a span crossing a boundary by more than o can still be split. Adjacent retrieval, parent expansion or multi-chunk evidence can recover it.

Metadata: the retrieval super-filter

Each chunk needs a stable document identity and source revision; exact source range; title/heading path; parser/chunker/tokenizer/embedding version; creation/update and effective dates where relevant; tenant and access policy; content type; position within document; and links to parent/neighbor units. Page numbers must come from the parser, not guesses. An update timestamp is not the same as a legal effective date; a newer draft can be less authoritative than an older currently effective policy.

Authorization must be enforced from trusted identity/policy outside the model and before unauthorized text is exposed. Query filters are not sufficient if a caller may choose its own tenant ID without authentication. Recheck access on neighbor/parent expansion, reranking, caching and final citations. Policy changes can invalidate cached answers even when source text stays unchanged.

Prefix chunks with their heading before embedding

A title such as “API > Auth > Refresh Tokens” can clarify an otherwise ambiguous “call it first.” Store source text and the enriched embedding text separately so citations still point to the exact source. Prefixes consume embedding tokens and may introduce false associations; evaluate rather than promise a fixed retrieval gain. Query/document embedding transforms must share a compatible model/version/normalization contract, including asymmetric prefixes when the embedding model requires them.

What you embed is not what you return

The index unit can be a focused passage while the returned context is its authorized parent section or a bounded neighbor window. This separates matching from sufficiency but does not eliminate all tradeoffs: expansion uses generator budget and may add irrelevant or conflicting text. Deduplicate overlapping ranges, retain source diversity and count the final prompt's tokens.

A simple dilution illustration averages k orthogonal unit topic vectors. After normalization its cosine with one topic is 1/√k. Real contextual encoders do not necessarily average orthogonal topics, and CLS pooling is not literally arithmetic mean pooling. The example motivates a risk; it is not a derivation of every model's retrieval curve.

Boundary coverage and evaluation

For labeled answer spans, measure whether one chunk contains each entire span. That is a useful single-chunk intact-span coverage metric. It is not an unconditional ceiling on general recall@1: the task might accept a partial passage, another duplicate source or generated synthesis. State the relevance definition and denominator. Separately measure retrieval recall@k, ranking quality, source-version correctness, authorization violations, evidence sufficiency and generated-answer support.

Use several realistic questions and a held-out set, not five handpicked examples as proof that 512 always wins. Keep embedding model, retrieval budget and evaluation labels fixed while changing chunking. Compare total returned-token budget as well as k, since k chunks of different sizes are not equal-cost inputs.

(c) Hands-on · 25 min

The following program builds a complete in-memory index lifecycle on a declared whitespace teaching tokenizer and fixed lexical vectorizer. That tokenizer is exact for this fixture, not a substitute for a neural model tokenizer. Structural sections preserve offsets and fenced blocks; an oversized fenced block is explicitly rejected instead of silently truncated. The fixture uses no filesystem corpus, model download, external service or installed vector database.

import copy,hashlib,json,math,re
from dataclasses import dataclass
 
def digest(x):return hashlib.sha256(x.encode()).hexdigest()
def canonical(x):return json.dumps(x,sort_keys=True,separators=(',',':'))
def tokens(x):return list(re.finditer(r'\S+',x)) # Declared teaching tokenizer, not neural tokens.
 
def sections(text):
    lines=text.splitlines(keepends=True);offset=0;start=0;heading=[];fence=None;rows=[]
    for line in lines:
        marker=re.match(r'^ {0,3}(`{3,}|~{3,})([^\n]*)\n?$',line)
        was_fenced=fence is not None
        if marker:
            m,tail=marker.groups()
            if fence is None:
                if m[0]!='`' or '`' not in tail:fence=(m[0],len(m))
            elif m[0]==fence[0] and len(m)>=fence[1] and not tail.strip(' \t\r'):
                fence=None
        h=re.match(r'^(#{1,6}) (.+?)\s*$',line) if not was_fenced and fence is None and not marker else None
        if h:
            if offset>start:rows.append((start,offset,' > '.join(name for _,name in heading)))
            depth=len(h.group(1))
            heading=[(level,name) for level,name in heading if level<depth]+[(depth,h.group(2))]
            start=offset
        offset+=len(line)
    if fence is not None:raise ValueError('unclosed code fence')
    if offset>start:rows.append((start,offset,' > '.join(name for _,name in heading)))
    return rows
 
def chunk_doc(text,size,overlap):
    if size<=0 or not 0<=overlap<size:raise ValueError('invalid size/overlap')
    out=[]
    for a,b,breadcrumb in sections(text):
        body=text[a:b];matches=tokens(body)
        prefix=(breadcrumb+'\n') if breadcrumb else ''
        budget=size-len(tokens(prefix))
        if budget<=overlap:raise ValueError('prefix leaves insufficient body budget')
        if '```' in body or '~~~' in body:
            if len(matches)>budget:raise ValueError('oversized fenced section needs structural review')
            starts=[0] if matches else []
        else:starts=range(0,max(0,len(matches)-overlap),budget-overlap) if len(matches)>budget else ([0] if matches else [])
        for start in starts:
            end=min(start+budget,len(matches))
            lo=a+matches[start].start();hi=a+matches[end-1].end()
            out.append({'start':lo,'end':hi,'text':text[lo:hi],'breadcrumb':breadcrumb,
                        'embedding_text':prefix+text[lo:hi]})
            assert len(tokens(out[-1]['embedding_text']))<=size
            if end==len(matches):break
    return out
 
VOCAB=('oauth','refresh','token','billing','invoice','refund','secret','python')
def lexical(text):
    words=re.findall(r'\w+',text.lower());v=[words.count(w) for w in VOCAB]
    norm=math.sqrt(sum(x*x for x in v));return [x/norm for x in v] if norm else [0.]*len(v)
@dataclass
class Doc:
    id:str
    text:str
    readers:tuple
    effective_from:int=0
    effective_to:int=9999
 
class Index:
    def __init__(self):self.records={};self.manifests={};self.cache={};self.embedded=0
    def reconcile(self,tenant,docs,size=24,overlap=3,embedding_version='lexical-v1',fail_text=None):
        # Caller has authenticated tenant and verified a COMPLETE source listing.
        if len({d.id for d in docs})!=len(docs):raise ValueError('duplicate document identity')
        staged={};manifest={};cache=dict(self.cache);new_count=0
        config={'size':size,'overlap':overlap,'tokenizer':'whitespace-v1','parser':'md-teaching-v1','embedding':embedding_version}
        for doc in docs:
            revision=digest(doc.text);ids=[]
            for piece in chunk_doc(doc.text,size,overlap):
                key=digest(canonical([tenant,doc.id,revision,piece['start'],piece['end'],config]))
                cachekey=digest(canonical([embedding_version,piece['embedding_text']]))
                if fail_text and fail_text in piece['embedding_text']:raise RuntimeError('injected embedding failure')
                if cachekey not in cache:cache[cachekey]=lexical(piece['embedding_text']);new_count+=1
                staged[key]=dict(piece,id=key,doc_id=doc.id,revision=revision,tenant=tenant,
                    readers=list(doc.readers),effective_from=doc.effective_from,effective_to=doc.effective_to,
                    config=config,vector=cache[cachekey])
                ids.append(key)
            manifest[doc.id]={'revision':revision,'ids':ids,'readers':list(doc.readers)}
        # Publish only after all staged processing succeeds; preserve other tenants.
        preserved={k:v for k,v in self.records.items() if v['tenant']!=tenant}
        preserved.update(staged);self.records=preserved
        self.manifests=dict(self.manifests);self.manifests[tenant]=manifest
        self.cache=cache;self.embedded+=new_count
        return new_count
    def search(self,tenant,user,query,asof=1,k=3,embedding_version='lexical-v1'):
        q=lexical(query);rows=[]
        for item in self.records.values():
            if item['tenant']!=tenant or user not in item['readers']:continue
            if not item['effective_from']<=asof<item['effective_to']:continue
            if item['config']['embedding']!=embedding_version:continue
            score=sum(a*b for a,b in zip(q,item['vector']))
            if score>0:rows.append((score,item['id'],item))
        return [item for _,_,item in sorted(rows,key=lambda r:(-r[0],r[1]))[:k]]
 
a=Doc('auth','# API\n## OAuth\nUse the refresh token to renew access.\n\nRepeat the refresh token policy.\n',('alice',))
b=Doc('billing','# Billing\nAn invoice refund needs approval.\n',('alice','bob'))
index=Index();first=index.reconcile('A',[a,b]);assert first>0
snapshot=canonical(index.records);assert index.reconcile('A',[a,b])==0
assert canonical(index.records)==snapshot
for item in index.records.values():
    doc=next(d for d in [a,b] if d.id==item['doc_id'])
    assert doc.text[item['start']:item['end']]==item['text']
assert index.search('A','alice','OAuth refresh')[0]['doc_id']=='auth'
assert not index.search('A','bob','OAuth refresh')
# Other tenant survives updates/deletions in A.
index.reconcile('B',[Doc('private','# Secret\nsecret token\n',('bob',))]);other=canonical({k:v for k,v in index.records.items() if v['tenant']=='B'})
assert not index.search('A','bob','secret')
# Shorten a document: none of its old IDs remain active.
old=set(index.manifests['A']['auth']['ids'])
short=Doc('auth','# API\nRefresh token.\n',('alice',))
index.reconcile('A',[short,b]);assert old.isdisjoint(index.records)
# Remove billing from complete listing.
oldbilling=set(index.manifests['A']['billing']['ids']);index.reconcile('A',[short])
assert oldbilling.isdisjoint(index.records)
# Metadata-only ACL update reuses vectors, yet access changes.
changed=Doc('auth',short.text,('bob',));assert index.reconcile('A',[changed])==0
assert not index.search('A','alice','refresh') and index.search('A','bob','refresh')
assert canonical({k:v for k,v in index.records.items() if v['tenant']=='B'})==other
# Failure before publication preserves records, manifest and cache.
before=canonical([index.records,index.manifests,index.cache]);count=index.embedded
try:index.reconcile('A',[Doc('new','# FAIL\nrefresh token',('bob',))],fail_text='FAIL')
except RuntimeError:pass
else:raise AssertionError('failure not injected')
assert canonical([index.records,index.manifests,index.cache])==before and index.embedded==count
# A version change requires new embedding cache keys and compatible query version.
assert index.reconcile('A',[changed],embedding_version='lexical-v2')>0
assert not index.search('A','bob','refresh')
assert index.search('A','bob','refresh',embedding_version='lexical-v2')
# Fenced heading is not a section; oversized code is rejected, not truncated.
fenced='# Code\n```python\n# Not a heading\nprint(1)\n```\n'
assert len(sections(fenced))==1
assert len(chunk_doc(fenced,24,3))==1
# A shorter run or an info-bearing run cannot close the outer fence.
nested='# Code\n````python\n```\n# Still code\n````not-a-close\n# Also code\n````\n# After\ntext\n'
assert [h for _,_,h in sections(nested)]==['Code','After']
assert [h for _,_,h in sections('# A\n### B\nx\n### C\ny\n')]==['A','A > B','A > C']
try:chunk_doc(fenced,5,1)
except ValueError:pass
else:raise AssertionError('oversized code silently truncated')
# Repeated same text at distinct offsets retains distinct identifiers.
repeat=Doc('repeat','# A\nrefresh token\n# A\nrefresh token\n',('bob',))
index.reconcile('A',[repeat]);assert len(index.records)==len(index.manifests['A']['repeat']['ids'])+len(index.manifests['B']['private']['ids'])
ids=index.manifests['A']['repeat']['ids'];assert len(ids)==len(set(ids))==2
# Empty authoritative snapshot deletes A, not B.
index.reconcile('A',[]);assert not any(v['tenant']=='A' for v in index.records.values())
# Coverage illustration in half-open token units.
def windows(n,s,o):return [(a,min(a+s,n)) for a in range(0,n,s-o)]
span=(9,14)
assert not any(a<=span[0] and b>=span[1] for a,b in windows(20,10,0))
assert any(a<=span[0] and b>=span[1] for a,b in windows(20,10,5))
assert math.isclose(1/(1-.15),1.1764705882352942)
for size in (8,12,24):
    pieces=chunk_doc(a.text,size,min(2,size//4));print('teaching size',size,'chunks',len(pieces))
print('source ranges, incremental manifests, deletion, ACL, rollback and negative controls passed')

Line-by-line

The teaching parser recognizes unindented ATX headings outside backtick/tilde fences and preserves character offsets into the original source. It tracks the opening fence's character and length; a closer must use the same character, at least that length, and only trailing whitespace. A shorter run inside a four-backtick block remains code. Breadcrumbs use heading depths, so two level-three headings under a level-one heading are siblings, not parent and child. This is a bounded Markdown subset, not a CommonMark parser: setext headings, container blocks and indented-code semantics require a real parser. Unlike CommonMark's end-of-document rule, it deliberately rejects unclosed fences for ingestion review (CommonMark 0.31.2, fenced code blocks). Chunk IDs bind tenant, document ID, source revision, range and chunking/embedding contract. Repeated identical text at two positions therefore remains distinguishable. A full SHA-256 digest avoids the original unnecessarily truncated SHA-1 identity, but uniqueness alone is not incremental ingestion.

reconcile treats its input as an authoritative full snapshot for one tenant, not an arbitrary delta. It stages the new records and manifest, reuses embeddings only for identical enriched text under the same embedding version, and swaps the active tenant view only after all processing succeeds. Missing documents and obsolete chunks disappear from that tenant's active view; other tenants remain untouched. This single-threaded fixture offers rollback before publication, not a concurrent multi-field transaction: records, manifests and cache are assigned separately. A real concurrent/distributed store needs a lock, immutable generation/alias or transactional publication protocol spanning the active view. An upsert by itself does not remove old IDs.

Metadata updates, including access policy/effective date, are applied even when an embedding is reused. Empty corpora are valid authoritative snapshots; failed/partial source listings are not authority to delete everything. The caller must distinguish successful empty enumeration from an upstream outage. A production pipeline also needs retries, duplicate-event handling, concurrency/version guards and reconciliation after interruption.

Adapting to real models and stores

Retain the same lifecycle with a trusted local SentenceTransformer checkpoint and its tokenizer. Determine its actual maximum input length, dimension, normalization and required text prefixes; do not rely only on the model name. Batch within a measured memory budget, cache by embedding-input hash and model identity, and retry only failed batches with bounded backoff. Dimension compatibility is necessary but not sufficient: two different 384-dimensional models do not share a semantic coordinate system.

For Chroma, the documented upsert behavior creates missing IDs and updates existing ones [4]. A shortened document produces new chunk IDs while the old IDs remain until explicitly deleted or excluded by an active-generation filter. Separate delete/update/upsert operations and query filtering must implement the lifecycle; do not call a multi-call update atomically incremental without a verified publication protocol. HNSW/IVF/flat search, filter selectivity and hybrid lexical+dense search affect different costs and should be tested against an exact baseline.

Practice: compare chunk sizes and lifecycle failures

The executable test checks unchanged re-ingestion, document shortening, document removal, metadata-only permission changes, embedding-version changes, cross-tenant isolation and rollback on an embedding failure. It also verifies source-slice membership, unique repeated-content IDs, no heading split inside a code fence and a retrieval miss/unauthorized-hit negative control. These are real software-contract results, not measured MiniLM recall.

For a corpus experiment, create separate versioned configurations for each tokenizer-aware size/overlap combination. Retain the same question/evidence labels, compare containment and recall at equal context budget, then inspect failed examples. A storage count alone cannot prove freshness, relevance or deletion correctness.

(d) Production reality · 15 min

Wrong quarter or edition: inspect source revision, title and effective-date filters before assuming the embedding model is broken. Breadcrumbs may help disambiguation but do not guarantee it. Do not attribute an unsupported 5–15 point gain to Notion.

Stale legal/medical material: distinguish superseded, future-effective and currently effective documents. A freshness score cannot replace an authoritative version constraint. Preserve who established the policy interval and verify the cited clause, not just the document title.

Code retrieval: parse language structure where possible and retain imports/enclosing definitions. Large functions require bounded subunits with context; malformed code needs an explicit fallback. Do not claim every code-RAG product uses the same Tree-sitter implementation.

Long context: it can reduce fragmentation for some tasks but still costs compute and context budget, and cannot ingest an arbitrarily large changing corpus in every request. Retrieval also supplies access filtering, freshness and inspectable evidence. Long-context quality is model/task dependent; a universal “every model loses the middle” claim or undated price/latency range is not evidence.

Where this shows up next

Retrieval/reranking consumes the active index; vector-index choices trade ingestion, filtering, memory and recall. Agents may retrieve during a conversation, requiring the same authorization and citation checks. Streaming updates need clear freshness/visibility semantics. Cost depends on parsed size, overlap, embedding batches, rebuild frequency and returned context—not just source file count.

(e) Recall + stretch · 10 min

1. Default size/overlap? There is no universal 512/15% optimum. Start with a model-compatible baseline, count actual tokenizer input including prefixes and tune on labeled evidence.
2. Mid-sentence splits? They can lose referents or conditions, but not every partial sentence is meaningless. Measure intact-span/context sufficiency and use structure/overlap/expansion appropriately.
3. Four strategies? Fixed token windows for a baseline; recursive separators for prose; structural parsing for documents/code; semantic change detection for unstructured topical transitions.
4. Breadcrumb prefix? It can add context to the embedding input. Keep it separate from cited source text, count its tokens and evaluate its effect.
5. Metadata? Source identity/revision/range, structure, effective/update times, tenant/access policy, content type, index position and parser/tokenizer/embedding versions.
6. Fifty new pages? Reconcile changes against a successful source manifest, embed only new/changed inputs, update metadata and remove obsolete active chunks. Deterministic IDs alone do not skip embedding or delete stale data.
7. Large overlap? It raises redundancy and cost; the long-document multiplier is 1/(1−o/s). Gains are empirical, not guaranteed negligible after 20%.
8. Outdated citations? Inspect source authority, effective intervals, active generation and cache invalidation, then ranking. Newest timestamp is not always the valid policy.
9. Python files? Use an actual syntax parser where available or clearly label a separator heuristic. Preserve enclosing context and handle oversized/malformed units explicitly.
10. Does long context kill RAG? No universal answer; corpus scale, freshness, authorization, evidence selection, quality and complete request cost still matter. They often complement each other.

Explain-out-loud test

Explain why MiniLM's 256-wordpiece default invalidates the old 512-token assumption, derive overlap cost, distinguish upsert from reconciliation and prove that a shortened document leaves no stale active chunks in the lab.

What comes next

Retrieval and reranking builds on these index contracts. Learning-plan hub.

Primary references

1. Lewis et al., Retrieval-Augmented Generation, 2020.
2. RecursiveCharacterTextSplitter, langchain-text-splitters 0.3.11 source.
3. all-MiniLM-L6-v2 model card, dated snapshot.
4. Chroma updating data, dated documentation snapshot.
5. OWASP LLM01 Prompt Injection.