S117 · RAG I — Chunking Strategies & Indexing

Module M14: LLMs & Applications · Session 117 of 130 · Track: LLM 📚

What you'll be able to do after this session

• Explain RAG I to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S106 — Tokenization — BPE, WordPiece, SentencePiece

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · What is RAG in 6 minutes — IBM Technology — Cleanest zero-jargon intro; use as your one-slide mental model.
• 🎥 Deep dive (30–60 min) · Building RAG applications — Jerry Liu (LlamaIndex CEO) — 60-min deep dive from the person building the reference framework.
• 🎥 Hands-on demo (10–20 min) · Build a RAG chatbot in 10 minutes — LangChain — Hands-on code walkthrough end-to-end.
• 📖 Canonical article · Retrieval-Augmented Generation for Knowledge-Intensive NLP — The original 2020 RAG paper (Lewis et al., Meta).
• 📖 Book chapter / tutorial · Pinecone — Chunking Strategies — The single best reference on how to split documents for retrieval.
• 📖 Engineering blog · Anthropic — Introducing Contextual Retrieval — Anthropic's 2024 engineering post: adding chunk context boosts recall by ~35%.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The '80% of LLM apps' pattern, part 1.

LLMs have a fixed context window (usually 4k–200k tokens) and their weights don't know anything that happened after their training cutoff. If you want them to answer questions using your documents (your company wiki, your PDFs, your Slack), you have two options: (1) fine-tune the model on your data (expensive, slow, hard to update) or (2) retrieve the relevant chunks at query time and stuff them into the prompt. Option 2 is called RAG (Retrieval-Augmented Generation), and it powers roughly 80% of shipped LLM apps in 2024.

RAG has two phases. Indexing (offline, one-time or nightly): take your documents, split them into chunks, embed each chunk into a vector, store the vectors in a database. Query (online, per user question): embed the question, find the most similar chunks by vector similarity, put those chunks into the prompt, ask the LLM to answer using only those chunks. This session is about the indexing half. Session 118 covers retrieval and reranking; 119 covers the vector databases themselves.

Gotcha to carry forever: garbage chunks in, garbage answers out. Retrieval quality is upper-bounded by chunk quality. If you split a table in half, or split a code function across two chunks, or embed a chunk that says "as mentioned above" with no context, the retriever will never recover. Chunking is not a preprocessing detail — it is the product. Most RAG projects that "don't work" have a chunking problem, not a model problem.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Chunking strategies — the four you must know.

Worked example — chunk this policy doc:

# Refund Policy (v2)
Refunds are available within 30 days of purchase for physical goods.
Digital goods are non-refundable except for double-charges.
Contact support@example.com to initiate a refund.

## Exceptions
- EU customers: 14-day cooling-off period per GDPR.
- Enterprise contracts: see your MSA.

Naive 100-token chunking might yield: chunk1 = "Refund Policy (v2) Refunds are available within 30 days of purchase for physical goods. Digital goods are non-refundable except for double-charges. Contact" and chunk2 = "support@example.com to initiate a refund. Exceptions EU customers: 14-day cooling-off period". The word "Exceptions" is now stranded — a user asking "Is there an exception for EU customers?" will retrieve chunk 2 but miss the parent context (refund policy).

Contextual chunking (Anthropic pattern): prepend "This chunk is from the Refund Policy v2 document, under 'Exceptions'. It clarifies rules for non-default customers." to chunk 2. Retrieval recall jumps ~35% on their internal benchmark. The cost: one small LLM call per chunk at index time (do it once, cache forever).

Overlap (~10–20%) exists so a sentence that lands on a chunk boundary appears fully in at least one chunk.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Index a small folder of Markdown files with LangChain-style chunking + OpenAI embeddings + FAISS. Runs on a laptop.

# pip install langchain-community langchain-openai faiss-cpu tiktoken
import os
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
 
os.environ.setdefault("OPENAI_API_KEY", "sk-...")
 
# 1. Load ~/notes as a corpus
loader = DirectoryLoader("./notes", glob="**/*.md",
                         loader_cls=TextLoader, show_progress=True)
docs = loader.load()
print(f"loaded {len(docs)} documents")
 
# 2. Recursive chunker: try \n\n, then \n, then '. ', then ' '
splitter = RecursiveCharacterTextSplitter(
    chunk_size=800, chunk_overlap=120,
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=len,
)
chunks = splitter.split_documents(docs)
print(f"split into {len(chunks)} chunks; sample sizes:",
      [len(c.page_content) for c in chunks[:5]])
 
# 3. Embed + index (uses text-embedding-3-small, 1536 dims, $0.02 / 1M tokens)
emb = OpenAIEmbeddings(model="text-embedding-3-small")
index = FAISS.from_documents(chunks, emb)
index.save_local("./rag_index")
print(f"indexed {index.index.ntotal} vectors, dim={index.index.d}")
 
# 4. Sanity-check the retrieval
for q in ["what is the refund window?",
          "how do we handle EU customers?",
          "who owns the deployment runbook?"]:
    hits = index.similarity_search_with_score(q, k=3)
    print(f"\nQ: {q}")
    for doc, score in hits:
        print(f"  [{score:.3f}] {doc.metadata.get('source','?')} :: "
              f"{doc.page_content[:100].strip()}...")

Checklist:

1. Chunk count is roughly total_chars / (chunk_size − overlap).
2. Sample chunk sizes should cluster near 800 — if many are ≪ 800, you have short files or aggressive separators.
3. index.index.d == 1536 for text-embedding-3-small.
4. Top-3 results for each query should include the semantically closest chunk, not always the exact keyword match.
5. Lower score = better match with FAISS L2 distance (0 = identical, larger = worse).

Try this: re-chunk with chunk_size=200, chunk_overlap=20 and rerun the queries. Notice how retrieval hits become more precise but often lose surrounding context. Then try chunk_size=2000, chunk_overlap=200 — precision drops, but you rarely lose context. Tuning this trade-off is Job #1 in any RAG rollout.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

The team meme is "just build a RAG on our docs." The reality is that going from prototype-RAG-that-demos-well to production-RAG-that-actually-answers is 3–6 months of unglamorous work — most of it on the indexing side you're building here.

War stories. (1) A team indexes 10k PDFs, users complain "it never finds the answer even though I know it's in there." Root cause: PDF parser silently strips tables (where the answer lives). Every RAG project needs a retrieval eval set — 100 real questions with known-good chunks, run nightly, alert on regression. (2) Same team fixes tables → users complain about hallucinations. Root cause: chunks include the header row twice (once as heading, once as table row), so the retriever returns duplicates that eat the context window. Deduplicate at index time by hashing content. (3) Chunk drift — the doc updates monthly, but the RAG index refreshes weekly, so users get 3-week-old policy answers alongside new ones. Store a last_updated timestamp on each chunk and either filter or re-embed on doc change. (4) "We embedded the whole 200-page contract as one chunk" — the LLM answers based on only the first few sentences the embedding was dominated by. Chunks over ~1500 tokens become semantic mush. (5) Multilingual retrieval fails silently — English embedding models rank Chinese chunks near random. Use a multilingual embedding (bge-m3, text-embedding-3-large, Cohere embed-multilingual-v3) if any of your corpus isn't English.

How top teams handle it. Anthropic's contextual retrieval (Sep 2024) recommends prepending 50–100 tokens of LLM-generated context per chunk before embedding — cuts retrieval failure by ~49% when combined with BM25 + reranker. Microsoft's Azure AI Search RAG guide pushes hybrid chunking (semantic + fixed) with rich metadata (title, section, source URL). Everyone at scale ends up with chunks + parent-doc links so retrieval can hit a small chunk and then hydrate the surrounding paragraph or full doc for the LLM. Nobody uses default 500/50 chunk settings in production forever — expect to tune per corpus.

(e) Quiz + exercise · 10 min

1. What are the two phases of a RAG system?
2. Why does chunk size matter — what breaks when chunks are too small? Too large?
3. What is chunk overlap for?
4. Name three chunking strategies from cheapest to most sophisticated.
5. What is "contextual retrieval" (Anthropic 2024) and roughly how much does it help recall?

Stretch (connects to S106 — Word Embeddings): Chunks are embedded into a vector space to enable similarity search. Given what you know about how embeddings encode meaning, why do embedding models struggle to retrieve chunks that contain only keywords (e.g. tables of SKUs, part numbers) rather than natural language? How would you handle a corpus of product tables?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is RAG I? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S118): RAG II — Retrieval, Hybrid Search, Reranking
• Previous (S116): Prompting — Zero-Shot, Few-Shot, Chain-of-Thought, ReAct
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.