DL S076 · Retrieval-Augmented Generation (RAG) from Scratch

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 76 / 80 · Module M12 · ~2 hours

The story we're starting with

Your LLM's context window is 8k tokens. Your company has 50 million tokens of internal documentation. You want the model to answer questions about that documentation without lying. What do you do?

The obvious answer — "fine-tune the model on it" — is a trap. Fine-tuning teaches the model style and format; it's a terrible way to teach it facts. Facts change. Fine-tuning is expensive. And the moment the docs update, your fine-tune is stale. Retrieval-Augmented Generation is the practical answer: at query time, find the 3–5 most relevant paragraphs, paste them into the prompt, and ask the model to answer using them. The model becomes an interpreter over ephemeral evidence, not a knowledge base.

That sentence hides an entire subfield. Every word — "find", "most relevant", "paragraph" — is a decision that costs you accuracy if you make it badly. Today we build the full pipeline from scratch: how to chunk documents (harder than it sounds), how to embed them (S055 sentence-transformer territory), how to build a vector index (FAISS internals for real), how to retrieve, how to rerank, and how to stuff the results into a prompt without confusing the model. By the end you'll be able to look at any RAG framework and know exactly what each of its 40 config knobs does — and, more importantly, when to ignore all of them and write 200 lines of your own.

Prerequisites

• S055 · Sentence embeddings. We're standing on top of that session — you need to understand what a sentence encoder produces.
• S040 · Transformer generation. RAG's generator is the same model.generate() you built earlier.
• S066 · Prompt engineering. The prompt template is half of RAG quality.
• Familiarity with cosine similarity — you already have this from CLIP in S074.

1 · The whole pipeline in one diagram

Six real components. Every framework wraps these in a config-heavy DSL. We're going to write them by hand.

2 · Chunking — the part that decides your ceiling

You have a 200-page PDF. You need to split it into passages the retriever can index. This is the highest-leverage decision in the pipeline. Chunk it wrong and no amount of fancy retrieval fixes the ceiling.

2.1 · Three chunking strategies

Fixed-size — split every N characters or N tokens. Fast, dumb, sometimes fine.

def chunk_fixed(text: str, size: int = 500, overlap: int = 50):
    step = size - overlap
    return [text[i:i+size] for i in range(0, len(text), step)]

The overlap lets a semantic unit that got cut mid-sentence appear (at least partially) in two chunks. Rule of thumb: 10–15% overlap.

Recursive — split by paragraph first, then by sentence if paragraph is too long, then by character as last resort. This is what LangChain's RecursiveCharacterTextSplitter does. Better on structured docs.

SEPARATORS = ["\n\n", "\n", ". ", " ", ""]
 
def chunk_recursive(text: str, size: int = 500, seps=SEPARATORS):
    if len(text) <= size:
        return [text]
    for sep in seps:
        if sep in text:
            parts = text.split(sep)
            chunks, buf = [], ""
            for p in parts:
                if len(buf) + len(sep) + len(p) <= size:
                    buf += (sep if buf else "") + p
                else:
                    if buf: chunks.append(buf)
                    if len(p) > size:
                        chunks.extend(chunk_recursive(p, size, seps[seps.index(sep)+1:]))
                        buf = ""
                    else:
                        buf = p
            if buf: chunks.append(buf)
            return chunks
    return [text[i:i+size] for i in range(0, len(text), size)]

Uglier, but preserves natural boundaries. Use for docs with clear structure (Markdown, HTML, technical books).

Semantic — embed each sentence, cluster consecutive similar sentences into a chunk, cut when similarity drops. Expensive to build, best quality on prose (novels, long-form articles). Usually overkill.

2.2 · The chunk-size sweet spot

Too small (100 tokens): each chunk lacks context. "It was destroyed in 1789" is useless without the "Bastille" reference three sentences up. Retrieval will surface it as a match for "1789 events" but the model can't answer.

Too big (2000 tokens): retrieval becomes noisy. A chunk about "Napoleon's early career" also happens to mention Josephine in one sentence — now queries about Josephine surface this whole irrelevant chunk. Also: fewer chunks means less discrimination in the vector space; embeddings are averages of many topics.

500 is the sweet spot for most English prose. Code and structured data want different sizes (see §7).

3 · Embedding

Grab a sentence-transformer. all-MiniLM-L6-v2 is 22M params, fast, and produces 384-d vectors. Good enough baseline; upgrade to bge-large or mxbai-embed-large when you need +5 points of retrieval accuracy.

from sentence_transformers import SentenceTransformer
import numpy as np
 
embedder = SentenceTransformer("all-MiniLM-L6-v2")
 
def embed(texts: list[str]) -> np.ndarray:
    return embedder.encode(texts, normalize_embeddings=True)  # (N, 384)

normalize_embeddings=True L2-normalizes; then cosine similarity is just dot product. Every trick from CLIP applies.

4 · The vector index — no FAISS

Everyone's first RAG is "install FAISS, don't understand it." Let's do it in NumPy so you understand what FAISS is optimizing.

4.1 · Brute-force cosine search

class SimpleIndex:
    def __init__(self):
        self.vectors: np.ndarray | None = None
        self.metadata: list[dict] = []
 
    def add(self, vecs: np.ndarray, meta: list[dict]):
        assert vecs.shape[0] == len(meta)
        self.vectors = vecs if self.vectors is None else np.vstack([self.vectors, vecs])
        self.metadata.extend(meta)
 
    def search(self, query_vec: np.ndarray, k: int = 20) -> list[tuple[float, dict]]:
        sims = self.vectors @ query_vec           # (N,)
        top_idx = np.argpartition(-sims, k)[:k]
        top_idx = top_idx[np.argsort(-sims[top_idx])]
        return [(float(sims[i]), self.metadata[i]) for i in top_idx]

That's a vector database. Twelve lines. It's O(N) per query, which is fine for up to ~100k chunks on a laptop.

4.2 · Why FAISS exists

For 10M+ chunks the brute-force scan becomes slow. FAISS gives you:

• IVF (Inverted File Index) — partition vectors into K clusters (via k-means), search only the closest few clusters at query time. ~10-50× speedup, ~1% recall loss.
• PQ (Product Quantization) — compress vectors to 8 bytes each using learned subspace clustering. 30× memory reduction, ~2-3% recall loss.
• HNSW — a graph-based index with logarithmic search. Best latency, higher memory.

You don't need any of this for < 1M chunks. Start with brute force. Add FAISS when you measure the bottleneck.

4.3 · Numeric example

Say you index 4 chunks, each with 3-d embeddings (unrealistic, but shows the arithmetic):

Notice retrieval surfaces the Eiffel chunk too — it's topically close even though it doesn't answer the query. That's why we rerank.

5 · Reranking

Retrieval gives you top-20 based on bi-encoder similarity (query and doc encoded separately, then compared). That's fast but coarse. Now you feed the top-20 into a cross-encoder that jointly encodes (query, doc) and predicts a relevance score. Cross-encoders are ~100× slower per pair but far more accurate.

from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
 
def rerank(query: str, candidates: list[str], top_n: int = 4):
    scores = reranker.predict([(query, c) for c in candidates])
    order = np.argsort(-scores)[:top_n]
    return [(float(scores[i]), candidates[i]) for i in order]

Two-stage retrieval (retrieve 20, rerank to 4) beats single-stage retrieve-top-4 by 8–15 nDCG points on BEIR benchmarks. This is the single highest-ROI upgrade in most RAG pipelines. People skip it because it adds 300ms; add it anyway.

6 · Generation — the prompt template

PROMPT = """You are a helpful assistant. Answer the question using ONLY the context provided.
If the answer is not in the context, say "I don't know based on the provided context."
Cite the chunk IDs you used in square brackets, like [chunk-3].
 
Context:
{context}
 
Question: {question}
 
Answer:"""
 
def build_context(chunks: list[tuple[str, str]]) -> str:
    # chunks: list of (chunk_id, text)
    return "\n\n".join(f"[{cid}]\n{text}" for cid, text in chunks)
 
def answer(question: str, index, llm_call):
    q_vec = embed([question])[0]
    hits = index.search(q_vec, k=20)
    reranked = rerank(question, [h[1]["text"] for h in hits], top_n=4)
    context_chunks = [(f"chunk-{i}", text) for i, (_, text) in enumerate(reranked)]
    prompt = PROMPT.format(context=build_context(context_chunks), question=question)
    return llm_call(prompt)

Two design choices worth calling out:

1. "Answer using ONLY the context" + "If not present, say I don't know" are the two sentences that dramatically reduce hallucination. Test both — remove either and watch the model start improvising.
2. Chunk IDs in brackets create a citation surface you can validate. Post-process: if the model cites [chunk-5] and there is no chunk-5, flag it.

7 · War stories

8 · When RAG isn't the answer

• Static, small knowledge base (< 20k tokens): just put it all in the system prompt. No retrieval needed.
• Highly structured queries ("what is the price of SKU 12345?"): use SQL, not RAG. Text embeddings are bad at exact-match numeric lookup.
• Truly novel reasoning ("what would our revenue be if we lowered prices 15%?"): RAG can't help. You need an agent with a calculator (see S075).

9 · The 2024–2025 RAG landscape — hybrid, graph, and long-context

The pipeline you just built is the canonical baseline. Here's what practitioners layered on top over the past 18 months, and what the research says about each.

9.1 · Hybrid retrieval (BM25 + dense)

Dense embeddings are great at semantics ("the CEO resigned" ≈ "the chief executive stepped down"). They're terrible at rare tokens: model numbers ("IX-2350-B"), people's names, novel acronyms. BM25 — a 30-year-old sparse lexical retriever — nails those cases because it just matches tokens.

The modern default: run both retrievers, reciprocal-rank-fuse (RRF) the results, then rerank. On BEIR, hybrid RRF beats either alone by 3–8 nDCG@10 across most datasets (Bruch et al., 2023, arXiv:2210.11934). Implementation is 20 lines:

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)
 
# usage
fused = rrf([bm25_hits, dense_hits])[:20]
reranked = rerank(query, fused)[:4]

If you take one thing from this section: add BM25 to your dense RAG this week. It's a free 5-point improvement.

9.2 · GraphRAG (Microsoft, Feb 2024 → GA 2025)

Edge et al. (arXiv:2404.16130) noticed that standard RAG fails on global questions: "What are the main themes across this 500-page corpus?" No single chunk contains the answer. GraphRAG first uses an LLM to extract an entity+relationship graph from the corpus, clusters the graph hierarchically (Leiden algorithm), and generates community summaries at each level. At query time it routes local questions to chunk retrieval and global questions to community summaries.

On their MultiHop-RAG-style benchmark, GraphRAG hit ~72% correctness vs ~50% for vanilla RAG on multi-doc synthesis queries. Cost: the indexing pass runs the LLM over the entire corpus once, so it's ~10–100× more expensive to build. Worth it for high-value corpora (legal, medical, intelligence briefs). Overkill for a FAQ bot.

9.3 · Late interaction (ColBERT / ColBERTv2 / PLAID)

Bi-encoders compress a document into one vector — you lose per-token info. Cross-encoders keep everything but are 10,000× slower. ColBERT (Khattab & Zaharia, 2020; v2 2022) is the middle ground: keep one vector per token, score with maxₑsim(qᵢ, dⱼ) summed over query tokens. Retrieval quality within 1–2 points of cross-encoders, 100× faster. PLAID (Santhanam et al., 2022) adds efficient indexing. If your bi-encoder recall is your bottleneck, ColBERTv2 is the next thing to try before reranker tuning.

9.4 · Modern rerankers (Cohere Rerank v3, bge-reranker-v2-m3, Jina-reranker-v2)

MS-MARCO cross-encoders from 2020 are still fine, but 2024–2025 open-source rerankers train on much more diverse data:

• bge-reranker-v2-m3 (BAAI, 2024) — multilingual, handles 8k context per (q,d) pair.
• Jina-reranker-v2-base-multilingual (2024) — open weights, fast.
• Cohere Rerank v3.5 (2024) — API only but often the quality ceiling.

Swap your reranker before you tune your embedder — the delta is often bigger for less work.

9.5 · Long-context alternative ("just stuff everything")

Gemini 1.5 Pro (2M tokens), Claude 3.5 Sonnet (200k), Llama 4 Scout (10M) — with context windows this large, why chunk at all? Real answer: you still chunk-and-retrieve for cost and latency. A 2M-token forward pass costs ~$20 and takes 60 seconds. RAG stays cheap and fast. But long context has replaced RAG in two settings: (1) one-off queries over a single moderate document ("here's a 300-page contract, answer this question") and (2) needle-in-haystack evals where you know the whole doc fits.

Liu et al. ("Lost in the Middle," 2023, arXiv:2307.03172) showed accuracy dips 30–40 points when the relevant chunk sits in the middle of a long context. RULER (Hsieh et al., 2024, arXiv:2404.06654) and Needle-in-a-Haystack-Plus evaluations confirm this hasn't gone away in 2025 despite marketing. The lesson: even with a 2M-token model, if you can shrink the prompt to just the 4 relevant chunks, do it.

9.6 · Query rewriting and HyDE

Users ask questions the way they think; documents are written the way authors thought. Two closers:

• Query decomposition: LLM splits a complex question into 2–4 sub-questions, retrieves for each, then answers with the union. Standard for multi-hop.
• HyDE (Hypothetical Document Embeddings, Gao et al., arXiv:2212.10496): ask the LLM to write a fake answer first, embed that fake answer, retrieve using it. The fake answer is closer in vector space to the real one than the raw question is. +5 to +12 recall on many domains, ~free compute.

9.7 · Further reading

• BEIR benchmark: https://arxiv.org/abs/2104.08663
• RRF hybrid retrieval: https://arxiv.org/abs/2210.11934
• GraphRAG: https://arxiv.org/abs/2404.16130 · https://github.com/microsoft/graphrag
• ColBERTv2: https://arxiv.org/abs/2112.01488
• Lost in the Middle: https://arxiv.org/abs/2307.03172
• RULER long-context eval: https://arxiv.org/abs/2404.06654
• HyDE: https://arxiv.org/abs/2212.10496
• bge-m3 (BAAI): https://arxiv.org/abs/2402.03216
• Ragas evaluation framework: https://arxiv.org/abs/2309.15217

10 · Try it yourself

One week of RAG mastery in three evenings:

1. Evening 1: Build the 6-component pipeline over your own notes (Obsidian, Notion export, ~5 MB). Pick 20 questions you know the answer to. Measure top-4 recall + end-to-end accuracy. This is your baseline.
2. Evening 2: Add BM25 hybrid + RRF. Re-measure. Add HyDE. Re-measure. Which delta was bigger?
3. Evening 3: Swap the reranker to bge-reranker-v2-m3. Try GraphRAG on the same corpus (Microsoft repo is one pip install). Compare on 5 global questions. Write up the numbers on your blog.

Budget: 3 evenings. Reward: you can now debug any RAG in production because you've felt every knob.

11 · Recap

RAG is six components in a pipeline: chunker → embedder → vector index → retriever → reranker → generator. Every 2024–2025 improvement is either (a) making one of those six smarter (bge-m3 embedder, bge-reranker-v2, ColBERT late interaction) or (b) adding a pre/post step (BM25 hybrid, GraphRAG communities, HyDE query rewriting, citation validation). The failure modes never change: bad chunks, bad embeddings for your domain, lost-in-the-middle, hallucinated citations. Fix them in that order.

🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: re-read §1 (pipeline) + §7 (war stories) in 24 hours. Revisit the full session on day 7 alongside the GraphRAG paper.

Next session (S077): RAG lets one model act as if it knows more. What if one architecture contains many specialists, each cheap to activate? That's Mixture of Experts — Mixtral, DeepSeek-V3 (256 experts, MLA), and the arithmetic that makes 671B parameters cost like a 37B dense model.

Sticky note (keep on your desk): RAG = chunk → embed → index → retrieve → rerank → generate. Everything else is a knob on one of those six.

Previous: ← DL S075 · Next: DL S077 → Mixture of Experts