RAG End-to-End — Chunking, Embeddings, Retrieval, Rerank, Eval

Why this session matters

Part of a 24-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.

Part 1: RAG Part 1 — Why, Chunking, Embeddings, Vector Stores

Why this session matters

It builds on the rhythm of one focused topic, paced so you have time to actually absorb it rather than rush.

Agenda

• Why RAG exists — the limits of context windows and fine-tuning
• Chunking strategies — fixed, sentence, semantic, structural
• Embedding overview — dense vectors, similarity, what 'good' looks like
• Vector stores — Faiss, pgvector, Pinecone, Weaviate, Milvus
• What we keep for Part 2 (retrieval ranking, re-rank, eval)

Pre-read (skim before the session)

• Greg Kamradt — 5 Levels Of Text Splitting (video)
• Lewis et al. 2020 — Retrieval-Augmented Generation paper
• LangChain RAG concepts
• OpenAI cookbook — RAG

Deep dive

1. The problem RAG solves

LLMs are great at language but they don't know your private docs, last week's PRs, or this quarter's incidents. Two options:

• Fine-tune them on your data — expensive, slow, freezes the knowledge.
• Retrieve relevant snippets at query time and stuff them into the prompt — cheap, fresh, debuggable.

RAG = retrieve relevant context, then generate. It's now the default architecture for product chatbots, copilots, search, anything that needs to ground a model in your data.

2. The simplest RAG pipeline

     Ingestion (offline)                  Query (online)
     -------------------                  --------------
   docs → chunk → embed → store      q → embed → top-k search →
                       │                                       │
                       ▼                                       ▼
               vector store                     prompt = template(q, top-k) → LLM → answer

Three decisions dominate the quality you ship:

1. How you chunk the documents.
2. Which embedding model you use.
3. How you retrieve (top-k? hybrid? re-rank?).

This session covers 1 and 2 deeply, 3 lightly. the next session (RAG Part 2) covers retrieval, re-ranking, generation, and evaluation in depth.

3. Chunking — the most underrated decision

The model can only retrieve what fits in a chunk. Chunks too big → noisy retrieval. Too small → lose context.

Five levels (Greg Kamradt's framing):

1. Fixed-size — every N characters / tokens. Simple, lossy on structure.
2. Recursive character splitting — split on \n\n, then \n, then space, then char. Respects paragraph + sentence boundaries.
3. Semantic chunking — embed sentences, group adjacent sentences when embedding similarity > threshold. Computational but quality bump.
4. Structural chunking — use the doc structure (Markdown headers, HTML divs, code blocks). Best for technical docs.
5. Agentic chunking — ask an LLM to chunk it. Expensive at ingest, sometimes worth it.

Defaults that work:

• Chunk size: 300–800 tokens (model-dependent).
• Overlap: 50–100 tokens so cross-chunk references aren't lost.
• Metadata per chunk: (doc_id, chunk_idx, headings[], url, page_no, last_modified).

4. Embeddings (overview — deep dive in the next session)

An embedding model maps text → a dense vector in R^d (typically d = 384, 768, 1024, 1536, 3072). Two texts with similar meaning map to nearby vectors (cosine similarity high).

For English RAG in 2026:

Multi-lingual: bge-m3, multilingual-e5-large. For code: voyage-code-2, bge-code.

5. Vector stores — picking the right one

Rule of thumb: until ~10M chunks, pgvector (or Faiss in-process) is enough. Past 10M, look at Qdrant / Weaviate / Pinecone for ANN performance + multi-tenant features.

6. ANN — approximate nearest-neighbour

Exact NN over 1B vectors is too slow. ANN trades a tiny recall hit for huge speed:

• HNSW — hierarchical small-world graph. The default everywhere (Faiss, Qdrant, pgvector). 90–99% recall at <10 ms for millions of vectors.
• IVF — cluster + inverted file. Better for billion-scale; slower to update.
• ScaNN (Google), DiskANN (Microsoft) — SOTA for very large indices.

For most product use cases: HNSW with M = 16, ef_construction = 200, ef_search = 50 is a strong default.

7. A minimal RAG in 40 lines

import openai, numpy as np, faiss

docs = [open(p).read() for p in document_paths]
chunks = [c for d in docs for c in split_recursive(d, size=500, overlap=80)]

def embed(texts):
    r = openai.embeddings.create(model="text-embedding-3-small", input=texts)
    return np.array([e.embedding for e in r.data], dtype="float32")

emb = embed(chunks)
index = faiss.IndexFlatIP(emb.shape[1])  # inner-product = cosine on L2-normalised
faiss.normalize_L2(emb); index.add(emb)

def answer(q, k=4):
    qv = embed([q]); faiss.normalize_L2(qv)
    D, I = index.search(qv, k)
    ctx = "\n\n".join(chunks[i] for i in I[0])
    prompt = f"Context:\n{ctx}\n\nQuestion: {q}\nAnswer:"
    r = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return r.choices[0].message.content

It works. It's also terrible — no re-ranking, no hybrid, no eval. Sessions 13 and 25 fix that.

8. Metadata filtering

Real RAG is rarely 'search all docs'. Filter to the user's org / language / doc-type before ANN:

index.search_with_filter(qv, k=4, filter={"org_id": user.org, "lang": "en"})

Qdrant and Weaviate make this a first-class feature; pgvector does it via SQL WHERE. Pre-filter is faster than post-filter because ANN scoring only happens on the filtered subset.

9. Common failure modes (RAG “doesn't work”)

• Retrieves the right doc, wrong section → chunking is too coarse / overlap too small.
• Doesn't retrieve the right doc at all → embedding model isn't strong enough for your domain; try a domain-specific one or fine-tune.
• Top-k irrelevant → add a re-ranker (cross-encoder) over top-30 → top-5. the next session.
• Answer cites wrong source → prompt isn't telling the model to cite. Add: "Cite the source for every claim using [doc_id]".
• Hallucinated answer even with context → add an evaluation step (the next session — LLM evals).

10. What's next (the next session — RAG Part 2)

• Retrieval depth: BM25 + vector hybrid, ColBERT late interaction
• Re-ranking: cross-encoders, LLM-as-reranker
• Generation prompts: structured, citation-bound
• Evaluation: faithfulness, answer relevancy, context precision (RAGAS)

Reading material

Books:

• Building LLMs for Production — Louis-François Bouchard, Louie Peters (covers chunking, retrieval, evaluation)
• Natural Language Processing with Transformers — Tunstall, von Werra, Wolf (ch. 7: question answering with retrievers)

Papers:

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020) — the original RAG paper.
• Dense Passage Retrieval for Open-Domain QA (Karpukhin et al., 2020) — DPR, the bi-encoder retrieval baseline.
• Sentence-BERT (Reimers & Gurevych, 2019) — efficient sentence embeddings via siamese BERT.
• REALM: Retrieval-Augmented Language Model Pre-Training (Guu et al., 2020) — Google's precursor to RAG.

Official docs:

• LangChain — Document loaders & text splitters
• LlamaIndex — Chunking docs
• Pinecone — Vector indexing
• pgvector — Postgres vector extension

Blog posts:

• The 5 Levels of Text Splitting for Retrieval — Greg Kamradt — the canonical chunking deep dive.
• Anyscale — RAG done right
• Pinecone — Chunking strategies

In-depth research material

• haystack — github.com/deepset-ai/haystack — ~16k ★, production RAG pipelines.
• llama_index — github.com/run-llama/llama_index — ~38k ★, the RAG framework that started the chunker wars.
• langchain — github.com/langchain-ai/langchain — ~95k ★.
• ColBERT (Khattab & Zaharia, 2020) and ColBERTv2 — late-interaction retrieval that beats DPR.
• Stanford CS25 — Retrieval-Augmented Language Models lecture — guest lectures incl. Doug Downey & Patrick Lewis.
• Latent Space podcast — RAG episode with Vespa & LangChain founders — the production lens.

Videos

• What is Retrieval-Augmented Generation (RAG)? — IBM Technology · 7 min — the cleanest 7-minute primer; great opener.
• RAG Explained in 12 Minutes — Aishwarya Srinivasan · 12 min — punchy, modern explanation of why retrievers > fine-tuning for knowledge.
• Chunking Strategies in RAG: Optimising Data — Mervin Praison · 14 min — the five chunking approaches with concrete code.
• RAG Explained | All about RAG — Retrieval Augmented Generation — codebasics · 15 min — Indian-tech-content quality at its best; covers the full pipeline.
• Vector Embeddings Tutorial — Code Your Own AI Assistant — freeCodeCamp · 36 min — the hands-on lab; build embeddings + index + query end-to-end.

LeetCode — Design Search Autocomplete System

• Link: https://leetcode.com/problems/design-search-autocomplete-system/
• Difficulty: Hard
• Why this problem: Trie of sentences with frequency at terminal nodes; sort matches by (freq desc, lex asc).
• Time-box: 30 minutes. Look up the editorial only after.

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

• State two reasons RAG beats fine-tuning for fresh / private data.
• Pick a chunking strategy for a markdown technical doc.
• Choose an embedding model for English product docs and defend it.
• Compare Faiss vs pgvector vs Pinecone for your scale.
• Build the 40-line RAG above and improve one number (recall@5).
• Solve design-search-autocomplete-system (Trie + frequency).

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.

Part 2: RAG Part 2 — Retrieval, Re-Ranking, Generation, Evaluation

Why this session matters

It builds on the rhythm of one focused topic, paced so you have time to actually absorb it rather than rush.

Agenda

• Retrieval — dense vs sparse vs hybrid; BM25 + embeddings together
• Re-ranking — cross-encoders, ColBERT, LLM-as-reranker
• Generation — context construction, system prompt, citation grounding
• Evaluation — RAGAS, recall@k, faithfulness, answer relevance
• Production patterns — multi-hop, hypothetical document embeddings (HyDE), routing

Pre-read (skim before the session)

• Retrieval-Augmented Generation (Lewis et al., 2020)
• RAGAS — Automated Evaluation of RAG
• Pinecone — Hybrid Search
• LlamaIndex — Re-ranking Guide

Deep dive

1. The full RAG pipeline

the next session covered ingestion: chunking, embedding, vector store. Now we cover what happens at query time.

user query
   │
   ▼
[query rewrite/expand]  ← optional, big win for short queries
   │
   ▼
[retrieval]            ← dense + sparse, k≈50
   │
   ▼
[re-rank]              ← cross-encoder, k≈5
   │
   ▼
[context construction] ← order, dedupe, fit in token budget
   │
   ▼
[generate]             ← LLM with grounded prompt
   │
   ▼
[post-process]         ← citations, hallucination check

Each stage is independently improvable. Most teams optimise retrieval and never touch the rest — you leave 30% recall on the table.

2. Dense vs sparse vs hybrid

Dense (embeddings + cosine): great for semantic matches. "How do I cancel?" finds "refund policy" even if the words don't overlap.

Sparse (BM25): great for lexical matches. Acronyms, model names, exact phrases. BM25 still wins on entity-heavy queries — it's not obsolete.

Hybrid: run both, fuse the scores. Two common fusion methods:

# Reciprocal Rank Fusion (RRF) — k=60 is the canonical constant
def rrf(dense_ranks, sparse_ranks, k=60):
    scores = {}
    for rank, doc_id in enumerate(dense_ranks):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    for rank, doc_id in enumerate(sparse_ranks):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda x: -x[1])

RRF is parameter-light and robust. It almost always beats either method alone — I've seen +10-15 points of recall@10 vs dense-only on internal benchmarks.

3. Re-ranking — bi-encoder vs cross-encoder

The retrieval embedding is a bi-encoder: query and doc encoded separately. Fast (one query embed + ANN search) but loses interaction signal.

A cross-encoder takes [CLS] query [SEP] doc [SEP] jointly and outputs a relevance score. Way more accurate, way slower. Solution: use bi-encoder to fetch top-50, cross-encoder to re-rank to top-5.

Typical stack:

• Retrieval: text-embedding-3-large or bge-large-en-v1.5 → top-50.
• Re-rank: bge-reranker-large or cohere/rerank-3 → top-5.
• Latency: ~20 ms retrieval + ~80 ms rerank @ 50 docs.

4. ColBERT — the middle ground

ColBERT stores per-token embeddings instead of one vector per doc. At query time, MaxSim scores each query token against the doc's best matching token. Quality near a cross-encoder, latency near a bi-encoder. Storage cost is ~30× a single-vector index — usually only worth it for high-recall, low-volume corpora.

5. Context construction

Once you have 5 chunks, how do you present them?

• Order matters. Models attend more to the start and end of the context (lost-in-the-middle, Liu et al. 2023). Put the most relevant chunk first or last, not in the middle.
• Dedupe. Chunks that share >70% overlap waste budget. Hash-based or embedding-clustering dedupe.
• Header each chunk with metadata: [Source: docs/billing.md, last_modified: 2026-04-12]. Cheap, often free recall.
• Budget. Typical prompt: 4K system + 8K retrieved + 1K query + room for answer. Don't blow the budget — pay the latency for shorter prompts.

6. Grounded prompt template

You are a support assistant. Answer ONLY from the context below.
If the answer isn't in the context, say "I don't have that information."
Cite sources inline as [^1], [^2], …

Context:
[^1] {chunk_1.metadata} → {chunk_1.text}
[^2] {chunk_2.metadata} → {chunk_2.text}
…

Question: {user_query}

The "answer ONLY from the context" + "say I don't know" instructions are non-negotiable. Without them you get confident hallucinations.

7. Evaluation — RAGAS metrics

RAGAS scores a RAG system on four axes, computed by an LLM-as-judge:

You need a small (100–500) labelled eval set: (question, ground-truth answer, optional reference docs). RAGAS uses the LLM to synthesise some of these — start with synthetic, validate on a hand-curated subset.

8. HyDE — Hypothetical Document Embeddings

Trick from Gao et al. 2022: have an LLM write a hypothetical answer to the query, embed that, and search. Why? The query is often short and lexically far from the doc; the synthetic answer is doc-shaped.

def hyde_search(query, llm, index):
    hypo_answer = llm.complete(f"Write a paragraph that answers: {query}")
    return index.search(embed(hypo_answer), k=10)

Costs one extra LLM call per query — worth it for short, ambiguous queries.

9. Multi-hop and query routing

For "What did the CEO say about Q3 revenue last earnings call?":

1. Decompose → ["who is the CEO?", "when was last earnings call?", "what did <CEO> say about Q3?"]
2. Route → org-chart index, calendar index, transcripts index.
3. Synthesise → final answer references all three.

LangGraph / LlamaIndex agents do this; or you can hand-code a 50-line orchestrator. The simpler the better — every hop is a chance for the LLM to drift.

10. Production numbers (from a real customer support RAG)

Quality (human-rated): 87% useful answers, 9% partial, 4% wrong. Most wins came from re-ranking (+8%) and HyDE on short queries (+5%); BM25 hybrid gave +12% recall but only +3% answer quality (re-rank caught up the rest).

Reading material

Books:

• Building LLMs for Production — Bouchard, Peters (the evaluation chapters)
• Generative Deep Learning, 2nd ed. — David Foster (RAG framing in ch. 12)

Papers:

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) — the paper that justifies re-ranking and short context windows.
• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023) — the RAGAS framework.
• ColBERT (Khattab & Zaharia, 2020) — late-interaction retrieval; a re-ranker that beats cross-encoders.
• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (Thakur et al., 2021) — the standard retrieval benchmark.
• Self-RAG (Asai et al., 2023) — adaptive retrieval + self-critique.

Official docs:

• Cohere Rerank API
• Voyage AI — Embedding & Rerank models
• LangChain — Retrievers
• Ragas — Documentation

Blog posts:

• Pinecone — Re-ranking guide
• Anthropic — Contextual Retrieval — 67% reduction in failed retrievals; required reading.
• LlamaIndex — Advanced retrieval techniques

In-depth research material

• ragas — github.com/explodinggradients/ragas — ~10k ★, the RAGAS framework.
• trulens — github.com/truera/trulens — ~3k ★, RAG eval + tracing.
• BEIR benchmark — github.com/beir-cellar/beir — heterogeneous IR benchmark.
• The Pragmatic Engineer — RAG in production case study — long-form lessons.
• Latent Space podcast — RAG arc (multiple episodes) — the most current production-RAG conversation on the internet.
• Stanford CS336 — Language Modeling from Scratch — includes a retrieval lecture.

Videos

• RAG Reranking Explained: How To Improve RAG Results — pixegami · 14 min — the clearest re-ranking walkthrough; cross-encoder vs bi-encoder demystified.
• Rerank for better RAG (Explained) — vectorize · 14 min — same topic, different angle; pair with the previous video.
• Intro to Cascading Retrieval — Pinecone — Pinecone · 49 min — multi-stage retrieval (BM25 → dense → rerank) with up to 48% precision boost.
• RAGAS: How to Evaluate a RAG Application — Mervin Praison · 9 min — the canonical RAGAS intro; the four metrics + sample notebook.
• Do Reranking Models Actually Improve RAG? — Adam Lucek · 32 min — measured experiments showing when re-rankers help (and when they hurt); the empirical take.

LeetCode — Merge K Sorted Lists

• Link: https://leetcode.com/problems/merge-k-sorted-lists/
• Difficulty: Hard
• Why this problem: Min-heap of (val, list-idx); pop, push next from same list. Mirrors merging top-k from N retrievers.
• Time-box: 30 minutes. Look up the editorial only after.

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

• Explain dense vs sparse vs hybrid retrieval and when each wins.
• Implement Reciprocal Rank Fusion in 10 lines.
• Describe the bi-encoder vs cross-encoder trade-off; when ColBERT helps.
• List the 4 RAGAS metrics and what each catches.
• Apply HyDE and explain when it pays back its extra LLM call.
• Solve merge-k-sorted-lists — same heap-merge pattern as multi-retriever fusion.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.