S118 · RAG II — Retrieval, Hybrid Search, Reranking

Module M14: LLMs & Applications · Session 118 of 130 · Track: LLM 🔎

What you'll be able to do after this session

• Explain RAG II to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S117 — RAG I — Chunking Strategies & Indexing

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Hybrid search explained — Weaviate (5 min) — Fast intuition: BM25 vs dense vs hybrid, one whiteboard.
• 🎥 Deep dive (30–60 min) · Advanced RAG: retrieval and reranking — LlamaIndex — 45-min tour of every retrieval trick that actually moves numbers.
• 🎥 Hands-on demo (10–20 min) · Reranking with Cohere Rerank + LangChain — hands-on — Copy-paste live coding of rerank in a real RAG pipeline.
• 📖 Canonical article · Pinecone — Hybrid Search Intro — The canonical explainer of BM25 + dense fusion.
• 📖 Book chapter / tutorial · Anthropic — Contextual Retrieval (Contextual BM25 + reranker) — Reports 67% failure-rate drop with contextual BM25 + reranker.
• 📖 Engineering blog · Weaviate — Hybrid Search Explained — Deeper dive on RRF fusion and score normalization.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Making retrieval actually work.

You built an index in Session 117. Now a user asks a question. What do you retrieve? Nearest 5 chunks by cosine similarity? That's the naive answer, and it's wrong about 40% of the time. Production retrieval is a funnel: broad recall first (grab 50–200 candidates fast), precise ranking second (rerank the top few with a smarter model), then hand only the top 3–10 to the LLM.

Three techniques you must know:

1. Dense retrieval (embeddings). What Session 117 built. Good at semantic similarity — "cheap accommodation" matches "budget hotels". Bad at exact terms — "product SKU X8271" against an embedding of "our newest widget" — the model doesn't care about the exact string.
2. Sparse retrieval (BM25 / keyword). The 1990s technique. Beats embeddings on rare tokens, product IDs, code snippets, legal citations. Fast, no model needed.
3. Hybrid (dense + sparse). Run both, fuse the ranked lists (usually Reciprocal Rank Fusion — RRF). Almost always beats either alone by 5–15 points on recall@10.

Then, on top of retrieval: reranking. Retrieval fetches 100 candidates in milliseconds using cheap vector math. A cross-encoder reranker (Cohere Rerank, bge-reranker, or an LLM) scores each (query, chunk) pair jointly — 10× slower per pair but way more accurate. The typical pattern: retrieve top-100 (fast), rerank to top-10 (slow but small batch), pass to LLM.

Gotcha to carry forever: the LLM cannot answer questions from chunks it never sees. Retrieval quality is the ceiling on RAG quality. Improving your model from GPT-4o-mini to Claude Opus won't help if the right chunk was ranked 47th and you only fed the top 5. Fix the retriever first, always.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

RRF fusion in one line. For each document that appears in either list, score = sum over lists of 1 / (k + rank_in_list), k usually 60. Rank the merged list by score. That's it — no learned weights, works surprisingly well, no score normalization needed.

Worked example — query: "How do I renew my SSL cert on nginx?"

Dense retrieval top-5:

1. Blog: "TLS termination best practices" (semantically related)
2. Doc: "Certbot renewal automation"
3. Blog: "Nginx security hardening"
4. Doc: "Setting up SSL for the first time"
5. Doc: "Debugging 502 errors"

BM25 top-5 (matching literal "SSL cert" + "nginx"):

1. Runbook: "Manual SSL cert renewal on nginx" ← the exact answer
2. Doc: "Setting up SSL for the first time"
3. FAQ: "SSL cert expired — 5 things to check"
4. Doc: "Nginx + Let's Encrypt walkthrough"
5. Blog: "TLS termination best practices"

The exact answer was not in dense top-5 (short, keyword-heavy runbook titles don't embed well). Hybrid + RRF pulls "Manual SSL cert renewal on nginx" up because it's rank 1 in BM25 (score = 1/61 = 0.0164) even though it's absent from dense. After RRF the runbook now ranks 1st or 2nd overall.

Add a reranker (Cohere Rerank v3, bge-reranker-v2-m3, or an LLM prompt): it scores each (query, chunk) pair jointly with cross-attention. On the same query, the reranker will strongly boost the runbook because its content matches the intent, not just the words. Final top-3 fed to LLM: runbook, Certbot doc, Let's Encrypt walkthrough. LLM answers correctly.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Hybrid retrieval + reranker in ~50 lines, using the index from Session 117:

# pip install langchain-community langchain-openai rank-bm25 sentence-transformers
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings
from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever
from sentence_transformers import CrossEncoder
 
emb = OpenAIEmbeddings(model="text-embedding-3-small")
vect = FAISS.load_local("./rag_index", emb, allow_dangerous_deserialization=True)
docs = [vect.docstore._dict[i] for i in vect.docstore._dict]
 
# 1. Dense retriever (top 20)
dense = vect.as_retriever(search_kwargs={"k": 20})
 
# 2. Sparse BM25 retriever (top 20)
bm25 = BM25Retriever.from_documents(docs); bm25.k = 20
 
# 3. Hybrid via Reciprocal Rank Fusion (LangChain's EnsembleRetriever uses RRF)
hybrid = EnsembleRetriever(retrievers=[dense, bm25], weights=[0.5, 0.5])
 
# 4. Cross-encoder reranker (open-source, runs locally)
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
 
def retrieve_and_rerank(q, k_final=5):
    candidates = hybrid.invoke(q)              # ~40 docs (dedupe from 20+20)
    pairs = [(q, d.page_content) for d in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])[:k_final]
    return ranked
 
for q in [
    "how do I renew SSL cert on nginx?",
    "who owns the deploy runbook?",
    "what is the refund window?",
]:
    print(f"\nQ: {q}")
    for doc, score in retrieve_and_rerank(q):
        src = doc.metadata.get("source", "?")
        preview = doc.page_content[:80].replace("\n", " ")
        print(f"  [{score:+.2f}] {src} :: {preview}...")

Checklist:

1. First run downloads bge-reranker-v2-m3 (~1 GB). Subsequent runs cache it.
2. Hybrid returns up to 40 chunks; reranker picks the best 5.
3. Reranker scores are unbounded; typically positive for good matches, negative for bad ones.
4. Compare against dense-only (vect.similarity_search(q, k=5)) — the reranker-augmented list should be visibly better for keyword-heavy or ambiguous queries.
5. Latency: dense ~50 ms, hybrid ~100 ms, hybrid+rerank ~300–800 ms on CPU (much faster on GPU).

Try this: replace BAAI/bge-reranker-v2-m3 with an LLM-as-reranker: prompt Claude/GPT with "Rate 0–10 how relevant this chunk is to the query." for each candidate. Compare speed, cost, and ranking quality. Then decide which one you'd actually ship.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Retrieval is where LLM apps go to die. The demo works, then real users type real questions, and the retriever returns junk. Nine out of ten "make our RAG better" projects are actually "make our retrieval better."

War stories. (1) Team ships dense-only retrieval — accuracy stalls at 65%. Adding BM25 + RRF jumps it to 78% in a weekend. Anthropic reports that adding contextual BM25 alone cuts retrieval failures by ~35%, and adding a reranker cuts them another ~50% on top. (2) Users type short queries ("SSL renewal") — embeddings underperform because there's little text to embed. Query rewriting (LLM expands the query to a paragraph before embedding) is a cheap win here. (3) Metadata filtering is non-negotiable at scale — a legal search over 1M docs must be filterable by jurisdiction/date first, then semantic-search the filtered subset. Vector DBs that don't support metadata filters (or that scan then filter) will not survive. (4) Reranker latency — cross-encoders are ~50× slower per doc than dense retrieval. If you rerank 500 candidates you'll blow your latency SLO. Rule: retrieve at most 100, rerank down to 10, feed to LLM 3–5. (5) Duplicate chunks in the LLM's context — retrievers happily return three near-identical chunks (docs get copied around wikis). Add a post-retrieval dedupe step (MinHash or embedding cosine ≥ 0.95) or you'll waste half your context window. (6) Cold-cache reranker — the first request after a deploy takes 2 seconds because the reranker loads weights. Warm it in a health-check.

How top teams handle it. Anthropic ships contextual retrieval (chunk-level context prepended before embedding + BM25 + reranker) as the current recipe. Perplexity does query rewriting → hybrid dense/BM25 → LLM reranker → answer synthesis. Enterprise Azure AI Search does semantic ranker on top of Cognitive Search's BM25 index. The industry consensus in 2024 is: dense alone is table stakes; hybrid + reranker is where good RAGs actually live; contextual/graph-aware retrieval is the frontier.

(e) Quiz + exercise · 10 min

1. Why does BM25 sometimes beat dense embeddings on the same query?
2. Explain RRF (Reciprocal Rank Fusion) in one line.
3. What does a cross-encoder reranker do that a bi-encoder dense retriever cannot?
4. Why is the funnel pattern (retrieve 100, rerank 10, feed 5) preferred over "just retrieve top-5 directly"?
5. What was Anthropic's Contextual Retrieval headline number, and what change caused it?

Stretch (connects to S117 — Chunking): Imagine your dense retrieval is missing the answer 30% of the time. Give three separate hypotheses (one about chunking, one about the query, one about the embedding model) and one experiment for each.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is RAG II? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S119): Vector Databases — pgvector, HNSW, IVF
• Previous (S117): RAG I — Chunking Strategies & Indexing
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.