R24 · Week 24 Recall & Drill

Weekly revision · Week 24 · Covers 5 sessions from Mon–Fri.

Sessions covered

• S116 — Prompting — Zero-Shot, Few-Shot, Chain-of-Thought, ReAct
• S117 — RAG I — Chunking Strategies & Indexing
• S118 — RAG II — Retrieval, Hybrid Search, Reranking
• S119 — Vector Databases — pgvector, HNSW, IVF
• S120 — LLM Agents — Function Calling, Tools, Planning

90-min structure

Blank-page reconstruction · 30 min

S116 · Prompting

1. Describe the four patterns in one sentence each.
2. Say when intermediate reasoning hurts rather than helps.
3. Explain what the thought step adds over a pure action loop.

Gotcha you probably forgot: intermediate reasoning degrades extraction tasks. When the job is to pull a value out of provided text, generating reasoning first gives the model room to reconsider, elaborate, and drift away from the literal answer — so accuracy falls and output format compliance falls with it. Reasoning helps where serial computation is genuinely needed, and hurts where the answer is already present.

S117 · Chunking & Indexing

1. Name the four splitting strategies and one corpus suited to each.
2. Say why splitting mid-sentence is so damaging.
3. List the minimum metadata every chunk should carry.

Gotcha you probably forgot: prefixing each chunk with its heading path before embedding is one of the cheapest quality improvements available. A chunk reading "the limit is thirty per minute" is nearly meaningless in isolation and becomes retrievable once it carries the section it came from, because the embedding then encodes the topic rather than just the sentence.

S118 · Retrieval & Reranking

1. Say what dense retrieval catches that keyword scoring misses, and the reverse.
2. Explain why rank-based fusion works without weight tuning.
3. Distinguish the two encoder architectures and their cost profiles.

Gotcha you probably forgot: you cannot compare similarity scores and keyword relevance scores directly, because they live on different, unbounded, corpus-dependent scales. Any attempt to blend them with a weight requires recalibration whenever the corpus changes, which is exactly why fusing by rank position rather than by score is the robust default.

S119 · Vector Indexes

1. Describe the graph-based and partition-based index families in one sentence each.
2. Name the two tuning knobs for each.
3. Say why building the graph index takes so much longer.

Gotcha you probably forgot: applying a heavy metadata filter to an approximate search can make latency explode rather than improve, because the index traverses towards nearest neighbours that the filter then discards, forcing it to search far more of the structure to fill the result set. The fixes are partitioning the index by the filter dimension, or filtering during traversal rather than after it.

S120 · Agents

1. Describe the control loop in three steps.
2. Say why tools should return errors as data rather than raise.
3. Name three limits every production agent must enforce.

Gotcha you probably forgot: deterministic decoding is the default for tool-using agents because sampling variety in a control loop means a different plan each run, which makes failures irreproducible and evaluation meaningless. Variety belongs in content generation, not in the step that decides which function to call.

Hands-on drill · 30 min

Task: measure what chunk boundaries destroy, fuse two rankings, tune an index dial, and compute a trajectory's odds.

mkdir -p ~/projects/w24-drill && cd ~/projects/w24-drill

Step 1 — boundaries decide what is retrievable (8 min)

# chunking.py
doc = ("Rate limits. The public API allows a maximum of thirty requests per minute "
       "per API key. Exceeding this returns a 429 response with a Retry-After header. "
       "Enterprise keys are provisioned separately and are not subject to this limit. "
       "Billing. Invoices are issued monthly in arrears and are due within fourteen days.")
 
def fixed(text, size, overlap):
    words = text.split()
    step = max(1, size - overlap)
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]
 
FACT = ["thirty", "requests", "per", "minute"]
def contains_fact(chunk):
    return all(w in chunk for w in FACT)
 
print(f"{'size':>5} {'overlap':>8} {'chunks':>7} {'chunks with the complete fact':>32}")
for size, overlap in [(8, 0), (8, 2), (20, 0), (20, 3), (40, 6)]:
    ch = fixed(doc, size, overlap)
    hits = sum(contains_fact(c) for c in ch)
    print(f"{size:>5} {overlap:>8} {len(ch):>7} {hits:>32}")
 
print("\nAny configuration with zero hits means no embedding model, no reranker,")
print("and no amount of prompt engineering can ever surface that fact. The chunk")
print("is the atomic unit of retrieval, so chunking sets the ceiling.\n")
 
# Structural splitting keeps a coherent topic per chunk.
for section in doc.split(". "):
    pass
sections = ["Rate limits." + doc.split("Rate limits.")[1].split("Billing.")[0],
            "Billing." + doc.split("Billing.")[1]]
for s in sections:
    print(f"structural chunk ({len(s.split()):>2} words): {s.strip()[:70]}...")
print("\nEach structural chunk covers one topic, so its embedding points in one")
print("direction rather than averaging several -- which is why a mixed chunk")
print("retrieves poorly for every topic it contains.")

Expected outcome: small chunks with no overlap split the fact across a boundary and it becomes unreachable — the hit count is zero, and that is a hard ceiling nothing downstream can lift. Adding overlap recovers it, which is the entire justification for overlap existing. The structural split shows the other half of the argument: a chunk spanning two topics gets an embedding that is roughly the average of two directions and therefore sits close to neither, so it ranks poorly for both queries.

Step 2 — hybrid retrieval and rank fusion (8 min)

# hybrid.py
docs = {
    1: "how to reset your password from the account settings page",
    2: "error ERR_4471 occurs when the auth token has expired",
    3: "changing your credentials requires email verification first",
    4: "troubleshooting expired sessions and re-authentication flows",
    5: "the billing portal is separate from the account settings page",
}
 
def sparse_rank(query):
    """Exact term overlap: finds rare identifiers, blind to synonyms."""
    q = set(query.lower().split())
    scored = [(d, len(q & set(t.split()))) for d, t in docs.items()]
    return [d for d, s in sorted(scored, key=lambda x: -x[1]) if s > 0]
 
SYNONYMS = {"password": {"credentials", "password"}, "reset": {"reset", "changing"},
            "expired": {"expired", "expiry"}, "token": {"token", "session", "sessions"}}
def dense_rank(query):
    """Crude synonym expansion standing in for embedding similarity."""
    q = set(query.lower().split())
    expanded = set(q)
    for w in q:
        expanded |= SYNONYMS.get(w, set())
    scored = [(d, len(expanded & set(t.split()))) for d, t in docs.items()]
    return [d for d, s in sorted(scored, key=lambda x: -x[1]) if s > 0]
 
def rrf(rankings, k=60):
    scores = {}
    for r in rankings:
        for pos, d in enumerate(r):
            scores[d] = scores.get(d, 0) + 1 / (k + pos + 1)
    return [d for d, _ in sorted(scores.items(), key=lambda x: -x[1])]
 
for q in ["reset password", "ERR_4471"]:
    s, dn = sparse_rank(q), dense_rank(q)
    print(f"\nquery: {q!r}")
    print(f"  keyword ranking: {s}")
    print(f"  semantic ranking: {dn}")
    print(f"  fused by rank  : {rrf([s, dn])}")
 
print("\nThe identifier query is where semantic retrieval fails: rare tokens are")
print("compressed away by embedding models precisely because they were rare in")
print("training. Rank fusion needs no score calibration, which is why it survives")
print("corpus changes that would invalidate any weighted score blend.")

Expected outcome: the natural-language query benefits from semantic matching, which finds the document using different words for the same concept, while keyword matching misses it entirely. The identifier query inverts this completely — exact term matching finds it instantly and semantic matching does not, because rare identifiers are exactly the tokens embeddings compress away. Fusing by rank position combines both without needing to reconcile two incomparable score scales, which is why it holds up when the corpus changes.

Step 3 — recall is a dial, not a property (7 min)

# index.py
import numpy as np
 
rng = np.random.default_rng(0)
N, d, Q = 20_000, 64, 200
data = rng.normal(size=(N, d))
data /= np.linalg.norm(data, axis=1, keepdims=True)
queries = rng.normal(size=(Q, d))
queries /= np.linalg.norm(queries, axis=1, keepdims=True)
 
truth = np.argsort(-(queries @ data.T), axis=1)[:, :10]
 
# Partition-based index: cluster once, search only the nearest partitions.
nlist = 128
centroid_idx = rng.choice(N, nlist, replace=False)
centroids = data[centroid_idx]
assign = np.argmax(data @ centroids.T, axis=1)
buckets = {c: np.flatnonzero(assign == c) for c in range(nlist)}
 
print(f"{'nprobe':>7} {'recall@10':>10} {'vectors scanned':>17} {'fraction of corpus':>20}")
for nprobe in (1, 4, 16, 64, 128):
    hits, scanned = 0, 0
    for i, q in enumerate(queries):
        order = np.argsort(-(q @ centroids.T))[:nprobe]
        cand = np.concatenate([buckets[c] for c in order if len(buckets[c])])
        scanned += len(cand)
        top = cand[np.argsort(-(data[cand] @ q))[:10]]
        hits += len(set(top) & set(truth[i]))
    print(f"{nprobe:>7} {hits/(Q*10):>10.3f} {scanned//Q:>17,} {scanned/(Q*N):>20.1%}")

Expected outcome: recall rises monotonically with the number of partitions probed, and so does the amount of the corpus scanned — at the maximum you have recovered exact search and paid exact search's cost. That is the whole point: approximate search does not have a recall, it has a recall-versus-latency curve, and the operating point is yours to choose. Any vendor claim of a recall figure is meaningless without the corresponding latency and the parameters that produced it, and the correct way to evaluate an index is to measure that curve on your own data.

Step 4 — trajectories compound (7 min)

# agent_math.py
print(f"{'per-step reliability':>21} {'5 steps':>9} {'10 steps':>10} {'20 steps':>10} {'50 steps':>10}")
for p in (0.99, 0.97, 0.95, 0.90):
    row = "  ".join(f"{p**n:>8.1%}" for n in (5, 10, 20, 50))
    print(f"{p:>21.0%}  {row}")
 
print("\nNow with verification: a check after each step that catches a fraction of errors.")
def with_verification(p, n, catch):
    effective = p + (1 - p) * catch          # errors caught are retried successfully
    return effective ** n
 
print(f"{'catch rate':>21} {'20 steps at p=0.95':>22}")
for catch in (0.0, 0.5, 0.8, 0.95):
    print(f"{catch:>21.0%} {with_verification(0.95, 20, catch):>22.1%}")
 
print("\nThe lesson is structural: adding iterations to an unreliable loop lowers")
print("success, because a wrong intermediate result is carried forward and")
print("everything after it is built on it. Verification per step is what makes")
print("long trajectories viable -- not a better model and not more attempts.")

Expected outcome: a per-step reliability that sounds excellent produces a poor end-to-end success rate once the trajectory is long, and the decline is steep because the terms multiply. The second table shows the lever: catching and correcting errors at each step raises the effective per-step reliability, and because that number is then raised to the same power, modest verification produces a large improvement in the trajectory. This is why bounded, verified workflows beat open-ended loops in production, and why "let it iterate more" makes things worse rather than better.

Gap analysis + next week preview · 10 min

• Did the zero-hit rows in Step 1 land? That ceiling is invisible in every evaluation that only measures ranking quality.
• Could you explain rank fusion's advantage without the drill? It comes up whenever someone proposes a tuned score blend.
• Did the trajectory table change your view of "let it try more times"? That intuition is backwards and costs real money.

Next week (S121–S125) covers the production side of these systems: evaluating generative applications when there is no single correct answer, guardrails and failure containment, cost and latency engineering for serving, fine-tuning versus retrieval as competing answers to the same problem, and the deployment and monitoring practices that keep any of it working. The retrieval and agent trade-offs from this week are the material those decisions operate on.

Part of the 6-month evergreen learning plan.