R24 · Week 24 Recall & Drill

Assessment scope

A retrieval-and-tool application succeeds only when evidence boundaries, ranking, index search and state transitions all work together. This revision keeps the original prompting, chunking, hybrid retrieval, vector-index and agent objectives, while replacing non-demonstrations with real boundary failures, explicit rank arithmetic, exact-neighbor oracles and a bounded recovery model. No embeddings, LLM API, pgvector server or production agent benchmark is executed.

Sessions covered

• S116 — Prompting — Zero-Shot, Few-Shot, Chain-of-Thought, ReAct
• S117 — RAG I — Chunking Strategies & Indexing
• S118 — RAG II — Retrieval, Hybrid Search, Reranking
• S119 — Vector Databases — pgvector, HNSW, IVF
• S120 — LLM Agents — Function Calling, Tools, Planning

90-min structure

The timings are optional pacing. A real index/LLM evaluation needs a separately measured workload and service budget.

Blank-page reconstruction · 30 min

S116 · Prompting

1. Define zero-shot, few-shot, intermediate reasoning and ReAct.
2. When may extra reasoning hurt, and what evidence would establish it?
3. What does a planning/reasoning step contribute over a pure action loop?

Solutions. Zero-shot supplies instructions without worked task examples. Few-shot adds representative input/output examples, including format and boundary cases. Intermediate-reasoning prompting asks the model to work through a task before its answer; generated intermediate tokens add sequential computation but are not a guaranteed faithful explanation of internal causation. ReAct interleaves reasoning/planning, actions and observations so the next action can respond to external information rather than only a static prompt.

For literal extraction, prefer a precise field/schema contract and evidence span; unnecessary elaboration can increase latency, unsupported additions or format failures. But it is not a universal empirical law that reasoning decreases extraction accuracy: model, prompt, ambiguity and evaluation determine that. Compare exact-match/field accuracy, abstention, formatting and cost on held-out cases. For example, given “Invoice total: USD 120; tax included,” require currency=USD,total=120 and the source span rather than a creative financial explanation.

A planning step can track unresolved subgoals, choose a discriminating tool and revise after an error. The ReAct paper reports benefits on particular benchmarks; that does not certify every generated plan. Treat external text/tool observations as data, not authority to override instructions or permissions. Verify observable answers and tool effects; do not use a fluent rationale as the acceptance criterion.

S117 · Chunking & Indexing

1. Compare four splitting strategies and choose size/overlap.
2. Why do boundaries matter, and how can a system recover cross-boundary evidence?
3. Specify chunk metadata and access-control filtering.

Solutions. Fixed token windows are simple for homogeneous text but can split tables/sentences. Sentence/recursive splitting preserves linguistic boundaries while packing toward a budget. Structural splitting follows headings, code symbols or document sections, useful for manuals and code. Semantic splitting uses topic-change signals, with extra model cost and its own evaluation; it is not inherently superior. Choose size to contain useful evidence while avoiding irrelevant context, and overlap to cover likely boundary spans without excessive duplicate storage/retrieval. Measure answer-span coverage and end-to-end retrieval, not a universal 500-token rule.

Corpus choices to defend. Start with fixed windows for uniform short support logs; sentence/recursive packing for prose FAQs; heading/function boundaries for an API manual or source repository; and evaluated semantic boundaries for long mixed-topic transcripts without reliable headings. These are candidates, not measured winners. In the policy fixture below, size 8 and overlap 4 rescue the short numeric span, but a policy-level question also needs the enterprise qualifier, so choose the complete policy section or authorized neighbor expansion rather than declaring that overlap alone solved the question.

A small chunk may detach “thirty requests per minute” from the API-key scope or the enterprise exception; a large chunk may dilute a focused query or waste context. A boundary does not make a fact logically unreachable in every system: overlap, neighbor expansion, parent retrieval or multi-chunk evidence assembly can recover it. The lab separately measures single-chunk span containment and neighbor recovery, not an embedding accuracy claim.

A useful minimum record includes stable document/chunk ID, source URI, document version/content hash, heading path, start/end offsets, tenant/access labels, and index/embedding model version; timestamps and language/type can support freshness/filtering. Prefixing an appropriate heading path before embedding can add context, but must be tested rather than declared a guaranteed improvement. Keep displayed citations bound to exact source/version and validate that the cited span supports the answer.

Authorization must exclude inaccessible evidence before it can reach a model, reranker or user-visible result, and permissions should be rechecked when serving stale cached/indexed content. This logical rule does not assert every database physically prefilters its ANN traversal. Apply tenant/ACL constraints safely in candidate generation where supported; if an engine filters later, ensure no intermediate unauthorized content escapes and measure candidate starvation. Deletion/version updates must invalidate obsolete chunks, not leave stale contradictory evidence indefinitely.

S118 · Retrieval & Reranking

1. Compare lexical and dense retrieval failure cases.
2. Derive reciprocal rank fusion and state its parameters.
3. Distinguish bi-encoders and cross-encoders.

Solutions. Lexical retrieval (for example BM25) rewards matching terms with corpus-dependent weighting and can excel at exact identifiers such as ERR_4471 if analysis preserves them. Dense embeddings can retrieve semantic paraphrases without shared words, but do not guarantee exact-match behavior; they may also retrieve identifiers successfully. Neither universally dominates. A lexical analyzer can destroy an identifier too, so measure the actual tokenization/case policy instead of blaming vector semantics for a string-processing bug.

With one-based ranks r_j(d), reciprocal rank fusion computes sum_j 1/(k+r_j(d)) for lists containing d. A document absent from a list contributes zero. It avoids directly adding incomparable raw scores, but still has a rank constant k, candidate depths, weighting options, tie rules and duplicate policy. The original paper found k=60 useful in its experiments; not tuning score-scale weights does not mean no choices or no evaluation. Cosine is bounded between -1 and 1 for real nonzero vectors, while other scores have different scales; “all scores are unbounded” is false. Learned/calibrated score blending can also work and should be compared when appropriate.

A bi-encoder embeds query and documents independently so document embeddings can be precomputed and ANN searched. A cross-encoder jointly processes each query-document pair for interaction-rich scoring, usually too expensive to score the whole corpus per query but useful to rerank a bounded candidate set. A reranker cannot rank a document never retrieved unless the pipeline expands candidates. Measure candidate recall, final ranking and answer faithfulness separately, under the same permission filters.

S119 · Vector Indexes

1. Compare HNSW and IVF, including build work.
2. Name construction and query controls with a pinned implementation.
3. Interpret recall/work/latency and selective filters.

Solutions. HNSW searches a hierarchical proximity graph, navigating from coarse upper levels to a finer neighborhood. IVF partitions vectors around trained centroids and searches selected lists, optionally with compressed vectors depending on the variant. HNSW construction performs neighbor searches and graph connections, consuming build time and memory; IVFFlat also requires training/assignment. Which is faster depends on data, dimension, parameters and implementation—“graph always takes much longer” is not a theorem.

In pgvector v0.8.0, HNSW has m (connections per layer), ef_construction (construction candidate list) and query hnsw.ef_search. IVF has construction lists and query ivfflat.probes. More search effort often improves recall at greater work; graph structure and build controls influence the attainable tradeoff. Evaluate recall@k against exact nearest neighbors under the same distance/filter/tie contract, across multiple queries and relevant cohorts. Report latency percentiles, throughput, memory/build cost and settings, not a single recall number. General real-system latency and recall need not be monotonic on every run.

The pinned pgvector documentation says approximate-index filtering is applied after the index scan, potentially returning fewer qualifying rows. v0.8.0 adds iterative scans that expand search until sufficient results or a configured limit. Other options include an exact filter-column index for selective subsets, partial indexes for a few filter values, or partitioning where justified. A filter may reduce returned candidates, increase search work, or enable a faster exact plan; do not promise it always accelerates search or always explodes latency. Tenant partitioning also has operational cost and does not replace authorization.

S120 · Agents

1. Describe the observation/decision/action loop.
2. Classify errors and state four recurring failure modes.
3. Enforce limits and explain conditional trajectory reliability.

Solutions. Observe current task state and permitted tool results; choose a schema-valid, authorized next action; execute it through a bounded runtime, record the effect and verify progress before continuing or stopping. Schema validity does not authorize an action. Scope credentials, tool names, resource identifiers and side effects outside the model's discretion; require human approval for sensitive actions when policy calls for it. Retrieved pages and tool messages can carry indirect prompt injection, so they must not acquire instruction authority.

Four useful failure categories are (1) wrong or unauthorized tool/arguments, (2) misinterpreted or fabricated observations, (3) looping/no progress or poor decomposition, and (4) unsafe/repeated side effects or corrupted state. This is a teaching taxonomy, not a universal canonical list. Log the states and outcomes that distinguish them, avoiding secrets. Expected tool failures can return typed error data with retryability and safe details; catch transport exceptions at the orchestration boundary. Do not swallow programming bugs, authorization failures or corrupted invariants into a generic “try again” response. A timeout after a side effect is ambiguous: retry only with an idempotency/reconciliation strategy.

Bound iterations, wall-clock time, tokens/cost, tool-call count, data volume and concurrency as appropriate; the first three are a minimum, not the only limits. Detect repeated states and lack of progress; provide a safe stop/escalation result. Lower-variance/greedy decoding can aid reproducibility, but temperature zero does not guarantee deterministic serving, correct tool choices or meaningful evaluation by itself. Record model/tool versions and replayable state; sample-based policies can be evaluated too.

For n required steps, exact chain-rule success is the product of each step's probability conditional on previous success. If those conditional probabilities all equal p, it is p^n; independence with identical success probability is a sufficient toy assumption, not something measured by the formula. Shared failures, verification/recovery and adaptive stopping change the model. A deterministic workflow still has component failures; known control flow does not remove the reliability product.

Hands-on drill · 30 min

The fixtures are intentionally small so each expected result is independently inspectable. No live tools, external effects, ANN library or embedding model are invoked.

Step 1 — boundaries decide what is retrievable (8 min)

words="Public API keys allow at most thirty requests per minute per key unless enterprise policy applies".split()
fact="thirty requests per minute".split()
start=words.index('thirty');end=start+len(fact)
def chunks(words,size,overlap):
    if size<=0 or not 0<=overlap<size:raise ValueError('invalid chunk configuration')
    return [(i,min(i+size,len(words)),words[i:i+size]) for i in range(0,len(words),size-overlap)]
def covers(record):return record[0]<=start and record[1]>=end
plain=chunks(words,8,0);overlap=chunks(words,8,4)
assert sum(map(covers,plain))==0
assert sum(map(covers,overlap))==1
assert words[plain[0][0]:plain[1][1]][start:end]==fact
for label,records in [('plain',plain),('overlap',overlap)]:
    print(label,[(lo,hi,' '.join(text),covers((lo,hi,text))) for lo,hi,text in records])
# Neighbor expansion joins source offsets, not duplicated overlapping text.
neighbor_lo,neighbor_hi=plain[0][0],plain[1][1]
assert neighbor_lo<=start and neighbor_hi>=end
metadata={'doc_id':'api-policy','version':'v1','chunk_id':'api-policy:4:12',
          'source':'https://example.invalid/api-policy','heading':'Public API / Limits',
          'start':4,'end':12,'tenant':'tenant-a','allowed_roles':['reader'],
          'embedding_version':'toy-no-embedding'}
assert metadata['tenant']!='tenant-b'  # a tenant-b request must not receive this chunk
print('cross-boundary fact recovered by overlap and by neighbor expansion',start,end)

The no-overlap boundary at token 8 separates the required span [6,10). Overlap produces [4,12), which contains it; expanding adjacent plain chunks also recovers it. Neither contains the entire exception policy by implication: an answer about enterprise keys needs the qualifier too. Span coverage is necessary for this single-span answer contract but not sufficient for ranked retrieval or faithful generation. A structural policy chunk would keep rule and exception together; that is a candidate to evaluate, not an embedding win asserted without embeddings.

Step 2 — hybrid retrieval and rank fusion (8 min)

import re
from fractions import Fraction
from collections import defaultdict
docs={1:'reset your password from account settings',2:'error ERR_4471 means auth token expired',
      3:'changing credentials requires email verification',4:'billing portal settings'}
def terms(text):return set(re.findall(r'[a-z0-9_]+',text.lower()))
def lexical(query):
    scored=[(doc,len(terms(query)&terms(text))) for doc,text in docs.items()]
    return [doc for doc,score in sorted(scored,key=lambda pair:(-pair[1],pair[0])) if score>0]
assert lexical('ERR_4471')==[2]
def rrf(rankings,k=60):
    if k<0:raise ValueError('nonnegative rank constant required')
    scores=defaultdict(Fraction)
    for ranking in rankings:
        seen=set();rank=0
        for doc in ranking:
            if doc in seen:continue
            seen.add(doc);rank+=1
            scores[doc]+=Fraction(1,k+rank)
    order=sorted(scores,key=lambda doc:(-scores[doc],doc))
    return order,dict(scores)
# Fixed candidate rankings stand in for two retrievers; the second is NOT a measured dense model.
keyword=lexical('reset password');semantic_fixture=[3,1]
order,scores=rrf([keyword,semantic_fixture])
assert keyword==[1] and order==[1,3]
assert scores[1]==Fraction(1,61)+Fraction(1,62)
assert scores[3]==Fraction(1,61)
assert rrf([[1,1,2],[2]])==rrf([[1,2],[2]])
assert rrf([],60)==([],{})
print('keyword',keyword,'fixed semantic fixture',semantic_fixture,'fused',order,'exact scores',scores)
# Permissions checked before any candidate text goes to a reranker/model.
allowed={1,3};candidate_order,_=rrf([[2,1],[3,2]])
visible=[doc for doc in candidate_order if doc in allowed]
assert set(visible)=={1,3} and 2 not in visible
print('authorized candidate IDs',visible)

The identifier succeeds because both query and document use the same case policy. The old example lowercased only the query, so its empty result was a lexical bug, not evidence that dense search loses rare identifiers. Here the semantic candidate list is explicitly supplied; no synonyms are mislabeled as embeddings. RRF rewards a document present in both lists; duplicate IDs within one list count once, ranks are compacted after deduplication, and ties use document ID. Evaluate real retrievers and rank constants on held-out queries before claiming improvement.

Step 3 — recall is a dial, not a property (7 min)

import numpy as np
rng=np.random.default_rng(0)
N,d,Q,nlist,k=600,12,12,16,5
data=rng.normal(size=(N,d));data/=np.linalg.norm(data,axis=1,keepdims=True)
queries=rng.normal(size=(Q,d));queries/=np.linalg.norm(queries,axis=1,keepdims=True)
centroids=data[rng.choice(N,nlist,replace=False)]
assign=np.argmax(data@centroids.T,axis=1)
buckets={c:np.flatnonzero(assign==c) for c in range(nlist)}
def top_ids(q,candidates):
    candidates=np.asarray(candidates,dtype=int)
    return candidates[np.lexsort((candidates,-(data[candidates]@q)))[:k]]
truth=[top_ids(q,np.arange(N)) for q in queries]
previous_hits=-1;previous_scanned=-1
for probes in [1,2,4,8,16]:
    hits=scanned=0
    for i,q in enumerate(queries):
        order=np.lexsort((np.arange(nlist),-(centroids@q)))[:probes]
        candidates=np.concatenate([buckets[c] for c in order])
        got=top_ids(q,candidates);scanned+=len(candidates)
        hits+=len(set(got)&set(truth[i]))
    assert hits>=previous_hits and scanned>=previous_scanned
    previous_hits,previous_scanned=hits,scanned
    print('probes',probes,'recall@5',hits/(Q*k),'mean vectors scanned',scanned/Q)
assert hits==Q*k and scanned==Q*N
# Independent scalar-loop exact-neighbor oracle for every query.
for i,q in enumerate(queries):
    reference=sorted(range(N),key=lambda j:(-sum(float(a)*float(b) for a,b in zip(data[j],q)),j))[:k]
    assert truth[i].tolist()==reference
# Selective authorization/filter example: compare to exact neighbors in that subset, not global truth.
allowed=np.arange(0,N,10)
filtered_truth=top_ids(queries[0],allowed)
assert set(filtered_truth)<=set(allowed)
print('filtered exact neighbors',filtered_truth)

Filtered-recall counterexample. Suppose the distance-ordered IDs are [0,1,2,3], only {2,3} are authorized, and k=2. Taking global top two then filtering returns nothing; exact search inside the permitted set returns [2,3]. ACL checks must still reject 0 and 1, but merely applying the check is not a sufficient recall strategy. Expand candidates safely or search the permitted subset; compute filtered recall against {2,3}, not the unauthorized global truth. This is an inspectable starvation example, not a pgvector execution.

This is a coarse partition-search model using sampled centroids, not trained IVF or HNSW. For each fixed query, increasing probes takes a prefix of the same centroid order, so candidate sets nest. Exact reranking with a stable tie rule means the number of global top-k neighbors recovered cannot decrease in this model. At all lists every vector is scanned and recall is 1. Real libraries may use compression, traversal heuristics and scan limits; their measured curves and latency need separate testing. “Vectors scanned” is a work count, not elapsed latency or a cloud benchmark.

Step 4 — trajectories compound (7 min)

from fractions import Fraction
for p in [.99,.97,.95,.90]:print('step reliability',p,'trajectory',[p**n for n in [5,10,20,50]])
def verified_step(p,catch,retry_success):
    # One retry of a detected error; no false positives, no undetected repair, no extra harm.
    return p+(1-p)*catch*retry_success
for catch in [0.,.5,.8,.95]:
    effective=verified_step(.95,catch,.95)
    print('catch',catch,'one-retry effective',effective,'20 steps',effective**20)
assert verified_step(Fraction(9,10),Fraction(1,2),Fraction(4,5))==Fraction(47,50)
# A bounded tool-workflow model with validation, idempotent effects and ambiguous post-effect timeout.
def workflow(schedule,max_calls=3):
    visible={};completed=False;history=[]
    for call,outcome in enumerate(schedule[:max_calls],1):
        if outcome=='forbidden':
            history.append(('denied',call));return {'status':'denied','effects':visible,'history':history}
        if outcome=='transport-before':
            history.append(('retryable',call));continue
        if outcome not in {'ok','timeout-after'}:
            raise ValueError('programming/protocol error is not a retryable tool observation')
        key='task-7:reserve-item-9'
        visible.setdefault(key,{'item':9,'quantity':1}) # same operation key on retries
        if outcome=='timeout-after':
            history.append(('ambiguous',call));continue
        assert visible[key]=={'item':9,'quantity':1}
        completed=True;history.append(('verified',call));break
    return {'status':'complete' if completed else 'bounded-stop','effects':visible,'history':history}
baseline=workflow(['ok'])
for schedule in [['transport-before','ok'],['timeout-after','ok'],['timeout-after','timeout-after','ok']]:
    result=workflow(schedule)
    assert result['status']=='complete' and result['effects']==baseline['effects']
    print('recovery',schedule,result['history'])
assert workflow(['transport-before']*4)['status']=='bounded-stop'
assert workflow(['forbidden','ok'])['effects']=={}
ambiguous=workflow(['timeout-after']*4)
assert ambiguous['status']=='bounded-stop' and len(ambiguous['effects'])==1
# Negative control: naive append after an ambiguous success duplicates the effect.
naive=[]
for outcome in ['timeout-after','ok']:naive.append({'item':9,'quantity':1})
assert len(naive)==2 and len(baseline['effects'])==1
print('naive duplicate effects',len(naive),'bounded ambiguous status',ambiguous['status'])

At p=0.95, 20 independent required steps succeed about 35.85% of the time. The verification calculation includes a retry-success factor instead of assuming every detected failure is repaired perfectly. It still omits checker false positives, correlated failures and harmful retries; those need a richer model or actual measurements.

The workflow separates externally visible effects from verified completion. A timeout after mutation leaves an ambiguous result; reusing the operation key avoids duplicating the modeled effect, while bounded-stop does not assert rollback. The dictionary simulates an idempotent sink; it does not prove durable crash safety, multi-process atomicity or a real API's idempotency contract. A production equivalent needs atomic key/effect storage or an actual provider contract plus reconciliation, authorization and approval gates.

Answer constraints and corrections

Misconception: “Dense embeddings replace keyword search.” Dense retrieval can generalize wording; lexical retrieval can preserve exact terms, provided its analyzer does. Neither guarantees every case. Compare exact-ID, paraphrase, multilingual and permission-filtered query cohorts and combine/rerank only when evaluation supports it.

Quiz solutions:

1. Extra generated reasoning can support multi-step computation but is not a faithful explanation guarantee; extraction effects must be measured, with strict output/evidence contracts.
2. Chunk boundaries affect candidate evidence and context quality, but overlap/parent/neighbor retrieval can recover a split fact. The executed [6,10) span demonstrates both failure and recovery.
3. RRF avoids raw score-scale calibration, not all parameter choices. k, depth, deduplication and ties matter; weighted score fusion is also a candidate, not inherently invalid.
4. Approximate indexes expose a recall/work/latency tradeoff. Nesting and monotonicity are proved for the exact-reranked partition model, not every ANN implementation or measured latency sample.
5. If longer production tasks fail more, inspect categorized per-step traces, distribution shift, tool errors, permissions and correlated dependencies rather than assuming demos were short. Introduce typed errors, bounded retries, idempotent effects, state validation, explicit human escalation and deterministic control flow where appropriate. Limit iterations/time/cost and measure verifier false positives/negatives. Greedy decoding may aid replay but does not ensure correctness; a workflow with fixed steps can still fail. More attempts help only when the recovery policy and added risk justify them.

Gap analysis + next week preview · 10 min

Retest a split qualifier, an identifier with different case, duplicate rank entries, a highly selective ACL or a timeout after a successful side effect. Keep candidate recall, answer faithfulness and verified effect completion as separate scores. The next phase evaluates and operates generative applications; no future-session navigation claim here substitutes for reviewing those lessons.

Corrective source notes

Primary references checked 2026-10-07: ReAct v3 record, OWASP indirect prompt injection, Cormack et al. RRF, pgvector v0.8.0 README, Faiss index documentation and OWASP authorization checks. Chunk-span, fusion, neighbor and workflow claims above are explicit model contracts with exact tests. No real dense retrieval, HNSW/IVF latency, distributed retry guarantee or deployed agent success rate is claimed.

Additional primary references: chain-of-thought prompting experiments, chunking strategies and context, and Sentence Transformers retrieve/rerank architecture.

Part of the 6-month evergreen learning plan.