R25 · Week 25 Recall & Drill

Assessment scope

Reconstruct five LLM-engineering topics: orchestration boundaries, evaluation, low-rank adaptation, serving and multimodal input. This revision supplies worked answers and four CPU-only Python drills, not a LangGraph/CrewAI deployment, a trained adapter, a GPU benchmark or a vision-model evaluation. The examples use synthetic data and existing Python/NumPy; no model, dataset, API account or download is needed. The 90-minute agenda is optional pacing, not a mastery guarantee.

Sessions covered

• S121 — Multi-Agent Orchestration
• S122 — LLM Evaluation
• S123 — Fine-Tuning and LoRA
• S124 — Serving, KV Cache and Batching
• S125 — Multimodal LLMs

By the end, aim to choose an orchestration pattern with a stopping rule; construct a calibrated evaluation set; derive adapter and cache sizes; distinguish prefill/decode bottlenecks; compare bounded schedulers; and separate image-text scoring from generative extraction. The answers below correct the original false absolutes rather than asking you to reproduce them.

90-min structure

The sum is 90. Installation, service setup and extended debugging are not included; split the session if needed.

Blank-page reconstruction · 30 min

S121 · Multi-Agent Orchestration

Prompts: name three useful orchestration patterns; decide when to split work; identify two runaway failures.

Worked answer. A fixed pipeline passes validated outputs through stages; a supervisor/worker pattern routes bounded subtasks and combines results; a peer handoff passes control between specialists. This is a useful taxonomy, not an exhaustive set of three canonical architectures. A fixed pipeline suits stable dependencies, a supervisor suits dynamically decomposable work, and peer handoff can suit changes in tool/domain ownership but complicates termination and provenance. Anthropic’s engineering guidance distinguishes predefined workflows from model-directed agents and recommends the simplest adequate design.

Add workers when separable work, tools or isolation requirements justify coordination overhead. Independent subtasks are one useful case, not a necessary condition: dependent pipeline stages can also be worthwhile. Compare a single-worker baseline on the same task set, quality rubric, elapsed latency and token/call budget. Parallelism can reduce elapsed time while increasing total work.

Two runaway cases are cyclic delegation and recursive task expansion/duplicated work. Give every task an ID, owner, input contract, allowed actions, maximum hops/calls, deadline and terminal state. A concise supervisor prompt can help, but shortness does not make routing deterministic. Enforce bounds in code. A handoff can be lossless with respect to required structured fields; it is not inherently a lossy summary. Field presence also does not guarantee truthful content.

S122 · Evaluation

Prompts: distinguish metric types; name three judge biases with mitigations; explain the role of a small hand-graded set.

Worked answer. Reference-based metrics compare with a known answer or cited evidence; reference-free judges score the response from a rubric/context without a gold response; task-specific checks measure the actual contract, such as valid JSON, exact invoice cents or a correct SQL result. These categories can overlap. A fluent answer can pass a style judge while failing the task.

Zheng et al. (2023, v4) discuss position, verbosity and self-enhancement biases as well as limited reasoning. Their self-enhancement analysis was inconclusive because of limited data and small differences; treat self-preference as a risk to test, not an established property of every judge. Balance A/B order to diagnose position effects; score required facts rather than length to reduce verbosity rewards; blind model identities and compare independent judges/human labels for self-preference. None of these guarantees elimination. Preserve disagreements and ties rather than forcing a winner.

For a starter golden set, hand-check 12 cases: four ordinary, two ambiguous, two missing-evidence, two malicious-input and two schema-boundary cases. Store input, expected properties, acceptable alternatives, severity and source. Separate development cases from later held-out cases; twelve is an illustrative debugging set, not statistically precise production coverage. Generated cases can broaden coverage but need verification. Human labels can also be inconsistent, so resolve rubric disagreements.

Gotcha correction: low-temperature or greedy decoding reduces one source of variation but does not guarantee deterministic API behavior or correct judgment. Pin model/prompt versions where possible, record settings, repeat a subset to quantify instability and compare paired results with uncertainty. Stochastic evaluation is usable if noise is measured; it is not automatically a coin flip.

S123 · Fine-Tuning

Prompts: explain low-rank updates, rank tradeoffs and three reasons not to fine-tune.

For a frozen weight matrix W with shape (d_out, d_in), LoRA learns B @ A, with B shaped (d_out, r) and A shaped (r, d_in). The output is W @ x + (alpha/r) * B @ A @ x. Trainable size is r * (d_in + d_out), and the update has rank at most r. The LoRA paper calls low intrinsic rank a hypothesis and investigates it empirically; it is not a theorem that every useful adaptation is low-rank or only changes behavior. A rank-one update cannot represent the 2×2 identity exactly.

Larger rank expands the representable update family and parameter/optimizer storage. Validation decides whether that helps; overfitting does not monotonically increase just because rank does. Choice of data, regularization, optimization and target layers also matters. QLoRA (Dettmers et al., 2023) combines a quantized frozen base with trainable adapters; that does not mean gradients or all training state use four bits.

Avoid fine-tuning as the first response when (1) needed facts change frequently and require provenance, (2) the task lacks a reliable evaluation set, or (3) the problem is retrieval, permissions or tool correctness rather than the model’s response policy. Retrieval can supply current evidence; fine-tuning can also learn facts, with no guarantee of reliable or erasable recall. Neither makes every answer accurate or cited. Narrow training may regress other capabilities; test task and general slices, and evaluate whether a representative data mixture helps instead of declaring that mixing always prevents forgetting.

S124 · Serving

Prompts: explain prefill versus decode, continuous batching and speculative-decoding regressions.

Prefill processes the prompt’s positions and constructs keys/values; decode produces successive tokens while attending to retained history. In a weight-read-only approximation, a dense projection performs about 2 * batch * d_in * d_out FLOPs while reading bytes_per_weight * d_in * d_out weight bytes. Intensity is therefore about 2 * batch / bytes_per_weight FLOPs per byte. Increasing reuse can move the bottleneck. Activations, KV reads, attention arithmetic, kernel efficiency, communication and hardware are excluded from that simple estimate.

Small-batch decode is often bandwidth-sensitive; prefill often has higher arithmetic intensity. Neither is universally bandwidth- or compute-bound. Long contexts can make attention/cache work significant, and sufficiently large decode batches reuse weights. Quantization changes bandwidth/storage and numerical error; evaluate the actual structured-output/code/safety slices rather than assuming a flat quality loss or an inevitable regression.

Continuous batching admits new requests as others finish instead of waiting for a whole fixed batch. The PagedAttention paper explains how KV fragmentation and duplication limit batching. A cache is important but not the only memory consumer: reserve weights, activations, workspace, allocator overhead and headroom. Use KV heads, not automatically query heads, for GQA/MQA sizing.

Speculative decoding verifies proposed tokens with a target model; its exact sampling algorithm preserves the target distribution under the paper’s assumptions. It is not simply accepting plausible draft text. Poor acceptance, expensive drafting/verification or scheduler interactions can erase the benefit. Compare measured end-to-end latency and output quality under the actual implementation; the paper’s acceleration is not a universal speedup.

S125 · Multimodal

Prompts: distinguish dual encoders from generative VLMs; explain projection; identify extraction failures.

CLIP (2021) learns image/text representations from paired data. A dual encoder embeds the image and candidate text separately and scores similarity; it does not itself generate an invoice response token by token. LLaVA (2023, v2) connects a vision encoder with an LLM for generative instruction following. A learned projection/connector can align visual features with the language model’s embedding width and training distribution. It is an architectural choice, not proof that all vision encoders require the same separate projection layer.

For an explicitly assumed 224×224 image split into 16×16 non-overlapping patches, there are 14 * 14 = 196 patches. Special tokens, multiple crops, pooling, resampling and vendor image billing can change the count. It is not an API token-price estimate. With a 10-second video sampled at 2 frames/second, this simplified rule gives 3,920 patch positions before compression and special tokens; temporal subsampling can miss events between frames. Audio systems may encode waveform/spectrogram segments or pass through speech recognition; background noise, accent and timestamp alignment create different failure modes from image patches.

Small print, downsampling, blur, rotation, occlusion, ambiguous layout, OCR error and generative inference can each corrupt an invoice total. “One digit smaller than a grid cell” is not a general proof of irreversible loss: encoders can encode within-patch detail. Cropping or using higher-quality input may help but cannot reconstruct information absent from the capture.

A safer extraction pipeline retains the source region, uses an evaluated OCR/document model if appropriate, validates the schema/currency and line-item arithmetic, and escalates ambiguous or high-consequence cases. OCR confidence is model-dependent and must be calibrated; it is not guaranteed per-character truth. Even a consistent total can be wrong if every value is scaled together. Treat image text, transcripts and retrieved documents as untrusted data. OWASP guidance includes multimodal injection; delimiters alone do not secure a model. Enforce permissions and human approval for consequential actions outside the model.

Hands-on drill · 30 min

Run the four Python blocks in order or separately as indicated by their imports. Every dataset below is synthetic. Their assertions establish only the stated finite checks; no printed ratio is a measured model-service speedup.

Step 1 — every boundary is lossy (7 min)

The heading preserves the old bookmark, but the universal claim is false. Compare deliberately lossy projection with a required-field handoff and a fail-closed validator. Unlike the original helper, serialise must use its argument, not a global dictionary.

# Structured fields are preserved by policy, not by the number of agents.
FULL = {
    "intent": "plan a migration",
    "constraints": "no downtime; two engineers",
    "prior_attempts": "dual write failed",
    "tone": "concise tradeoffs",
}
REQUIRED = ("intent", "constraints", "prior_attempts")
 
def serialise(state, fields):
    return {key: state[key] for key in fields if key in state}
 
def validate_handoff(state):
    missing = sorted(set(REQUIRED) - state.keys())
    if missing:
        raise ValueError(f"missing: {missing}")
    return state
 
lossy = serialise(FULL, ("intent", "constraints"))
try:
    validate_handoff(lossy)
except ValueError as exc:
    assert str(exc) == "missing: ['prior_attempts']"
else:
    raise AssertionError("loss was silently accepted")
kept = serialise(FULL, REQUIRED)
for _ in range(8):
    kept = validate_handoff(serialise(kept, REQUIRED))
assert kept == {key: FULL[key] for key in REQUIRED}
assert serialise({"intent": "different"}, ["intent"]) == {"intent": "different"}
# Explicit scheduling/cost model: supervisor initial + N workers + final merge.
def call_budget(workers):
    if type(workers) is not int or workers < 1:
        raise ValueError("positive integer workers required")
    return workers + 2
assert [call_budget(n) for n in (1, 2, 4)] == [3, 4, 6]
print("handoff: missing prior_attempts rejected; required fields survive 8 hops")
print("assumed calls for 1/2/4 workers:", [call_budget(n) for n in (1, 2, 4)])

Answer: two retained fields lose the prior failed attempt; three required fields survive eight serializations exactly. This is a counterexample to inevitable per-agent loss, not a test of summarization quality. The N+2 call budget is one architecture’s assumption, not a universal receive/report cost. Production handoffs also need types, provenance, authorization and semantic validation.

Step 2 — judge bias is measurable (7 min)

Use exact probabilities rather than a lucky random sample. Equal-quality answers have a baseline 1/2 preference; the simulated judge adds 0.15 for first position and 0.12 for greater length. Swap answer identity and its length together. Real judges need paired observations and confidence intervals, not this imposed probability model.

from fractions import Fraction as F
 
def probability_a(first, lengths, position=F(15, 100), verbosity=F(12, 100)):
    p = F(1, 2) + (position if first == "A" else -position)
    p += verbosity * ((lengths["A"] > lengths["B"]) -
                      (lengths["A"] < lengths["B"]))
    return max(F(0), min(F(1), p))
 
def balanced(lengths):
    return (probability_a("A", lengths) + probability_a("B", lengths)) / 2
 
assert probability_a("A", {"A": 100, "B": 100}) == F(65, 100)
assert balanced({"A": 100, "B": 100}) == F(1, 2)
assert balanced({"A": 100, "B": 400}) == F(38, 100)
# Human-labeled factual rubric: synthetic values, not an LLM call.
golden = [({"amount_cents": 1250}, 1250), ({"amount_cents": 1200}, 1250),
          ({"amount_cents": "1250"}, 1250), ({}, 1250)]
def exact_amount(answer, expected):
    value = answer.get("amount_cents")
    return type(value) is int and value == expected
assert [exact_amount(a, e) for a, e in golden] == [True, False, False, False]
print("A-first/equal:", float(F(65, 100)), "balanced/equal:", float(balanced({"A":100,"B":100})))
print("balanced/B-longer:", float(balanced({"A":100,"B":400})))
print("exact-amount checks: 1 pass, 3 rejected")

Answer: order balancing cancels the deliberately symmetric position term here but leaves a 0.38 A preference when B is longer. Interaction, asymmetric bias, stochastic sampling and saturation can break exact cancellation for real judges. Even this helper stops cancelling after clipping: with A longer, position=0.45 and verbosity=0.20, its two A probabilities are 1 and 0.25, averaging 0.625; removing position bias would give 0.70. Thus balancing is not a proof of an unbiased evaluator even with symmetric pre-clipping terms. The deterministic amount validator catches these four schema/value cases, but a wrong gold value would make its score misleading. Use both task checks and calibrated quality judgments.

Step 3 — adapter and cache arithmetic (8 min)

For d_in = d_out = 4096, rank 8 uses 65,536 parameters versus 16,777,216, or 0.390625% of one full matrix. That excludes the frozen base, other adapted layers, gradients and optimizer state. For 32 layers, 8 KV heads, 128 head dimension, 2,048 tokens, 4 requests and 2 bytes per scalar, the cache is 1,073,741,824 bytes = 1 GiB, not 1 GB. Four times as many KV heads gives four times the cache under this representation.

import numpy as np
 
def lora_params(d_in, d_out, rank):
    if any(type(v) is not int or v <= 0 for v in (d_in, d_out, rank)):
        raise ValueError("positive dimensions required")
    return rank * (d_in + d_out)
 
def kv_bytes(layers, kv_heads, head_dim, seq, batch, bytes_per=2):
    dims = (layers, kv_heads, head_dim, seq, batch, bytes_per)
    if any(type(v) is not int or v <= 0 for v in dims):
        raise ValueError("positive dimensions required")
    return 2 * layers * kv_heads * head_dim * seq * batch * bytes_per
 
assert lora_params(4096, 4096, 8) == 65536
assert kv_bytes(32, 8, 128, 2048, 4) == 2**30
W = np.array([[1., 2., 3.], [4., 5., 6.]])
A = np.array([[1., 0., -1.]])
B = np.array([[2.], [3.]])
x = np.array([2., 1., 0.])
np.testing.assert_array_equal((W + B @ A) @ x, W @ x + B @ (A @ x))
assert np.linalg.matrix_rank(B @ A) == 1
assert np.linalg.matrix_rank(np.eye(2)) == 2  # cannot be a rank-one update
# Allocate only a tiny cache to check representation, not a 1-GiB fixture.
tiny = np.empty((2, 3, 2, 5, 4, 7), dtype=np.float16)
assert tiny.nbytes == kv_bytes(3, 2, 7, 4, 5)
assert [2*b/2 for b in (1, 8, 32)] == [1., 8., 32.]
print("rank-8 parameters:", lora_params(4096,4096,8), "cache GiB:", kv_bytes(32,8,128,2048,4)/2**30)
print("LoRA merged/unmerged outputs:", ((W+B@A)@x).tolist(), "tiny cache bytes:", tiny.nbytes)

Answer: the projection outputs [8, 19]; associativity explains the equality before quantization/rounding complications. This fixture uses an effective scale alpha/r = 1; for any other chosen scale, multiply the update in both merged and unmerged forms. For example scale 2 gives [12, 25], not [8, 19]. Multiple unmerged adapters can share the same frozen W while selecting distinct A/B pairs, but merging adapters with different effective updates creates different weight matrices; it does not make one merged matrix implement both tasks. A small trainable adapter does not remove the serving base or KV cache. The intensity figures 1/8/32 FLOPs per weight byte are a simplified reuse model, not measured prefill/decode speed.

Step 4 — batching regimes (8 min)

Assume all jobs arrive at time zero, occupy one of two slots, and require known positive integer durations. Fixed batches start only when the previous batch finishes; continuous scheduling fills a freed slot with the next job. Count idle capacity over the same full makespan, including the drain tail. This is an abstract slot model, not GPU token timing.

import heapq
from itertools import product
 
def validate_jobs(lengths, slots):
    if type(slots) is not int or slots <= 0:
        raise ValueError("positive slots required")
    if any(type(t) is not int or t <= 0 for t in lengths):
        raise ValueError("positive durations required")
 
def static_batching(lengths, slots):
    validate_jobs(lengths, slots)
    elapsed = sum(max(lengths[i:i+slots]) for i in range(0, len(lengths), slots))
    return elapsed, slots * elapsed - sum(lengths)
 
def continuous_batching(lengths, slots):
    validate_jobs(lengths, slots)
    if not lengths:
        return 0, 0
    available = [0] * slots
    for duration in lengths:
        start = heapq.heappop(available)
        heapq.heappush(available, start + duration)
    elapsed = max(available)
    return elapsed, slots * elapsed - sum(lengths)
 
jobs = [2, 8, 2, 2]
assert static_batching(jobs, 2) == (10, 6)
assert continuous_batching(jobs, 2) == (8, 2)
for n in range(6):
    for seq in product((1, 2, 4), repeat=n):
        for slots in (1, 2, 3):
            for schedule in (static_batching, continuous_batching):
                elapsed, idle = schedule(seq, slots)
                assert idle >= 0 and slots * elapsed == sum(seq) + idle
assert continuous_batching([], 2) == (0, 0)
print("static / continuous (elapsed, idle):", static_batching(jobs,2), continuous_batching(jobs,2))
print("model makespan ratio:", 10/8, "not a GPU speedup")

Answer: static work lasts 10 units with 6 idle slot-units; continuous work lasts 8 with 2 idle. Both do 14 units of useful work. Equal-length jobs can remove most or all of the benefit. Prefill admission, memory pressure, fairness, queues, arrivals and hardware batching efficiency are absent; a real scheduler must balance throughput against tail latency and admission safety.

Answer constraints and corrections

Misconception: fine-tuning is the reliable way to install internal facts. It can change factual responses, but weights are not an auditable document store. If a policy changes tomorrow, retrieval with access checks can expose a current passage without retraining; it can still retrieve the wrong passage or generate an unsupported answer. Tune response behavior when evaluation justifies it, and test factuality/grounding separately. No “fine-tuning never learns facts” or “retrieval solves hallucinations” rule is valid.

Quiz and worked answers

1. Why might another agent worsen quality? A poorly specified handoff can omit constraints or provenance; duplicated/looping work can exhaust budgets. A validated structured boundary can preserve required information, so compare quality and cost rather than assume every hop loses context. Separate tools or parallel subtasks can justify added coordination.
2. Which judge bias does order swapping fix? It diagnoses and can mitigate position bias. Exact cancellation occurs in the additive symmetric toy, not by construction for every judge. Verbosity and self-preference require distinct checks, a property-based rubric and human calibration.
3. Why can LoRA work despite fewer parameters? It restricts the update to a low-rank family that empirically fits some adaptations while freezing the base. The parameter reduction is exact; sufficient quality is task-dependent. The rank-two identity counterexample disproves universal rank-one adequacy.
4. Why do prefill and decode often differ? Many prompt positions can reuse weights in prefill; small-batch decode has less weight reuse and repeatedly reads a growing cache. Batch, sequence, hardware and kernels decide the bottleneck; the simple FLOPs/byte calculation is not a universal roofline measurement.
5. An invoice total is wrong about one time in twenty: what now? First measure representative cases and categorize capture, OCR, layout and model errors; the observed fraction does not identify the cause. Preserve crops/evidence, test a document extractor, validate integer cents/currency and arithmetic, and route uncertain or consequential cases to review. Arithmetic can catch inconsistency but not a consistent wrong extraction. Calibrate confidence on labeled data, constrain tool permissions and treat embedded instructions as untrusted. No prompt, crop or OCR component alone establishes safety.

Gap analysis + next week preview · 10 min

Write one ordinary case and one counterexample for each of the five subjects. If you can name LoRA but cannot derive its shapes, return to the matrix example; if you can recite “continuous batching” but cannot reconcile idle capacity, redraw the four-job timeline. Retest a changed dimension, length sequence or judge interaction, not only the same fixture.

Next, Week 26 carries these boundaries into shortener, chat, feed and AI-product designs. This is curriculum navigation, not a scheduled event or proof that the prior 25 weeks were completed.

Corrective source notes

Primary sources are linked beside the claims: Anthropic workflow guidance (published 2024-12-19, live page checked 2026-10-08), Zheng et al. v4 (2023-12-24), LoRA HTML labeled Version 2, PagedAttention v1 (2023-09-12), speculative decoding ICML 2023, CLIP ICML 2021, LLaVA v2 (2023-12-11), and live OWASP guidance checked 2026-10-08. Model-paper results support architecture and reported experimental scope, not reproduction here. All four printed programs are finite local models; no model quality, paid service, agent framework, quantized training or GPU serving claim is asserted as executed.