R25 · Week 25 Recall & Drill

Weekly revision · Week 25 · Covers 5 sessions from Mon–Fri.

Sessions covered

• S121 — Multi-Agent Orchestration — LangGraph, CrewAI Patterns
• S122 — LLM Evaluation — LLM-as-Judge, RAGAS, Golden Sets
• S123 — Fine-Tuning — LoRA, QLoRA, PEFT, When NOT to Fine-Tune
• S124 — LLM Serving — KV Cache, Batching, Speculative Decoding
• S125 — Multimodal LLMs — CLIP, VLMs, Audio, Video

90-min structure

Blank-page reconstruction · 30 min

S121 · Multi-Agent Orchestration

1. Name the three canonical orchestration patterns.
2. Say when to split a task across agents rather than improve one.
3. Give the two runaway failure modes specific to multiple agents.

Gotcha you probably forgot: the supervisor's instructions should be small and close to deterministic. A supervisor with an elaborate prompt starts doing the work itself instead of routing, and its routing decisions become unpredictable — which is the failure that produces duplicated work and handoffs that never terminate.

S122 · Evaluation

1. Distinguish reference-based, reference-free, and task-specific metrics.
2. Name three biases a pairwise judge exhibits and a mitigation for each.
3. Say why a small hand-graded set beats a large generated one on day one.

Gotcha you probably forgot: the judge must decode deterministically. A judge that samples gives different verdicts on identical inputs, so a regression check becomes a coin flip and you cannot distinguish a real quality change from judge noise. This is non-negotiable in a way that is easy to skip when copying a prompt from a generation pipeline.

S123 · Fine-Tuning

1. State in one sentence why updating a tiny fraction of parameters works.
2. Say what the rank parameter does to capacity, trainable size, and overfitting risk.
3. Name three situations where fine-tuning is the wrong answer despite available budget.

Gotcha you probably forgot: mix general instruction data into a specialised fine-tuning set. Training only on narrow task data degrades general capability, so the model gets better at your task and measurably worse at everything else — and that regression will not appear in an evaluation that only measures your task.

S124 · Serving

1. Explain why the two phases of inference saturate different resources.
2. Describe what continuous batching fixes that fixed batching cannot.
3. Say when speculative decoding hurts rather than helps.

Gotcha you probably forgot: quantisation damage is not uniform across capabilities. An aggregate benchmark can move barely at all while code generation or structured output degrades badly, because those tasks depend on precise low-probability distinctions. Evaluate quantisation on the capability you actually ship, not on a general score.

S125 · Multimodal

1. Distinguish the dual-encoder architecture from a generative vision-language model.
2. Say what the projection layer does and why the vision encoder cannot feed the language model directly.
3. Give two failure modes of extracting structured values from images.

Gotcha you probably forgot: text rendered in an image is instruction-shaped input arriving through a channel most systems do not sanitise. An image containing written directions can steer a model that would have refused the same text typed directly, which makes image input an injection surface rather than merely a perception problem.

Hands-on drill · 30 min

Task: measure handoff loss, expose judge bias, size adapters and caches, and compare batching regimes.

mkdir -p ~/projects/w25-drill && cd ~/projects/w25-drill

Step 1 — every boundary is lossy (7 min)

# handoff.py
FULL = {
    "user_intent": "wants a migration plan, is risk-averse, has a hard March deadline",
    "constraints": "no downtime, two engineers, existing Postgres 13",
    "prior_attempts": "tried a dual-write approach last year and it corrupted data",
    "tone": "prefers blunt trade-offs over reassurance",
}
 
def serialise(state, budget_fields):
    """A handoff compresses shared state into a message of bounded size."""
    return {k: FULL[k] for k in list(FULL)[:budget_fields]}
 
print(f"{'agents in chain':>16} {'fields surviving':>18} {'lost context'}")
for n_agents in (1, 2, 3, 4):
    surviving = max(0, len(FULL) - (n_agents - 1))
    kept = serialise(FULL, surviving)
    lost = [k for k in FULL if k not in kept]
    print(f"{n_agents:>16} {surviving:>18} {', '.join(lost) or '-'}")
 
print("\nEach boundary drops whatever was implicit rather than written down.")
print("The prior failed attempt is exactly the sort of detail that never makes it")
print("into a handoff summary, and its loss produces a plan that repeats it.")
 
# The cost side.
print(f"\n{'agents':>8} {'llm calls':>11} {'relative token cost':>21}")
for n in (1, 2, 3, 5):
    calls = 1 if n == 1 else n * 2          # each worker: receive + report
    print(f"{n:>8} {calls:>11} {calls * 1.0 / 1:>20.1f}x")
print("\nAdd an agent only when the subtasks are genuinely independent and each")
print("needs a different tool set -- not to give one model a tidier prompt.")

Expected outcome: the fields that vanish first are the implicit ones — the prior failed attempt, the tone preference — because a handoff summary records the task rather than the context around it. The cost table shows the other half: agents multiply calls and tokens. Together these are the argument that a single agent with the full context usually beats a crew, and that the reason to split is genuinely independent subtasks needing different tools, not a desire for tidier prompts.

Step 2 — judge bias is measurable (7 min)

# judge_bias.py
import random
 
random.seed(0)
PAIRS = [("A", "B")] * 200
 
def biased_judge(first, second, position_bias=0.15, verbosity_bias=0.0,
                 len_first=100, len_second=100):
    """Model a judge that mildly prefers whichever answer it reads first,
    and mildly prefers the longer one."""
    p = 0.5 + position_bias
    if len_second > len_first:
        p -= verbosity_bias
    else:
        p += verbosity_bias
    return first if random.random() < p else second
 
def rate(**kw):
    wins = sum(biased_judge("A", "B", **kw) == "A" for _ in PAIRS)
    return wins / len(PAIRS)
 
print(f"single ordering, equal length      A wins {rate():.1%}  (truth: 50%)")
print(f"single ordering, B is longer       A wins "
      f"{rate(verbosity_bias=0.12, len_second=400):.1%}")
 
# Mitigation: present each pair in both orders and average.
def swapped_rate(**kw):
    a = sum(biased_judge("A", "B", **kw) == "A" for _ in PAIRS)
    b = sum(biased_judge("B", "A", **kw) == "A" for _ in PAIRS)
    return (a + b) / (2 * len(PAIRS))
 
print(f"both orderings averaged            A wins {swapped_rate():.1%}  (truth: 50%)")
print("\nPosition bias cancels when you evaluate each pair in both orders and")
print("average. Verbosity bias does not cancel that way -- it needs a rubric that")
print("scores specific properties rather than asking which answer is better.")

Expected outcome: with a single presentation order the judge reports a clear winner between two equally good answers, which is entirely artefact. Averaging over both orders cancels the positional component and recovers the truth. Verbosity bias survives that mitigation because it is tied to the content rather than the position, so it needs a different fix — an explicit rubric scoring named properties instead of an open "which is better" question. The general lesson is that judges fail through specific, measurable, fixable biases rather than through vague subjectivity.

Step 3 — adapter and cache arithmetic (8 min)

# sizing.py
def lora_params(d_in, d_out, r):
    return r * (d_in + d_out)
 
d = 4096
print(f"{'rank':>5} {'adapter params per matrix':>27} {'vs full matrix':>16}")
full = d * d
for r in (4, 8, 16, 64, 256):
    p = lora_params(d, d, r)
    print(f"{r:>5} {p:>27,} {p/full:>15.2%}")
 
print("\nThe update is constrained to a low-rank form, which is enough because the")
print("change fine-tuning needs to make is itself low-rank -- it adapts behaviour")
print("rather than rewriting the representation. Higher rank means more capacity")
print("and more overfitting risk on a small dataset.\n")
 
def kv_bytes(layers, heads, head_dim, seq, batch, bytes_per=2):
    return 2 * layers * heads * head_dim * seq * batch * bytes_per
 
L, H, HD = 32, 32, 128
print(f"{'batch':>6} {'seq len':>9} {'KV cache':>12}")
for batch in (1, 8, 32):
    for seq in (2048, 8192, 32768):
        print(f"{batch:>6} {seq:>9} {kv_bytes(L,H,HD,seq,batch)/1e9:>10.2f}GB")
 
print("\nThe cache grows linearly in both batch and sequence length, and it is the")
print("real constraint on how many concurrent users a device can hold -- not the")
print("weights, which are fixed.")

Expected outcome: the adapter table shows a tiny fraction of parameters carrying the update, and makes the rank trade-off concrete — more rank is more capacity and more room to overfit a small dataset. The cache table shows why serving capacity is a memory question: weights are a fixed cost paid once, while the cache scales with both concurrency and context length and is what actually runs out. That is why memory management for the cache, rather than raw compute, determines how many users a device can serve.

Step 4 — batching regimes (8 min)

# batching.py
import random
random.seed(1)
 
requests = [(i, random.choice([20, 40, 600, 80, 1000])) for i in range(16)]
 
def static_batching(reqs, batch_size=4):
    """Every slot waits for the longest generation in its batch."""
    total, wasted = 0, 0
    for i in range(0, len(reqs), batch_size):
        group = reqs[i:i + batch_size]
        longest = max(t for _, t in group)
        total += longest
        wasted += sum(longest - t for _, t in group)
    return total, wasted
 
def continuous_batching(reqs, slots=4):
    """A finished slot is refilled immediately."""
    import heapq
    q = list(reqs)
    active, clock, wasted = [], 0, 0
    while q or active:
        while q and len(active) < slots:
            _, t = q.pop(0)
            heapq.heappush(active, clock + t)
        clock = heapq.heappop(active)
    return clock, wasted
 
s_total, s_wasted = static_batching(requests)
c_total, _ = continuous_batching(requests)
print(f"static batching   : {s_total:>6} steps, {s_wasted:>6} slot-steps idle")
print(f"continuous batching: {c_total:>6} steps, slots refilled on completion")
print(f"speedup           : {s_total / c_total:.2f}x")
print("\nGeneration lengths vary enormously between requests, so a fixed batch is")
print("held hostage by its longest member and most slots sit idle. Refilling each")
print("slot the moment it frees is the whole idea, and it is a scheduling change")
print("rather than a faster kernel.")

Expected outcome: static batching wastes a large number of slot-steps because a short request finishes early and its slot idles until the longest generation in the batch completes, and generation lengths differ by an order of magnitude in any realistic mix. Continuous batching refills each slot on completion and closes most of that gap. The point to internalise is that this is a scheduler change producing a large throughput gain with identical arithmetic — the same shape of insight as tiled attention.

Gap analysis + next week preview · 10 min

• Could you argue against adding an agent? That argument is more useful than knowing three frameworks.
• Did the judge drill land? Any evaluation you cannot reproduce is decoration.
• Can you size a key-value cache from memory? It is the number that decides serving capacity.

Next week (S126–S130) is the closing arc: the four-phase system design framework, then three canonical designs — a URL shortener, a chat system, and a newsfeed — and finally an end-to-end AI chat product that pulls the entire six months together. Everything from this week reappears there as a box on a diagram you have to defend.

Part of the 6-month evergreen learning plan.