LLM Eval — Judges, RAGAS, Golden Sets

Why this session matters

Part of a 24-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.

Why this session matters

Evaluation is the unsexy half of LLM work — and the half that decides whether your shiny demo survives contact with real users. You can't ship what you can't measure.

Agenda

• Why evaluating LLMs is genuinely harder than classical ML
• Static benchmarks — MMLU, HumanEval, GSM8K — and their limits
• LLM-as-judge — pairwise, score-based, the bias gotchas
• RAGAS, TruLens — RAG-specific eval frameworks
• Inspect, OpenAI Evals — building your own eval harness

Pre-read (skim before the session)

• Anthropic — Challenges in evaluating AI systems
• RAGAS docs — Metrics
• Judging LLM-as-a-Judge (Zheng et al., 2023)
• Inspect AI — UK AISI eval framework

Deep dive

1. Why LLM eval is hard

Classical ML: one label, one prediction, clean metric (accuracy, AUC). LLMs:

• Open-ended output — no single right answer. "Summarise this" has 1000 acceptable answers.
• Multi-dimensional quality — factuality, fluency, safety, tone, helpfulness; can trade off.
• Reference-free settings — you often don't have a ground truth ("write a marketing email").
• Long-tail failures — the model is fine 99% of the time and catastrophically wrong on edge cases.
• Contamination — test set leaked into pre-training corpus → benchmark inflated.

You can't compute "accuracy" against a list of strings and call it a day.

2. The 4 layers of eval

[ Unit-level ]      → does the prompt produce well-formed JSON?
[ Behavioural ]     → does it refuse the jailbreak? answer the maths question?
[ Capability ]      → MMLU, HumanEval, MATH on held-out data
[ User-outcome ]    → did the user accept the suggestion? task success?

The bottom two cost the most and matter the most. Most teams over-invest in capability benchmarks and under-invest in user-outcome metrics.

3. Static benchmarks — useful, but...

Rule: use benchmarks to exclude models, not to pick them. If MMLU is < 60% your candidate, drop it. Above some threshold, benchmarks stop correlating with what you actually need.

4. LLM-as-Judge — the workhorse

Use a strong model (GPT-4-class) to evaluate outputs of another model. Three flavours:

• Pairwise — show judge two outputs A and B, ask which is better. Most reliable.
• Scalar — rate output 1–10 on dimension X. Easy but noisy.
• Rubric-based — multi-criteria scoring against a written rubric. Best for production.

Pairwise example:

You are a judge. Compare two assistant responses to the same prompt.
Decide which is better on: helpfulness, accuracy, conciseness.
Output JSON: {"winner": "A" | "B" | "tie", "reason": "..."}
Prompt: {user_prompt}
Response A: {output_a}
Response B: {output_b}

5. Judge biases (this will burn you)

• Position bias — first option wins more often. Mitigation: swap order, average.
• Length bias — longer answers preferred even when worse. Mitigation: length-controlled rubric.
• Self-preference — model judges its own family higher. Mitigation: use a different model family as judge.
• Verbosity bias — flowery > terse. Mitigation: explicit rubric criterion.
• Authority bias — confident wrong > tentative correct. Mitigation: factuality sub-score.

Validate the judge against ~200 human-labelled pairs before you trust it for thousands of evals.

6. RAGAS — RAG-specific metrics

For retrieval-augmented systems:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
result = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness, answer_relevancy],
)

These give you a numeric, comparable score per retrieval/generation strategy. Crucial for the eval loop in RAG (see S13).

7. TruLens, DeepEval, Inspect

• TruLens — instrumented runtime; collects traces; defines "feedback functions" (RAGAS-ish).
• DeepEval — pytest-style assertions for LLM output. assert_relevance(), assert_no_hallucination().
• Inspect (UK AISI) — capability + safety eval framework, used for frontier-model red-teaming. Plugin model.
• OpenAI Evals — open-source harness, plug-in eval types.
• Promptfoo — YAML-driven A/B prompt eval; great for prompt regression.

8. Building your own eval set

Don't skip this. A 100-example, hand-crafted eval set tailored to your product beats any public benchmark.

Process:

1. Mine real user prompts (anonymise!).
2. Bucket by intent (10–15 buckets is enough).
3. For each bucket, write 5–10 gold-standard answers (or rubrics).
4. Add adversarial examples — your top 20 known failure modes.
5. Version the set; checksum it; track it like a dataset, not a doc.

Re-run the whole set on every model/prompt change. Track scores in MLflow.

9. Online vs offline eval

• Offline — fixed dataset, runs on every release. Fast, repeatable, but stale.
• Online — production traffic, label by user reaction (👍/👎, accept-rate, edit-distance to final answer). Slow, biased by current cohort, but real.

Both. Offline catches regressions before ship; online catches reality drift.

10. The eval flywheel

[ Real user fails ] → [ Add to eval set ] → [ Run all candidates ] → [ Pick winner ] → [ Ship ]
        ▲                                                                                   │
        └───────────────────────────────────────────────────────────────────────────────────┘

Every bug becomes a permanent test. After 6 months you have 2000 examples that capture exactly what your product needs. That set is your moat.

11. Cost-aware eval

LLM-as-judge is expensive. Budgeting tips:

• Sample, don't run-all-on-everything.
• Cache judge results keyed on (prompt, output_a, output_b, judge_model).
• Use a cheaper "screening" judge → escalate ambiguous cases to a stronger judge.
• Score per-dollar: improvements aren't free; track quality_delta / cost_delta.

12. Reality check

A pragmatic minimum eval stack for a startup:

• 100 hand-crafted prompts with rubrics (versioned in git).
• A pytest job that runs them on every PR using promptfoo or homemade.
• Pairwise LLM-judge with order-swap, against GPT-4o.
• Production logging of (input, output, model_version, user_feedback).
• Weekly review of bottom-decile responses → add to eval set.

This is enough. Buy the platform when you outgrow it, not before.

Reading material

Books:

• AI Engineering — Chip Huyen (the entire evaluation chapter; the best modern treatment)
• Building LLMs for Production — Bouchard & Peters (the eval chapter with hands-on RAGAS examples)
• Speech and Language Processing, 3rd ed. — Jurafsky & Martin (the classical NLP eval chapter; ROUGE, BLEU, BERTScore origins)

Papers:

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al. 2023 — the canonical LLM-as-judge paper.
• RAGAS: Automated Evaluation of Retrieval Augmented Generation — Es et al. 2023 — faithfulness + answer relevance + context recall.
• HELM: Holistic Evaluation of Language Models — Liang et al. 2022 (Stanford CRFM) — the multi-axis benchmark framework.
• BIG-bench — Srivastava et al. 2022 — 200+ tasks from 450 collaborators.
• GPT-4 Technical Report — OpenAI 2023 (evaluation sections) — how a frontier lab discloses (or doesn't disclose) evals.

Official docs:

• RAGAS — metrics docs — faithfulness, answer relevance, context precision/recall.
• Inspect AI — UK AISI — the UK AI Safety Institute's framework; the modern standard.
• Promptfoo docs — declarative prompt eval, regression tests.
• OpenAI Evals — the format OpenAI uses internally.
• LangSmith docs — Evaluation — hosted LLM tracing + eval.

Blog posts:

• Eugene Yan — Patterns for Building LLM-based Systems & Products: Evals — the single best practitioner essay.
• Hamel Husain — Your AI Product Needs Evals — the canonical "how to actually run evals in production" post.
• Anthropic — Challenges in evaluating AI systems — the honest "why this is hard" post from Anthropic.

In-depth research material

• RAGAS — github.com/explodinggradients/ragas — ~10k ★, RAG-specific evals.
• DeepEval — github.com/confident-ai/deepeval — ~7k ★, pytest-flavoured LLM evals.
• OpenAI Evals — github.com/openai/evals — ~16k ★, the open eval framework + dataset.
• LM Evaluation Harness — github.com/EleutherAI/lm-evaluation-harness — ~9k ★, the de-facto OSS benchmark runner (HF Open LLM Leaderboard uses it).
• Inspect AI — github.com/UKGovernmentBEIS/inspect_ai — the UK AISI framework, now adopted by NIST.
• HELM — github.com/stanford-crfm/helm — ~2k ★, Stanford holistic eval.
• Chatbot Arena — LMSys — crowdsourced head-to-head rankings.
• Vellum — LLM Leaderboard — the most current production-facing leaderboard.
• Hamel Husain — Field guide to rapidly improving AI products — the workflow that ties evals + error analysis + fixes.
• Eugene Yan — LLM evaluators (a survey) — the meta-paper on LLM-as-judge biases.

Videos

• A Practitioner's Guide to LLM Evals — Hamel Husain — Hamel Husain · 1 h 06 min — the talk version of his canonical blog post; watch this first.
• LLM Evaluation Explained — Maven AI Eng course · 1 h 21 min — the workshop-format walkthrough.
• Evaluating LLM Systems with Eugene Yan — Eugene Yan · 50 min — from the patterns-for-LLM-systems author.
• LangSmith Evaluations Deep Dive — LangChain · 33 min — hands-on with the most popular hosted eval tool.
• Building Reliable AI Systems: Evals — Shreya Shankar (Stanford MLSys) — 1 h — the academic-rigour view on eval systems.

LeetCode — Evaluate Division

• Link: https://leetcode.com/problems/evaluate-division/
• Difficulty: Medium
• Why this problem: Build a graph of equations and answer arbitrary queries — exact shape of evaluating one model output against a chain of judge criteria.
• Time-box: 30 minutes. Look up the editorial only after.

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

• Explain why open-ended LLM output makes eval qualitatively harder than classical ML.
• List the 4 layers of eval (unit, behavioural, capability, user-outcome).
• Design a pairwise LLM-judge prompt with bias mitigations.
• Pick the right RAGAS metric (faithfulness vs context_precision vs answer_relevancy) for a given failure.
• Sketch the 5-step "eval flywheel" loop.
• Solve evaluate-division — graph traversal of weighted edges; mirrors chained eval criteria.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.