S122 · LLM Evaluation — LLM-as-Judge, RAGAS, Golden Sets

Module M14: LLMs & Applications · Session 122 of 130 · Track: LLM ⚖️

What you'll be able to do after this session

• Explain LLM Evaluation to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S117 — RAG I — Chunking Strategies & Indexing

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · How to Evaluate LLMs — DeepLearning.AI's 12-min primer on why LLM eval is hard.
• 🎥 Deep dive (30–60 min) · LLM Evaluation with RAGAS — full walkthrough of the RAGAS framework with real code.
• 🎥 Hands-on demo (10–20 min) · LLM-as-a-Judge with OpenAI Evals — build a judge in 15 min.
• 📖 Canonical article · OpenAI — Evals — the reference framework and eval registry.
• 📖 Book chapter / tutorial · Chip Huyen — Building LLM Applications for Production — the "Evaluation" section is the best distilled intro.
• 📖 Engineering blog · Anthropic — Evaluating AI Systems — practical playbook: golden sets, judges, rubrics.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Measuring quality of a non-deterministic system.

Evaluating a traditional ML model is easy: you have labels, you compute accuracy or F1, you're done. Evaluating an LLM is like grading essays — there's no single "right answer", the same question can have five equally good responses, and the grader (a human, or another LLM) is themselves imperfect. Worse, the same prompt with the same model can produce different outputs on different runs. You're measuring quality on a non-deterministic, unbounded-output system.

Three tools do most of the heavy lifting. Golden sets are a handpicked set of ~50-500 input/expected-output pairs — your regression test suite; whenever you tweak a prompt or model, you re-run them and diff. LLM-as-Judge uses another (usually stronger) LLM to score outputs against a rubric — "rate this answer 1-5 on faithfulness to the sources" — turning subjective judgement into automatable numbers. RAGAS is a specialised library for RAG systems that computes context precision, context recall, faithfulness, and answer relevance without needing ground truth for every field.

The gotcha you must carry forever: the judge is not the ground truth. If you use GPT-4 to judge GPT-4's own outputs, you get systematic bias (LLMs prefer verbose, hedge-y answers written in their own style). Always calibrate the judge against ~50 human-labelled examples before trusting it, and re-calibrate whenever you change judge model or rubric.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

LLM eval decomposes into three orthogonal axes: what you measure, who measures it, and when it runs.

Worked example — evaluating a RAG chatbot:

Concrete run: you have 100 golden (question, expected-answer, expected-source-doc) triples. On each release: (1) run the RAG pipeline on all 100, (2) compute the 4 RAGAS metrics via gpt-4o judge, (3) fail the CI if any metric drops by more than 3 percentage points. This is your "unit test" for a non-deterministic system.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Build a mini LLM-as-Judge that scores answer faithfulness against a source document.

# pip install openai pydantic
import os, json
from openai import OpenAI
from pydantic import BaseModel
 
client = OpenAI()
 
golden = [
    {
        "context": "The Eiffel Tower was built in 1889 for the World's Fair. It is 330 meters tall.",
        "question": "When was the Eiffel Tower built and how tall is it?",
        "answer_good": "It was built in 1889 and is 330 meters tall.",
        "answer_bad": "It was built in 1887 by Napoleon and is 400m tall.",
    },
]
 
class Verdict(BaseModel):
    faithful: bool
    unsupported_claims: list[str]
    score: int  # 1-5
 
JUDGE_PROMPT = '''You are a strict fact-checker. Given a CONTEXT and an ANSWER, decide if EVERY factual claim in the ANSWER is supported by the CONTEXT.
 
CONTEXT:
{context}
 
ANSWER:
{answer}
 
Return JSON: {{"faithful": bool, "unsupported_claims": [str], "score": 1-5 where 5=fully supported, 1=mostly fabricated}}'''
 
def judge(context: str, answer: str) -> Verdict:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        response_format={"type": "json_object"},
        messages=[{"role": "user",
                   "content": JUDGE_PROMPT.format(context=context, answer=answer)}],
    )
    return Verdict(**json.loads(resp.choices[0].message.content))
 
for row in golden:
    good = judge(row["context"], row["answer_good"])
    bad = judge(row["context"], row["answer_bad"])
    print(f"GOOD: score={good.score} faithful={good.faithful}")
    print(f"BAD:  score={bad.score} unsupported={bad.unsupported_claims}")

Observe when you run:

1. GOOD scores 5, faithful=True. BAD scores 1-2, faithful=False.
2. temperature=0 — critical, otherwise the judge gives different scores per run.
3. response_format=json_object — forces valid JSON, otherwise you spend hours parsing.
4. Judge cost: ~200 tokens per call ≈ $0.0001. You can run 10,000 judgements for$1.
5. The judge lists which claims are unsupported — this is your debugging goldmine.

Try this modification: add a human_label field to each golden row (Yes/No faithful). Run 20 examples through the judge, compute agreement with your human label. If agreement is < 85%, rewrite the prompt or upgrade to gpt-4o.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story — the OneNote Copilot judge that lied. A team (this actually happened at Microsoft) used GPT-3.5 to judge GPT-4 outputs for a note-summarisation product. The judge scored everything 4/5 average, engineers celebrated. Then real users started complaining. Root cause: GPT-3.5 was too weak to spot subtle factual errors in the summaries, and it systematically preferred longer answers. Fix: (a) upgrade judge to GPT-4, (b) add explicit "prefer concise, factually grounded answers" to the rubric, (c) require the judge to quote the exact source span for each claim it accepts.

Common bugs top teams handle for:

• Judge position bias — when comparing two answers, LLMs prefer the one shown first. Randomise order per call, run each pair twice with A/B swapped.
• Verbosity bias — LLMs prefer longer answers. Add "length is not quality" to the rubric; log answer length and check for correlation.
• Golden set decay — model gets better, golden set becomes trivially easy, ceiling effect hides regressions. Refresh 10% of golden set quarterly, add adversarial hard examples.
• Metric hacking — engineers optimise the metric, not the product. Always keep a small human-review set that doesn't get optimised against.
• Prod-eval drift — offline golden set is nothing like real user queries. Sample 100 real queries weekly, run them through the judge, alert if quality drops.

Anthropic's playbook (from their public eval docs): (1) start with 20 hand-graded examples on day one, (2) grow to 200 by week one, (3) automate the top 3 metrics that correlate with human judgement, (4) never trust a metric you haven't personally sanity-checked on 50 examples. Eval is a product, not a chore — invest engineer-weeks, not engineer-hours.

(e) Quiz + exercise · 10 min

1. Name three sources of non-determinism when evaluating an LLM output that don't exist when evaluating a traditional classifier.
2. Define faithfulness, context precision, and answer relevance in one sentence each.
3. What is position bias in LLM-as-Judge, and how do you mitigate it?
4. Why is a golden set of 50 hand-graded examples more valuable on day one than a golden set of 10,000 auto-generated ones?
5. When would you deliberately choose a weaker/cheaper model as your judge, despite the risk?

Stretch problem (connects to S117 — RAG): Take the RAG pipeline you built in S117. Design a golden set of 30 (question, expected-source-doc, expected-answer) triples for your domain. Compute the four RAGAS metrics. Which metric would fail first if you (a) swapped the embedding model? (b) reduced top-k from 5 to 2? (c) switched the generator from GPT-4 to GPT-3.5?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is LLM Evaluation? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S123): Fine-Tuning — LoRA, QLoRA, PEFT, When NOT to Fine-Tune
• Previous (S121): Multi-Agent Orchestration — LangGraph, CrewAI Patterns
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.