LLM Evaluation — LLM-as-Judge, RAGAS, Golden Sets

Why this session exists

A plausible answer is not a regression test. Evaluation makes a product's requirements inspectable: what is correct, for whom, under which evidence, at what latency and cost? The chef-versus-calculator analogy explains why open-ended tasks admit many valid outputs, but neither classifiers nor human judgments are inherently deterministic or unbiased.

You will build a representative golden set, separate reference-based/reference-free/task-specific metrics, calibrate judges, compute RAG metric definitions and design an offline/online release loop. Session 117 supplies RAG context, 116 prompting (not 119), and 92 classical metrics. Fifty examples can be a starting pilot, not a one-hour guarantee of trustworthy coverage.

The evaluation cube and testing pyramid

Three decisions are largely orthogonal: what property (exact answer, relevance, faithfulness, safety), who/what grades (program, model, human), and when (change-time, sampled production, periodic audit). Reference-based metrics compare to expected outputs; reference-free means no reference answer is supplied, not no supervision or assumptions; task-specific describes the requirement and can overlap either.

At the pyramid's base, deterministic schema, arithmetic, citation-range and tool-permission checks cheaply catch precise violations. A regex is not proof of no PII, and a citation's presence is not entailment. Next, frozen representative cases and explicit rubrics detect regressions. Sampled production detects distribution change; human audits calibrate judgments and investigate difficult cases. Online feedback, retention or refund rates describe product outcomes but can be confounded, delayed or gamed. Humans are a reference process with disagreement and adjudication, not unquestionable truth.

BLEU (2002) and ROUGE (2004) measure forms of lexical overlap for particular tasks; BERTScore (2019 preprint) uses contextual representations with references. MT-Bench's 2023 judge study and RAGAS broaden model-based evaluation, but no single metric or public leaderboard substitutes for your product's task distribution.

Four RAG metric contracts, not four interchangeable scores

The version contract here is RAGAS 0.3.7, with each variant named. Plain retrieval precision@k is relevant chunks divided by k. RAGAS Context Precision is ranking-sensitive:

Declare the zero-relevant convention rather than divide by zero. For relevance indicators [1,0,1], plain precision is 2/3 while CP is (1+2/3)/2=5/6. [0,1,1] has the same plain precision but CP (1/2+2/3)/2=7/12. This corrects the original equivalence claim. LLMContextPrecisionWithoutReference compares contexts with the response; LLMContextPrecisionWithReference uses a reference. Reference-based and non-LLM variants exist, so “only one of four metrics requires labels” is not universally true.

Context Recall in the LLM variant decomposes the reference answer into claims and asks what fraction is supported by retrieved context. If gold requires facts A, B, C and context supports only A and B, recall is 2/3 regardless of the number of chunks. Incomplete reference answers make the denominator incomplete.

Faithfulness decomposes the generated response into claims and checks support in the supplied context. Supported claims divided by all response claims measures grounding, not world truth: a faithful answer can repeat a false source. A true extra fact can be unfaithful to the provided evidence. An empty answer should not earn a silent perfect score; score abstention/usefulness separately.

Response Relevancy generates questions from the answer and averages cosine similarity to the original question's embedding. In the pinned 0.3.7 implementation, the mean is multiplied by zero when all generated judgments flag the response as noncommittal; one such flag alone does not zero it. If every generated question is empty, the implementation returns NaN instead. Report that as an invalid evaluation, not a zero or silently dropped row. It estimates on-topicness, not factual truth; cosine can be negative. Question generation, embeddings and model versions introduce variation. Changing retrieval can move all four metrics in either direction; there is no guaranteed “smaller embedding model hurts precision first” law.

The primary definitions are precision, recall, faithfulness and relevancy.

Perplexity, calibration and uncertainty

Perplexity is exp(total target-token NLL / target-token count). It measures predictive fit to a reference distribution, not whether a generated answer solves a task. Different valid wording need not receive the same penalty as an incorrect answer; the original asserted equality was false. Tokenization, masking, context and aggregation must match for meaningful comparisons. Perplexity remains useful for language modeling and checkpoint diagnostics; humans, task outcomes and hybrid metrics are alternatives to the original false “exact match or model judge only” dichotomy.

Judge calibration needs a rubric, blinded outputs, independently annotated calibration cases, and an untouched test set. Log confusion matrices, slice-specific errors and inter-human disagreement, not just agreement. With observed agreement po and chance agreement pe from marginal class rates, Cohen's kappa is (po-pe)/(1-pe) when the denominator is nonzero. Human-human kappa is context, not a mathematical upper bound or proof that a judge matching it is equally safe.

An always-positive judge can achieve 95% agreement on a 95%-positive sample and miss every negative. Four easy examples at 100% agreement do not establish calibration or CI safety. At n=100, p=.87, a simple binomial standard error is about .034; 87→85% alone cannot decide a release. Use paired case outcomes, repeated-run variance, confidence intervals and predeclared risk thresholds. A rare safety regression can block release even if aggregate confidence intervals overlap.

Hands-on: metric and judge-audit arithmetic

This CPU harness uses explicit relevance/claim labels and scripted pairwise judgments. It executes formulas and bias handling, not an LLM judge or a RAGAS model-backed run.

import math
from statistics import mean
 
def context_precision(labels):
    if not labels or any(x not in (0, 1) for x in labels):
        raise ValueError("binary relevance labels")
    positives, hits, total = sum(labels), 0, 0.0
    for k, value in enumerate(labels, 1):
        hits += value
        total += (hits/k)*value
    return total/positives if positives else 0.0
 
def supported_fraction(labels):
    if not labels or any(type(x) is not bool for x in labels):
        raise ValueError("nonempty claim support labels")
    return sum(labels)/len(labels)
 
def audit(human, predicted):
    if len(human) != len(predicted) or not human:
        raise ValueError("paired labels")
    if any(type(x) is not bool for x in human+predicted):
        raise ValueError("boolean labels required")
    n = len(human)
    po = sum(a == b for a, b in zip(human, predicted))/n
    ph, pp = mean(human), mean(predicted)
    pe = ph*pp + (1-ph)*(1-pp)
    return {"agreement": po, "kappa": (po-pe)/(1-pe) if pe < 1 else None,
            "false_positive": sum(not a and b for a, b in zip(human, predicted)),
            "false_negative": sum(a and not b for a, b in zip(human, predicted))}
 
def reconcile_orders(ab, ba):
    # Inputs name the presentation slot, not the underlying answer identity.
    allowed = {"first", "second", "tie"}
    if ab not in allowed or ba not in allowed:
        raise ValueError("invalid judge output")
    a = {"first":"A", "second":"B", "tie":"tie"}[ab]
    b = {"first":"B", "second":"A", "tie":"tie"}[ba]
    return a if a == b else "inconsistent"
 
def wilson(successes, n, z=1.96):
    if not 0 <= successes <= n or n <= 0:
        raise ValueError("counts")
    p = successes/n
    center = (p+z*z/(2*n))/(1+z*z/n)
    half = z*math.sqrt(p*(1-p)/n+z*z/(4*n*n))/(1+z*z/n)
    return center-half, center+half
 
assert math.isclose(context_precision([1,0,1]), 5/6)
assert math.isclose(context_precision([0,1,1]), 7/12)
assert context_precision([0,0]) == 0
assert supported_fraction([True, False, True]) == 2/3
bad = audit([True]*95+[False]*5, [True]*100)
assert bad["agreement"] == .95 and bad["kappa"] == 0 and bad["false_positive"] == 5
assert reconcile_orders("first", "first") == "inconsistent"
assert reconcile_orders("first", "second") == "A"
assert reconcile_orders("tie", "tie") == "tie"
low, high = wilson(4, 4)
assert low < .52 and high <= 1.0000001
try:
    supported_fraction([])
except ValueError:
    pass
else:
    raise AssertionError("empty claims must not pass")
try:
    audit([True, False], ["pass", "fail"])
except ValueError:
    pass
else:
    raise AssertionError("string grades must not count as boolean labels")
# Version-specific relevancy arithmetic over supplied cosine/flag fixtures.
# No questions are generated and no embeddings/model are loaded here.
def relevance_fixture(cosines, noncommittal):
    if not cosines or len(cosines) != len(noncommittal):
        raise ValueError("paired nonempty samples")
    if any(type(x) is not bool for x in noncommittal):
        raise ValueError("boolean flags required")
    if any(not math.isfinite(x) or not -1 <= x <= 1 for x in cosines):
        raise ValueError("finite cosine values required")
    return mean(cosines) * int(not all(noncommittal))
assert relevance_fixture([1., .5], [False, False]) == .75
assert relevance_fixture([1., .5], [True, False]) == .75
assert relevance_fixture([1., .5], [True, True]) == 0
assert relevance_fixture([-.5], [False]) == -.5
print("metric/calibration/position controls pass; no model judge evaluated")

Complete model-judge and RAGAS integration procedure

The original position_bias_test never made a judgment or counted a winner. Replace that stub with a real procedure when a model is authorized: send every pair in both orders, blind identity, require an enum verdict, map slot back to identity, and retain inconsistent results separately from genuine ties. Random order alone in ten calls confounds sampling and order. Swap-and-agree detects some bias but does not eliminate all bias.

For claim grading, require supported/unsupported claims with exact source spans, an explicit abstention/invalid state, and a rubric distinguishing grounding from relevance. JSON mode controls syntax under its API contract, not field validity. Validate types/ranges, refusal/truncation, contradictory faithful=true with unsupported claims, and missing output. Pin model snapshot/prompt/schema/version where possible and record actual usage/cost instead of old per-row prices. Low temperature is useful where supported but not a reproducibility guarantee; repeat unchanged rows to estimate variation.

For RAGAS 0.3.7, construct SingleTurnSample with user_input, response, retrieved_contexts and reference only where the selected metric requires it. Supply an authorized evaluator LLM and embeddings, instantiate the named metric variant, and await single_turn_ascore. Store per-row artifacts and errors rather than dropping failed calls. This source-supported recipe is not executed here and no evaluator/model/data is downloaded.

The original four-row calibration fixture can still teach grounding: given only “The tower was built in 1889 and is 330 meters tall,” the matching answer is supported, while “1887, Napoleon, 400 meters” is not. Treat this as supplied fictionalized evaluation context rather than an independently checked historical source. Given “Python 3.12 introduced PEP 695,” attributing that feature is supported but attributing match statements or the walrus operator is not supported by that context. Adding an author name absent from the context creates a partly unsupported answer even if the name is true elsewhere. Add such borderline cases and separate claim-level support from whole-answer pass/fail; four easy rows are not calibration.

Golden-set design, release gates and failure cases

Build a pilot from permission-cleared real tasks: common success, exact-ID retrieval, multi-hop, ambiguous questions, unanswerable cases, stale sources, injection attempts and sensitive actions. Label expected sources, acceptable answers, forbidden behavior and relevant slices. De-identify inputs, adjudicate disagreements, version the rubric and freeze a holdout. Synthetic expansion can help after validation but does not create independent ground truth. Fifty carefully selected examples can reveal more than thousands of unreviewed generated ones, yet neither number guarantees coverage.

A release loop freezes inputs and corpus/model/config versions, runs old/new systems, captures answers/contexts/latencies/errors, grades with the same metric versions, compares paired rows and slices, and sends disagreements for review. Avoid a universal 3-point gate. Keep a sealed final set separate from prompt iteration; repeated optimization makes a “test” set development feedback. Log leakage checks without inventing an acceptable contamination percentage. Update sets intentionally when the product changes; a fixed quarterly 10% rotation is not evidence-based law.

Zheng et al. study position, verbosity, self-enhancement and reasoning limitations. Probe answer order, matched-quality length/style perturbations and model identity separately; a different-family judge or ensemble can still share biases. Prestige/formality can be tested as a style confound, not asserted to have a fixed effect. Golden-set decay is a dataset problem, not itself a judge bias.

The original OneNote, Claude 62%-position and Arena-rotation “war stories” had no primary incident evidence and are not retained as facts. Their scenarios remain: an evaluator rewards verbosity while users dislike it; order changes a pairwise decision; benchmark tuning or leaked data inflates scores. Corrective actions are calibrated rubrics, both-order tests, private holdouts and monitored product outcomes, not invented universal prices, gains or rotation schedules.

Recall and stretch: complete answers

1. Variation sources: generation sampling/infrastructure, stochastic grading, changing contexts/model versions and annotator disagreement. Open-ended output space is not itself randomness, and classifiers can also be stochastic.
2. Definitions: faithfulness is claim support; RAGAS context precision is rank-sensitive, not plain precision; response relevance measures on-topicness, not truth.
3. Position bias: compare both orders mapped back to answer identity, report inconsistencies and genuine ties separately, and calibrate residual bias.
4. Small human set? It can anchor product-specific requirements if representative and adjudicated. Size alone does not confer truth.
5. Cheaper judge? Use deterministic code for JSON/schema/arithmetic; for semantic grading use the least costly judge whose measured slice errors satisfy risk limits.
6. 87→85%? Inspect paired differences, uncertainty, severity and predeclared gates; neither automatically pass nor automatically fail at a universal 3pp rule.
7. Temperature zero? Not universally available or non-negotiable; record settings and measured repeated-run variation.
8. Metric hacking: optimizing rubric quirks instead of outcomes. Defend with held-out cases, multiple grounded measures and audited online feedback, which itself can be gamed.
9. Which RAGAS needs references? LLM context recall uses a reference; context precision has with/without-reference variants. Name the actual class rather than claiming exactly one always does.
10. Smaller embeddings stretch: measure candidate recall/ranking and all downstream metrics. Direction and order of change depend on data, context selection and generator behavior.

Explain what a reported three-point gain used as its denominator, paired set, judge version and uncertainty. A golden set is an executable part of the specification, not the entire product; evaluation links retrieval, fine-tuning (123), serving (124) and the chat-product capstone (130).

Optional resources and onward navigation

The original optional pre-read links are retained as navigation, not fact-check evidence or promises about video length, quality or API stability. Use the versioned primary sources in the teaching for consequential claims. Original pacing was a suggested 90-minute session, not a completion guarantee.

• How to Evaluate LLMs
• LLM Evaluation with RAGAS — end to end
• LLM-as-a-Judge with OpenAI Evals
• Building LLM Applications for Production
• Anthropic — Evaluating AI Systems
• RAGAS · Metric definitions

Next in the original sequence: S123 · Fine-Tuning — LoRA, QLoRA, PEFT. Publication and release navigation are adjudicated by the parent.

Hub: The 6-Month Learning Plan. The original 130-session scaffold is retained as navigation, not evidence that every destination has been reviewed.

Review and execution boundary

Reviewed against complete original and pre-edit current teaching with claim-specific primary evidence and independently replayed coverage. The executable CPU program is a bounded teaching test, not a pretrained model, serving, RAG or training benchmark. External model/service recipes remain source-supported and unexecuted. No weights, datasets, GPU, paid API or new service were used. Source publication is not end-to-end application acceptance.