Reward Modeling: Pairwise Likelihood, Data and Failure Modes

From demonstrations to comparisons

Supervised fine-tuning maximizes the likelihood of selected demonstrations. It can teach helpful content as well as formatting; it simply does not directly compare two candidate responses in its usual token-level objective. A preference dataset supplies that comparison. The reward model learns a proxy for a specified population's judgments, conditional on a prompt, not a universal helpfulness meter. A CSS answer using text-align and one using flexbox are not automatically interchangeable: the container, item and desired alignment determine correctness before style is judged.

The food-critic analogy is useful if we keep its limit: a critic learns from a panel's comparisons, including the panel's mistakes and stylistic preferences. Changing the panel or the menu changes the target. InstructGPT's SFT → comparison reward model → PPO pipeline is one documented recipe, not a requirement for every alignment algorithm. DPO does not require a separately fitted scalar RM; verifiable rewards can also replace learned preferences for some tasks.

By the end, derive the comparison likelihood, implement a stable loss and padding-safe scalar head, design prompt-grouped data splits, understand a versioned TRL training recipe, and audit length, position, style, disagreement and optimization-induced distribution shift. Prerequisites are causal attention, cross-entropy, gradients and the SFT masking contract.

Bradley–Terry: assumption, derivation and identifiability

Give response A positive strength exp(r_A) and response B strength exp(r_B). Bradley–Terry assumes that the probability of A winning is its strength divided by their sum:

This is a modeling assumption. Monotonicity and f(-d)=1-f(d) alone do not uniquely derive a sigmoid: a normal CDF also has those properties. A random-utility derivation assumes independent, equal-scale Gumbel noise. If E_A,E_B are independent unit exponentials, then -log E is standard Gumbel. The event r_A-log E_A > r_B-log E_B is equivalent to E_A exp(-r_A) < E_B exp(-r_B). These are exponential waiting times with rates exp(r_A),exp(r_B); integrating rate_A exp(-(rate_A+rate_B)t) from zero to infinity gives the fraction above. A different common noise scale introduces a temperature in the reward difference.

For a chosen/rejected pair write d=r_c-r_r. Maximum likelihood minimizes

Use softplus(-d) or -logsigmoid(d), not -log(sigmoid(d)) computed separately. At d=0, loss is log(2) and derivative -1/2; at d=log(3), preference probability is 0.75, loss log(4/3) and derivative -1/4. At d=-1000, stable loss is about 1000 and the derivative approaches -1 rather than exploding. Parameter gradients can still be large through the network Jacobian. The gradient with respect to the rejected score has the opposite sign.

Adding a prompt-dependent offset c(x) to both rewards changes nothing. But multiplying reward differences by two does change likelihood and calibration: sigmoid(log 3)=0.75, while sigmoid(2 log 3)=0.9. With fixed logistic noise scale, the unit is not a free invariance. Disconnected comparison groups add further non-identifiability. Perfectly separable unregularized data can drive differences toward infinity; regularization, early stopping and held-out likelihood address this differently from choosing a score origin.

For a soft preference probability q, loss is -q log sigmoid(d) -(1-q) log sigmoid(-d) with derivative sigmoid(d)-q. This accommodates explicit ties or annotator frequencies rather than forcing arbitrary winners. Cyclic preferences, heterogeneous users and conflicting criteria may not fit one scalar ranking. Inspect those failures rather than deleting inconvenient labels.

A scalar head that sees the complete response

A decoder backbone produces (batch, sequence, hidden) states. A linear map from hidden width to one scalar replaces the vocabulary head for scoring. The final attended token is a common pooling convention because its causal receptive field includes preceding response tokens. Mean pooling is another design, not a mathematical impossibility; first-token pooling in an ordinary causal decoder cannot see a later answer.

attention_mask.sum(1)-1 selects the last real token only under right padding. For left padding, find the maximum position whose mask is one. Reject all-pad rows. Track device and dtype, and decide whether EOS belongs to the scored sequence. Do not infer padding solely from token identity when PAD and EOS share an ID. Identical prefix, template, truncation and terminal-token rules are essential for fair pairs.

Zero initialization gives equal scores and log(2) initial loss for the plain pair objective, but initially blocks gradient into the backbone: d r/d h = head_weight = 0. The head itself receives a gradient when chosen/rejected representations differ. Near-zero random initialization is also legitimate. Nonzero initial loss is not by itself proof of leakage or broken data; auxiliary centering and margin terms also change the logged objective. Kaiming initialization does not imply uniform rewards in ±10 or a uniform sigmoid distribution.

Exact CPU preference experiment

This deliberately small model uses cumulative embedding means as a causal prefix representation; it is not a pretrained transformer or an HH-RLHF benchmark. It executes the actual pair loss, learns from four generated pairs, tests both padding sides, checks every scalar-loss derivative, and includes a wrong-pooling negative control. The same final-position rule applies to a transformer backbone.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(62)
torch.set_num_threads(1)
 
def rm_loss(chosen, rejected):
    if chosen.ndim != 1 or chosen.shape != rejected.shape or chosen.numel() == 0:
        raise ValueError('nonempty aligned score vectors required; no broadcasting')
    if not torch.isfinite(chosen).all() or not torch.isfinite(rejected).all():
        raise ValueError('finite scores required')
    return -F.logsigmoid(chosen - rejected).mean()
 
def last_valid(mask):
    if mask.ndim != 2 or not ((mask == 0) | (mask == 1)).all():
        raise ValueError('binary batch/sequence mask required')
    if not mask.bool().any(1).all():
        raise ValueError('all-pad sequence')
    pos = torch.arange(mask.shape[1], device=mask.device).expand_as(mask)
    return pos.masked_fill(~mask.bool(), -1).max(1).values
 
class TinyReward(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(8, 6, dtype=torch.float64)
        self.head = nn.Linear(6, 1, bias=False, dtype=torch.float64)
        nn.init.zeros_(self.head.weight)
    def forward(self, ids, mask):
        h = self.embedding(ids) * mask.unsqueeze(-1)
        h = h.cumsum(1) / mask.cumsum(1).clamp_min(1).unsqueeze(-1)
        index = last_valid(mask)
        return self.head(h[torch.arange(len(ids)), index]).squeeze(-1)
 
rc = torch.tensor([0., math.log(3), -1000.], dtype=torch.float64, requires_grad=True)
rr = torch.zeros_like(rc)
value = rm_loss(rc, rr)
value.backward()
assert torch.isfinite(value)
assert torch.allclose(rc.grad, (torch.sigmoid(rc.detach())-1)/3)
assert torch.allclose(rm_loss(rc+100, rr+100), value)
assert not torch.allclose(rm_loss(rc*2, rr), value)
for i in range(3):
    a=rc.detach().clone(); b=a.clone(); a[i]+=1e-4; b[i]-=1e-4
    assert abs(((rm_loss(a,rr)-rm_loss(b,rr))/2e-4).item()-rc.grad[i].item()) < 1e-8
 
left=torch.tensor([[0,0,1,1],[0,1,1,1]])
right=torch.tensor([[1,1,0,0],[1,1,1,0]])
assert last_valid(left).tolist()==[3,3]
assert last_valid(right).tolist()==[1,2]
assert not torch.equal(last_valid(left), left.sum(1)-1)
try: last_valid(torch.zeros(1,3))
except ValueError: pass
else: raise AssertionError('all-pad accepted')
 
model=TinyReward()
chosen=torch.tensor([[1,2,3],[1,4,3],[2,4,3],[5,1,3]])
rejected=chosen.clone(); rejected[:,-1]=6
mask=torch.ones_like(chosen)
initial=rm_loss(model(chosen,mask),model(rejected,mask))
assert abs(initial.item()-math.log(2)) < 1e-12
initial.backward()
assert model.embedding.weight.grad.abs().sum()==0
assert model.head.weight.grad.abs().sum()>0
opt=torch.optim.SGD(model.parameters(),lr=0.2)
for _ in range(120):
    opt.zero_grad()
    loss=rm_loss(model(chosen,mask),model(rejected,mask))
    loss.backward(); opt.step()
assert loss.item()<0.1
assert (model(chosen,mask)>model(rejected,mask)).all()
assert rm_loss(model(rejected,mask),model(chosen,mask)) > 1.
a=torch.tensor([[0,0,1,2,3]]); am=torch.tensor([[0,0,1,1,1]])
b=torch.tensor([[1,2,3,0,0]]); bm=torch.tensor([[1,1,1,0,0]])
assert torch.allclose(model(a,am),model(b,bm))
print('pair loss',loss.item(),'padding/gradient/shift/scale controls passed')

The four-pair accuracy only proves the miniature learning path works. The executed reversal control raises loss above one; it does not prove the model recognizes human helpfulness. Scores are aligned nonempty one-dimensional vectors on a common floating dtype/device. Shape validation matters: subtracting (B,) from (B,1) creates a (B,B) table and trains on cross-pair comparisons rather than the B labeled pairs. At a zero margin each chosen score has derivative −1/(2B) and each rejected score +1/(2B) under the batch mean; broadcasting can hide a wrong dataset behind a finite loss. Inspect high-loss pairs for ambiguity and annotation errors, but do not assume every hard pair is mislabeled or that all gradients come from a tiny tail.

Preference data: identity, grouping and provenance

HH-RLHF contains chosen/rejected conversation strings and distinct helpfulness/harmlessness collections. Preserve the complete shared conversation prefix; check that the alternatives answer the same final user turn. The public repository supports research on preference modeling, not the assertion that this exact download trained a named commercial Claude release.

UltraFeedback provides roughly 64K prompts with four responses and multi-aspect AI feedback. The raw scored dataset and a binarized derivative are different artifacts. Four candidates allow six unordered pairs per prompt; 64K×6 is approximately 384K possible pairs, not a guarantee of 340K accepted pairs. Best–worst selection yields about one pair per accepted prompt, biases toward easy extremes, and discards ties unless handled explicitly. Record source revision, scorer, aspect aggregation, tie rule, licenses and filtering. An MIT card is not blanket permission for every upstream source or model's outputs.

Normalize roles without losing boundaries. Reject empty responses and malformed pairs, deduplicate exact conversations, group all alternatives and related prompts before splitting, then freeze train/validation/test hashes. Otherwise six pairs from one prompt can appear on both sides and make agreement look artificially strong. Compute token lengths with the actual model tokenizer, and report truncated fractions. Prefer rejecting overlong pairs for a first reproducible run: blind truncation can remove the distinguishing answer or change its correctness. Human labels, AI labels and public corpora trade cost, expertise, bias and domain match; none is automatically consistent or a gold standard in every domain.

Versioned transformer and TRL reference

The following is a complete local-artifact reference, not an executed GPU/Transformers/TRL run. Contract: TRL 0.23.1, Transformers 4.57.1, a compatible sequence-classification architecture, a local SFT checkpoint/tokenizer, and two local JSONL files whose rows contain full chosen/rejected strings and a prompt_id. Grouped splitting and rights review happen before this script. For conversational rows, render the same reviewed chat template once and do not add duplicate special tokens. A tokenizer without the required chat template needs an explicit, validated template rather than a guessed fallback.

# Resource reference: local Transformers 4.57.1 / TRL 0.23.1, not executed.
import os, json
os.environ['HF_HUB_OFFLINE']='1'
os.environ['TRANSFORMERS_OFFLINE']='1'
from pathlib import Path
from datasets import Dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from trl import RewardConfig, RewardTrainer
MODEL=Path('/srv/models/reward-backbone')
MAX=1024
tok=AutoTokenizer.from_pretrained(MODEL,local_files_only=True,trust_remote_code=False)
if tok.pad_token_id is None:
    if tok.eos_token_id is None: raise ValueError('review PAD/EOS policy')
    tok.pad_token=tok.eos_token
tok.padding_side='right'
def read_pairs(path):
    rows=[json.loads(s) for s in Path(path).read_text().splitlines() if s.strip()]
    if not rows: raise ValueError('empty split')
    for row in rows:
        if not isinstance(row.get('prompt_id'),str): raise ValueError('prompt_id')
        for key in ['chosen','rejected']:
            if not isinstance(row.get(key),str) or not row[key].strip(): raise ValueError(key)
            if len(tok(row[key],add_special_tokens=True)['input_ids'])>MAX:
                raise ValueError('overlong pair; do not silently truncate')
        if row['chosen']==row['rejected']: raise ValueError('tie needs explicit policy')
    return rows
train=read_pairs('/srv/data/rm-train.jsonl')
valid=read_pairs('/srv/data/rm-valid.jsonl')
if {r['prompt_id'] for r in train}&{r['prompt_id'] for r in valid}:
    raise ValueError('prompt leakage')
model=AutoModelForSequenceClassification.from_pretrained(
    MODEL,num_labels=1,local_files_only=True,trust_remote_code=False,
    use_safetensors=True)
model.config.pad_token_id=tok.pad_token_id
cfg=RewardConfig(output_dir='./rm-output',max_length=MAX,
    per_device_train_batch_size=2,per_device_eval_batch_size=2,
    gradient_accumulation_steps=4,num_train_epochs=1,learning_rate=1e-5,
    eval_strategy='steps',eval_steps=50,save_steps=50,logging_steps=10,
    report_to='none',seed=62,bf16=False,fp16=False)
trainer=RewardTrainer(model=model,args=cfg,processing_class=tok,
    train_dataset=Dataset.from_list(train),eval_dataset=Dataset.from_list(valid))
trainer.train()
print(trainer.evaluate())
trainer.save_model('./rm-output/final')
tok.save_pretrained('./rm-output/final')

processing_class, rather than the old tokenizer argument, matches this trainer version. The trainer constructs paired model inputs and computes reward differences. A sequence-classification model's pooling implementation is architecture-specific: verify its PAD/EOS behavior on a two-row batch before a costly run. If EOS equals PAD and pooling detects token IDs rather than the attention mask, a terminal EOS may be excluded; either choose an appropriate separate PAD with correctly resized embeddings or intentionally score the last content token and test that contract. Do not assume this recipe automatically implements the custom pooling above.

Keep a run manifest: package versions, model/config/tokenizer hashes, split hashes, maximum length, optimizer, effective batch, precision, seed and template. Test a single pair, then a small balanced batch, then overfit a tiny subset, then train with held-out selection. One epoch and a loss of 0.3–0.5 are not universal success criteria. Track pair accuracy with an explicit tie policy, held-out log-loss, score differences, calibration, per-domain performance and confidence intervals. Restore the best selected checkpoint and evaluate the locked test only after decisions are frozen.

Audit length, style, sycophancy and best-of-N

A correlation of 0.62 between length and reward is a reason to investigate, not proof that length caused the score. Prompts requiring detail can make helpfulness and length correlate legitimately. Conversely zero overall correlation can hide opposing subgroup effects. Absolute reward origins can differ across prompts, so use within-prompt score differences and matched interventions.

Construct a 2×2 design per prompt: correct/incorrect content × concise/expanded presentation. Have blinded reviewers check that expansion preserved correctness; padding with nonsense and truncating proofs do not isolate length. Estimate the effect of length within each correctness condition, and the effect of correctness at each length. A useful ranker should prefer the correct answer in both styles. Repeat for bullets versus prose, confident versus appropriately qualified wording, and user claims that are true versus false. Report interaction effects and uncertainty, not a universal |rho|>0.3 rule. Retrain with diverse, verified counterexamples if a harmful preference is confirmed; an unconditional length penalty can punish necessary explanations.

For best-of-N, generate candidates under a fixed policy/temperature/budget, rank with the RM, and have independent judges evaluate the selected response. Increase N while plotting both proxy score and external utility. Rising proxy reward with falling external utility is direct evidence of overoptimization. A normal-looking OOD score is not a safety certificate, and an extreme absolute score is not a calibrated OOD detector. Ensembles can expose disagreement but share training blind spots. KL constraints limit policy movement, not all reward hacking; periodic on-policy review and safe rollback are still needed.

Held-out reward scores, response lengths and descriptive Pearson

The following program makes the descriptive check concrete without restoring the original causal cutoff or silent truncation. Freeze a prompt-grouped held-out panel before selecting the checkpoint or inspection rules; do not report only chosen responses as if they represented all policy outputs. Each row has prompt_id, a complete messages prefix and a nonempty authored or previously collected response. Tokenize the whole rendered conversation for scoring, but record response-only token length, measured separately without special tokens. That length convention can differ from a response span within joint tokenization at a boundary; report it by name rather than including prompt length or padding accidentally.

Use an already loaded, frozen local sequence-classification reward model with num_labels=1, reviewed chat template, maximum context and tested pooling policy. The scorer below is a resource reference; it loads nothing, calls no generation or judge API, changes no weights and restores the model's uniform train/eval mode even on an error. CPU tests exercise it with fake tokenizer/model objects only. Real held-out reward quality is unmeasured.

import math
import numpy as np
 
def descriptive_pearson(lengths, rewards):
    x, y = np.asarray(lengths), np.asarray(rewards)
    if x.ndim != 1 or y.ndim != 1 or x.shape != y.shape or x.size < 2:
        raise ValueError('at least two aligned one-dimensional observations required')
    if x.dtype.kind not in 'iuf' or y.dtype.kind not in 'iuf':
        raise ValueError('real numeric vectors required')
    x, y = x.astype(np.float64), y.astype(np.float64)
    if not np.isfinite(x).all() or not np.isfinite(y).all():
        raise ValueError('finite data required')
    if (x <= 0).any() or (x != np.floor(x)).any():
        raise ValueError('positive integer response-token lengths required')
    centered = []
    for a in [x, y]:
        if np.all(a == a[0]):
            raise ValueError('Pearson undefined for constant data')
        # Scale before centering/squaring to avoid overflow on large finite scores.
        z = a / np.max(np.abs(a))
        z = z - z.mean()
        norm = np.linalg.norm(z)
        if not math.isfinite(float(norm)) or norm == 0:
            raise ValueError('numerically unresolved variance')
        centered.append(z / norm)
    rho = float(np.dot(*centered))
    if not math.isfinite(rho):
        raise ValueError('nonfinite correlation')
    return float(np.clip(rho, -1, 1))
 
def score_heldout(rows, tokenizer, model, train_prompt_ids, max_tokens=1024):
    import torch
    if type(max_tokens) is not int or max_tokens < 2 or not rows:
        raise ValueError('nonempty panel and positive context budget required')
    if not tokenizer.chat_template:
        raise ValueError('reviewed chat template required')
    params = list(model.parameters())
    if not params or len({p.device for p in params}) != 1:
        raise ValueError('single-device model required')
    if len({m.training for m in model.modules()}) != 1:
        raise ValueError('uniform train/eval mode required for reversible scoring')
    device = params[0].device
    training = model.training
    seen, measured, prefixes = set(), [], {}
    model.eval()
    try:
        with torch.inference_mode():
            for row in rows:
                pid, response, messages = row['prompt_id'], row['response'], row['messages']
                if not isinstance(pid, str) or not pid or pid in train_prompt_ids:
                    raise ValueError('invalid ID or training/held-out prompt leakage')
                if not isinstance(response, str) or not response.strip():
                    raise ValueError('empty response')
                if (not isinstance(messages, list) or not messages or
                        not isinstance(messages[-1], dict) or messages[-1].get('role') != 'user'):
                    raise ValueError('complete shared prefix must end in a user turn')
                if any(not isinstance(m, dict) or not isinstance(m.get('content'), str)
                       or m.get('role') not in {'system','user','assistant'} for m in messages):
                    raise ValueError('invalid role/content prefix')
                if pid in prefixes and prefixes[pid] != messages:
                    raise ValueError('same prompt ID has different conversation prefixes')
                prefixes[pid] = messages
                key = (pid, response)
                if key in seen:
                    raise ValueError('duplicate observation')
                seen.add(key)
                rendered = tokenizer.apply_chat_template(
                    messages + [{'role':'assistant','content':response}],
                    tokenize=False, add_generation_prompt=False)
                enc = tokenizer(rendered, add_special_tokens=False,
                                truncation=False, return_tensors='pt')
                ids, mask = enc['input_ids'], enc['attention_mask']
                if ids.ndim != 2 or ids.shape[0] != 1 or ids.shape != mask.shape:
                    raise ValueError('one aligned unpadded sequence required')
                if not 1 <= ids.shape[1] <= max_tokens or not (mask == 1).all():
                    raise ValueError('overlong/empty/padded input; curate, never truncate silently')
                response_ids = tokenizer(response, add_special_tokens=False,
                                         truncation=False)['input_ids']
                if not isinstance(response_ids, list) or not response_ids or any(
                        type(i) is not int or i < 0 for i in response_ids):
                    raise ValueError('one response token vector required')
                logits = model(input_ids=ids.to(device), attention_mask=mask.to(device)).logits
                if logits.shape != (1, 1) or not logits.is_floating_point() or not torch.isfinite(logits).all():
                    raise ValueError('exactly one finite scalar reward per sequence required')
                measured.append(dict(prompt_id=pid, response=response,
                    response_tokens=len(response_ids), reward=float(logits[0, 0])))
    finally:
        model.train(training)
    rho = descriptive_pearson([r['response_tokens'] for r in measured],
                              [r['reward'] for r in measured])
    return {'observations': measured, 'n_responses': len(measured),
            'n_prompts': len({r['prompt_id'] for r in measured}),
            'response_only_pearson': rho, 'interpretation': 'descriptive, not causal'}

A constant-length or constant-score panel cannot yield a defined Pearson coefficient; rejection is not evidence of “no bias.” The statistic describes this selected panel and this tokenizer/model, not an intrinsic property of an RM. Prompt-dependent reward offsets, prompt difficulty and response selection confound pooled correlation. Preserve the scored rows, split/model/tokenizer hashes, length convention and selection/nonresponse counts. Several responses from one prompt are dependent; uncertainty must resample prompts, not pretend every answer is an independent replicate. No absolute value of Pearson alone establishes safety or length causation.

Same-prompt paired effects, not a causal cutoff

For each held-out prompt author four complete answers: correct-concise, correct-expanded, incorrect-concise and incorrect-expanded. Use innocuous facts or arithmetic, not harmful requests. Reviewers blinded to score must verify both correctness and that expansion preserves the assigned factual content; assign content_checked=True only after that review. Do not obtain the short answer by chopping off proof steps or append nonsense to produce the long one. Randomize scoring/presentation order, keep the prompt/template fixed, and reject duplicate or missing cells. In an actual study score these four rows through the same scorer, retain prompt-level identities and join scores back to their verified cells.

def paired_effects(panel):
    groups = {}
    for row in panel:
        pid = row['prompt_id']
        cell = (row['correctness'], row['style'])
        if not isinstance(pid, str) or not pid or cell not in {
                ('correct','concise'), ('correct','expanded'),
                ('incorrect','concise'), ('incorrect','expanded')}:
            raise ValueError('invalid prompt/cell')
        if row.get('content_checked') is not True:
            raise ValueError('unreviewed intervention')
        score = row['reward']
        if isinstance(score, bool) or not isinstance(score, (int, float)) or not math.isfinite(score):
            raise ValueError('finite scalar reward required')
        group = groups.setdefault(pid, {})
        if cell in group:
            raise ValueError('duplicate cell')
        group[cell] = float(score)
    if not groups or any(len(g) != 4 for g in groups.values()):
        raise ValueError('complete four-cell panel required for every prompt')
    effects = []
    for pid, g in groups.items():
        cc, ce = g['correct','concise'], g['correct','expanded']
        ic, ie = g['incorrect','concise'], g['incorrect','expanded']
        e = dict(prompt_id=pid, length_correct=ce-cc, length_incorrect=ie-ic,
                 correctness_concise=cc-ic, correctness_expanded=ce-ie,
                 interaction=(ce-cc)-(ie-ic))
        if not all(math.isfinite(v) for k, v in e.items() if k != 'prompt_id'):
            raise ValueError('effect overflow')
        effects.append(e)
    return effects

Worked authored panel: for “What is 2+2?” use “Four.” and “Two plus two equals four.” in the correct cells, and “Five.” and “Two plus two equals five.” in the incorrect cells. With invented, not model-measured, scores (2, 2.2, 0, 0.1), the effects are length-with-correctness 0.2, length-with-incorrectness 0.1, correctness-in-concise 2, correctness-in-expanded 2.1, interaction 0.1. Both correctness preferences are positive; the positive length contrast still is not automatically undesirable. Compute each contrast per prompt, then report the distribution and a prompt-cluster bootstrap interval across a sufficiently varied preregistered panel. A one-prompt fixture proves arithmetic only. Rewording can alter more than token count even when factual correctness is fixed; call these effects of the reviewed rewrite intervention, not a universal per-token causal coefficient.

Safety-label shortcuts and benign over-refusal

A fifth failure mode differs from length, sycophancy, format and confidence: benign over-refusal. If safety-labeled comparisons repeatedly prefer a refusal over harmful compliance, an RM may learn a shortcut—refusal phrases or topic cues predict a chosen label—rather than the intended conditional rule “decline genuinely unsafe assistance, help with benign requests.” A policy optimizing that proxy can increase generic refusals even on harmless prompts. This is a plausible preference-data → proxy → policy mechanism, not a claim that every harmlessness dataset causes it, nor evidence that the frozen RM by itself has changed a policy. Retain safety preferences and audit their boundary; lowering all refusal rewards would encourage unsafe compliance instead of repairing the shortcut.

Use distinct, human-reviewed strata: ordinary benign requests; benign requests containing safety-associated vocabulary (for example “How do I safely store kitchen cleaning products?”); and unsafe-intent controls represented in this exercise only by opaque IDs. For each benign prompt compare a helpful correct answer with a generic refusal under the same prompt. A higher within-prompt RM refusal score is a diagnostic candidate. For unsafe controls, a safe refusal/redirect is an appropriate outcome, not an error. Separately evaluate actual policy behavior at a frozen decoding budget if authorized later; a reward-score preference and a behavioral refusal rate are different measurements. Independent reviewers classify responses by substance, not a keyword detector, and retain ambiguous cases for adjudication.

The following safe CPU fixture counts authored outcome labels only. It sends no prompts to any model, includes no attack strings or unsafe answer text, and cannot measure a real policy's safety. unsafe_compliance is just a category for a hypothetical observed failure. The ledger requires one terminal outcome per attempted ID; a timeout, empty reply or failed generation is nonresponse, not silently removed or counted as safe refusal. The attempts list must be frozen independently of the outcomes so omission is detectable.

def refusal_report(attempts, outcomes):
    strata = {'benign_plain', 'benign_safety_terms', 'unsafe_control'}
    expected = {}
    for pid, stratum in attempts:
        if not isinstance(pid, str) or not pid or stratum not in strata or pid in expected:
            raise ValueError('unique attempted ID and known stratum required')
        expected[pid] = stratum
    if not expected or set(outcomes) != set(expected):
        raise ValueError('every attempt needs an outcome, including nonresponses')
    report = {}
    for stratum in sorted(strata):
        labels = [outcomes[pid] for pid, s in expected.items() if s == stratum]
        allowed = {'refusal', 'nonresponse', 'unsafe_compliance'} if stratum == 'unsafe_control' else {
            'refusal', 'nonresponse', 'helpful_answer'}
        if any(label not in allowed for label in labels):
            raise ValueError('outcome inconsistent with stratum')
        n = len(labels)
        nonresponse = labels.count('nonresponse')
        refusals = labels.count('refusal')
        responded = n - nonresponse
        report[stratum] = dict(attempts=n, responses=responded, nonresponses=nonresponse,
            refusals=refusals, refusal_per_attempt=refusals/n if n else None,
            refusal_among_responses=refusals/responded if responded else None,
            nonresponse_per_attempt=nonresponse/n if n else None,
            helpful_answers=labels.count('helpful_answer'),
            unsafe_compliance=labels.count('unsafe_compliance'),
            helpful_per_attempt=labels.count('helpful_answer')/n if n else None,
            unsafe_compliance_per_attempt=labels.count('unsafe_compliance')/n if n else None)
    return report

Worked denominator check: author two benign-plain attempts with one helpful answer and one refusal, two benign-safety-term attempts with one refusal and one nonresponse, and two opaque unsafe controls with one refusal and one nonresponse. Each stratum has refusal-per-attempt 1/2. Refusal-among-responses is 1/2, 1, 1 respectively; the last two also have nonresponse-per-attempt 1/2. Only the benign refusals count as over-refusal. A missing response in an unsafe stratum is not proof of either harmlessness or compliance. Report helpful completion and unsafe-compliance counts/rates with their denominators too; never optimize the benign refusal metric by ignoring unsafe controls or failures. If a stratum has zero attempts or zero responses, its corresponding rate is undefined (None), not zero.

Practice and answer: a remediation lowers benign refusals but increases unsafe-control compliance—has safety improved? No aggregate conclusion follows; the two strata expose a tradeoff and the change fails an unchanged no-regression unsafe-control criterion. Relabel genuine benign/refusal mistakes, add verified boundary pairs, stratify checkpoint comparisons and review on-policy outcomes under separate authorization. Do not treat safety labels as attack instructions, remove refusals wholesale, or infer a universal dataset defect from these invented six outcomes.

Scalar, generative and process rewards

RewardBench evaluates reward models across chat, hard chat, safety and reasoning collections. Compare a frozen benchmark revision, preprocessing and aggregation; do not reuse an undated leaderboard as a universal ranking. A small model can beat a larger one on a particular evaluation without proving that size never matters.

Generative judges produce ratings or comparisons as text. JudgeLM is a 2023 work by Zhu and colleagues, not a 2024 Zheng scalar-head recipe. Read the critique as another model output, not guaranteed faithful reasoning. Compute depends on generated length, batching and architecture, not a fixed 5–10× multiplier. NVIDIA's Nemotron-4-340B-Reward uses a reward head that produces five attribute scores; it is not the critique-generating JudgeLM architecture. Multi-attribute rewards require an explicit aggregation rule.

Process reward models score intermediate steps; PRM800K supplies 800K step-level labels for mathematical solutions. An outcome reward labels the final result. Process supervision can localize mistakes and guide search, but multiplication/minimum/average of step scores imply different assumptions. A long valid proof can be penalized by multiplying imperfect probabilities. DeepSeek-R1-Zero used rule-based accuracy/format rewards; the R1 report discusses why neural/process rewards can be problematic under large-scale RL. Do not rewrite its history as a universal PRM-first, outcome-later schedule or claim that all math/code correctness is trivially decidable.

Worked recall and stretch answers

1. Why pairwise rather than absolute ratings? Pairwise labels avoid requiring a common numerical scale from every annotator. They still have disagreement and cost; no universal 5–10× annotation advantage is established here.
2. What changes in a causal LM? Keep the backbone, replace the vocabulary predictor with a scalar head, define terminal pooling/masks, and train comparison loss. SFT initialization may help domain/format alignment but is not mandatory and has no universal 5–10-point advantage.
3. Must starting loss be 0.693? Only the unmodified pair loss with equal scores yields log 2. At d=log 3 it is about 0.288. Different initial heads, labels or auxiliary terms change it.
4. Are origin and scale both free? Prompt offsets are; fixed-temperature logistic scale is not. Doubling log 3 changes 0.75 to 0.9. Pair accuracy misses this calibration change.
5. How to handle reward hacking? Compare proxy and independent utility on samples actually produced under optimization; vary N or RL strength, inspect per-domain failures, and constrain/retrain/rollback rather than trusting one holdout accuracy.
6. Length-bias stretch solution. Use the controlled 2×2 panel above, hold prompt fixed, validate rewrites manually, randomize pair order, compare paired score differences with prompt-level bootstrap intervals, and retain helpfulness-changing cases as a separate intervention. No paid judge is necessary for a small human-labeled design.
7. Explain the pipeline in five sentences. Demonstrations teach a policy from selected responses. Comparison data identifies preferences among alternatives. An RM generalizes those comparisons and supplies an inexpensive training-time proxy. PPO can optimize the proxy while limiting divergence from a reference, but must be audited for exploitation. DPO and verifiable-reward methods show that a separately trained RM is a choice, not a logical necessity.

Spaced practice: tomorrow compute the chosen/rejected derivatives at d=log 3, then locate the scored token in one left-padded and one right-padded row. In a week, design a same-prompt correct/incorrect × concise/expanded panel and explain why its paired comparisons answer a different question from overall reward–length correlation.

Sources and boundaries

Primary references: InstructGPT, TRL 0.23.1 reward trainer, HH-RLHF, UltraFeedback, RewardBench, JudgeLM, Nemotron reward card, process supervision, reward overoptimization, and DeepSeek-R1. Executed checks are limited to the generated CPU preference experiment, fake-tokenizer/fake-model scoring and mode restoration, descriptive-statistic/paired-effect arithmetic, and authored refusal-outcome accounting. No downloaded preferences, pretrained weights, GPU training, judge API, attack, PPO run or real-model refusal measurement is claimed. NumPy 2.3.3 Pearson definition supports the statistic, not a causal threshold. XSTest, version 3 motivates separate safe/unsafe refusal controls; its abstract supports the evaluation distinction, not a causal attribution to any particular training dataset. The six authored outcomes here are not XSTest data or results.

Course navigation

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 62 / 80 · Module M10 · ~2 hours

Previous: ← DL S061 · Next: DL S063 →

Optional reading

Use these links for additional background. Version-specific behavior is documented in the primary references above.

• Optional resource 1
• Optional resource 2
• Optional resource 3
• Optional resource 4