DPO — RLHF without the RL

An implicit reward, not the end of reinforcement learning

Deep Learning & LLMs From Scratch · Session 64 / 80 · M10 · Fine-tuning + alignment.

Direct Preference Optimization (DPO) turns same-prompt preference comparisons into a policy-training loss. Its derivation links a KL-regularized reward objective to a Bradley–Terry preference model. It eliminates a separately fitted reward network and rollout/value estimation from the basic offline update, not all assumptions about reward or all possible uses of online RL.

The DPO paper v3 supplies the derivation and experiments. It does not show that every alignment team switched, that DPO always reaches the same solution as finite neural-network PPO, or that training is automatically deterministic. Prerequisites are SFT sequence probabilities/masks, preference reward modeling and the distinction between policy and reference from PPO. This lesson restores the derivation, complete masked loss, pinned trainer reference, beta/log interpretation and the preference-method survey with worked answers. CPU loss/mask/gradient checks are executed; no model fine-tune, dataset download or GPU experiment is claimed.

Derivation: eliminate the explicit reward parameterization

For one prompt x, hold a reward r and reference q fixed. Assume beta is positive, q has support wherever the optimized policy may put mass, and the normalizer is finite. Optimize over distributions p:

Step 1: solve the constrained variational problem

Use multiplier lambda for normalization:

Stationarity gives r_y−beta*(log(p_y/q_y)+1)+lambda=0. Therefore p_y=q_y*exp(r_y/beta)*exp(lambda/beta−1). Normalization determines the constant:

On the supported finite case the objective is concave (strictly so in interior probabilities), giving the optimum. This unrestricted distribution result is not a closed-form solution for finite neural-network parameters. A support-zero reference excludes that action from finite-KL policies.

Step 2: solve for the reward

Taking logs and rearranging,

The prompt-dependent additive term is unidentifiable from within-prompt rankings. This is why an implicit reward is meaningful primarily as a difference, not a calibrated standalone measure of human utility.

Step 3: Bradley–Terry cancels the partition function

For chosen y_w and rejected y_l sharing x, Bradley–Terry models preference probability as sigmoid(r(x,y_w)−r(x,y_l)). The two beta*log Z(x) terms cancel. Parameterizing the policy with theta gives

Four sequence log probabilities are needed, only two differentiated. Two unrelated prompts do not share Z; arbitrary cross-prompt comparisons do not satisfy this derivation. Real preferences can violate Bradley–Terry through intransitivity or context, and finite datasets can miss useful responses. These are model assumptions, not bookkeeping details.

Numeric pair, gradients and the beta distinction

Let reference log probabilities be chosen −4 and rejected −3; policy values −3.5 and −3.2 give relative rewards .5 and −.2 before multiplying by beta. Delta=.7; beta=.1 gives logit .07, preference probability about .51749 and loss .65876. At identical policy/reference, delta=0 and loss=log 2≈.693147.

Differentiate with respect to the unscaled margin:

At delta=0 the magnitude is beta/2. Therefore the old claim “smaller beta creates a larger per-example gradient” is false at this point. For the original fixed-reward regularized optimum, larger beta discourages deviation from reference. In a finite DPO training run, beta also changes the loss scale, saturation and optimizer interaction; one cannot infer final drift from the local gradient alone. Tune beta and learning rate jointly under held-out evaluation rather than declaring a universal .1–.5 safe range or mandatory 5e−7 learning rate.

DPO pushes the relative chosen-versus-rejected log-ratio margin. It need not raise chosen likelihood absolutely. If chosen changes from −4 to −5 and rejected from −3 to −6, relative changes are −1 and −3, so delta=2: both likelihoods fell yet the pair loss improved. This may reflect useful redistribution or degradation; inspect independent behavior. It is not by itself proof of “log-ratio explosion.”

Complete masked sequence log probabilities and DPO loss

A causal logit at position t predicts token t+1. Mark response tokens on the unshifted input positions, then shift labels and masks once. Prompt tokens, padding and intentionally excluded tokens must not enter the sequence sum. EOS may be a real response target even if the pad token uses the same ID; use position/attention masks rather than token-value equality. Standard DPO uses sums of response log probabilities; means define a different objective.

# CPU teaching implementation, exercised on PyTorch 2.9.0+cpu.
import math
import torch
import torch.nn.functional as F
 
def response_logps(logits, input_ids, response_mask, attention_mask):
    if logits.ndim != 3 or input_ids.ndim != 2 or logits.shape[:2] != input_ids.shape:
        raise ValueError("logits must be batch,time,vocabulary")
    if input_ids.numel() == 0 or input_ids.shape[1] < 2 or logits.shape[2] == 0:
        raise ValueError("nonempty batch, vocabulary and at least two positions required")
    if response_mask.shape != input_ids.shape or attention_mask.shape != input_ids.shape:
        raise ValueError("matching mask shapes required")
    if any(t.device != logits.device for t in (input_ids, response_mask, attention_mask)):
        raise ValueError("all tensors must share a device")
    if input_ids.dtype != torch.long or bool(((input_ids < 0) | (input_ids >= logits.shape[2])).any()):
        raise ValueError("input IDs, including padding, must be valid vocabulary indices")
    if not logits.is_floating_point() or not torch.isfinite(logits).all():
        raise ValueError("finite floating logits required")
    if any(not bool(((m == 0) | (m == 1)).all()) for m in (response_mask, attention_mask)):
        raise ValueError("binary position masks required")
    response, attended = response_mask.bool(), attention_mask.bool()
    if bool((response & ~attended).any()) or bool(response[:, 0].any()):
        raise ValueError("response targets must be attended and have a preceding logit")
    valid = response[:, 1:]
    if not bool(valid.any(dim=1).all()) or bool((valid & ~attended[:, :-1]).any()):
        raise ValueError("each row needs response targets with attended predecessors")
    labels = input_ids[:, 1:]
    selected = logits[:, :-1].log_softmax(-1).gather(-1, labels[..., None]).squeeze(-1)
    return selected.masked_fill(~valid, 0.).sum(dim=-1)
 
def dpo_loss(policy_chosen, policy_rejected, ref_chosen, ref_rejected, beta=.1):
    if not math.isfinite(beta) or beta <= 0:
        raise ValueError("beta must be finite and positive")
    values = (policy_chosen, policy_rejected, ref_chosen, ref_rejected)
    if policy_chosen.ndim != 1 or policy_chosen.numel() == 0 or any(v.shape != policy_chosen.shape for v in values):
        raise ValueError("four nonempty aligned sequence log-probability vectors required")
    if any(not v.is_floating_point() or v.dtype != policy_chosen.dtype or v.device != policy_chosen.device or not torch.isfinite(v).all() for v in values):
        raise ValueError("finite log probabilities on one floating dtype/device required")
    chosen_reward = beta * (policy_chosen - ref_chosen.detach())
    rejected_reward = beta * (policy_rejected - ref_rejected.detach())
    margin = chosen_reward - rejected_reward
    return F.softplus(-margin).mean(), chosen_reward.detach(), rejected_reward.detach()
 
pc = torch.tensor([-3.5], dtype=torch.float64, requires_grad=True)
pr = torch.tensor([-3.2], dtype=torch.float64, requires_grad=True)
rc, rr = torch.tensor([-4.], dtype=torch.float64), torch.tensor([-3.], dtype=torch.float64)
loss, chosen, rejected = dpo_loss(pc, pr, rc, rr)
loss.backward()
assert pc.grad.item() < 0 < pr.grad.item()
assert abs(loss.item() - .6587595555) < 1e-8
same, _, _ = dpo_loss(rc, rr, rc, rr)
assert abs(same.item() - float(torch.log(torch.tensor(2., dtype=torch.float64)))) < 1e-12
 
# Uniform logits: two response targets contribute exactly -2*log(3).
ids = torch.tensor([[0, 1, 2, 0, 0]])
attention = torch.tensor([[1, 1, 1, 1, 0]])
mask = torch.tensor([[0, 0, 1, 1, 0]])
uniform = torch.zeros(1, 5, 3, dtype=torch.float64)
score = response_logps(uniform, ids, mask, attention)
torch.testing.assert_close(score, torch.tensor([-2*__import__("math").log(3)], dtype=torch.float64))

The helper consumes input IDs, not a label array containing −100: every ID, even at padding positions, must be a legal vocabulary index because gather happens before masking. Response and attention masks are binary; a response target on padding or at position zero is rejected rather than silently dropped. Include a real prompt/BOS predecessor for the first response token. Each selected target must have an attended predecessor. Attention/position IDs must still make the model's causal logits correct; these local checks do not reconstruct the model's attention topology. Finite logits are required even on unused positions to avoid NaN gradients leaking through softmax backward.

A nonuniform check catches bugs that uniform logits cannot: if target token probabilities at the two correct preceding positions are .3 and .1, the sequence score is log(.3)+log(.1)=log(.03)≈−3.506558. Selecting unshifted positions with probabilities .2 and .5 instead gives log(.1)≈−2.302585; averaging correct token log probabilities gives approximately −1.753279. Both are finite and both are the wrong standard-DPO score. For a selected target k, the derivative of its log probability with respect to vocabulary logit j is 1[j=k]−softmax(logits)[j]; unused predictive positions contribute zero.

Batching chosen/rejected together can reduce launch overhead, but “four log probabilities” is not four independent model instances. Reference computations run under no-grad; cache them only for an exact reference revision, tokenizer, prompt template, truncation and mask. For the same prompt with extra right padding, logits at valid causal positions and loss must agree. Left padding requires correct position/attention handling by the model; a mask alone does not prove it.

TRL reference and the real memory bill

A policy plus a frozen reference removes the explicit reward/value networks used in a typical PPO pipeline, but policy gradients/optimizer state, activations, two responses per pair and cached/reference work remain. One 7B full-training policy can exceed an A100-40 GB before adding any reference. LoRA, reference caching and offload change that budget; none gives a universal “half the peak” or “5–10× faster” result.

The following unexecuted reference pins TRL 0.11.4, Transformers 4.45.2, datasets 3.0.1 and uses an operator-reviewed compatible PyTorch/CUDA lock. DPOTrainer's tagged constructor uses tokenizer; DPOConfig specifies lengths and reference precomputation. It reads a local SFT checkpoint and local JSONL rows with prompt/chosen/rejected strings. These must already be rendered with the exact tokenizer/template and split by prompt, not mined from the test set.

# Reference only: no network resolution, no claim this full-policy run fits 24/40 GB.
import json
import os
import torch
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer
 
base = os.environ["LOCAL_SFT_MODEL"]
tok = AutoTokenizer.from_pretrained(base, local_files_only=True, trust_remote_code=False)
if tok.pad_token_id is None:
    tok.pad_token = tok.eos_token
rows = []
with open(os.environ["LOCAL_PREFERENCE_JSONL"], encoding="utf-8") as f:
    for line in f:
        row = json.loads(line)
        if not all(isinstance(row.get(k), str) and row[k] for k in ("prompt", "chosen", "rejected")):
            raise ValueError("nonempty prompt/chosen/rejected strings required")
        if row["chosen"] == row["rejected"]:
            raise ValueError("identical responses do not express this binary preference")
        # Budget before the trainer adds its declared special tokens.
        if len(tok(row["prompt"], add_special_tokens=False).input_ids) > 480:
            raise ValueError("prompt too long; review instead of silently truncating")
        if any(len(tok(row["prompt"]+row[k], add_special_tokens=False).input_ids) > 1000
               for k in ("chosen", "rejected")):
            raise ValueError("pair too long; do not change its meaning by blind truncation")
        rows.append({k: row[k] for k in ("prompt", "chosen", "rejected")})
if not rows:
    raise ValueError("empty preference set")
policy = AutoModelForCausalLM.from_pretrained(base, local_files_only=True,
                                           trust_remote_code=False, torch_dtype=torch.bfloat16)
reference = AutoModelForCausalLM.from_pretrained(base, local_files_only=True,
                                              trust_remote_code=False, torch_dtype=torch.bfloat16)
reference.requires_grad_(False)
reference.eval()
policy.config.use_cache = False
cfg = DPOConfig(output_dir="dpo-smoke", beta=.1, max_steps=10,
                per_device_train_batch_size=1, gradient_accumulation_steps=2,
                learning_rate=5e-7, max_length=1024, max_prompt_length=512,
                bf16=True, logging_steps=1, report_to=[], loss_type="sigmoid")
trainer = DPOTrainer(model=policy, ref_model=reference, args=cfg,
                     train_dataset=Dataset.from_list(rows), tokenizer=tok)
trainer.train()
trainer.save_model("dpo-smoke/final")
tok.save_pretrained("dpo-smoke/final")

Ten updates and that learning rate are reproducibility/smoke-test choices, not a converged recipe. Inspect trainer-produced labels for both responses; prompt/response boundary tokenization can merge subwords, so test decoded examples rather than assuming independently tokenized substrings always concatenate. A real experiment adds a held-out prompt split, fixed base/SFT baseline, controlled LR/beta sweep, generation settings, preference evaluation and capability regressions. Loading a LoRA model is a distinct configuration and needs verified reference-adapter behavior; do not imply this full-policy code is already LoRA.

Original throughput adjudication (10 October 2026; rejected claim). The previous claim was Mistral-7B, 60K UltraFeedback pairs, one epoch, one A100-40GB: LoRA r=16 ≈3 hours, full-precision/no-LoRA ≈6 hours, versus PPO 24–48 hours. No run logs, exact training lock, precision/state placement, response-length distribution, reference caching, rollout budget or matched PPO setup accompany it. The DPO paper v3, §6 and Appendix C evaluates controlled sentiment, TL;DR summarization and Anthropic-HH dialogue with models up to 6B; it does not establish those Mistral/UltraFeedback timings. The local reference above is full-policy, not LoRA, and does not establish the original claimed 40GB fit. Reject all three timing estimates as unsupported, rather than convert them into measured throughput or quietly treat them as retired narrative.

For planning, first specify resident parameter/gradient/optimizer bytes and offload, then measure warm-up-excluded step time and actual token/pair counts on the intended hardware. For PPO include generation, reward/reference/value passes, update epochs and collection frequency; offline pair count alone is not matched PPO work. This protocol preserves the useful cost-comparison objective without teaching fabricated hours.

Reading DPO logs without false diagnoses

The tagged implementation logs rewards/chosen, rewards/rejected, rewards/accuracies and rewards/margins, plus raw log probabilities. Implicit rewards are beta-scaled reference log-ratios, not external truthfulness scores. At an exact initial policy/reference match with dropout disabled, loss is log 2 and margins zero. A strict chosen_reward > rejected_reward accuracy is zero on all exact ties, not automatically 50%; random tie-breaking would be a different reporting convention.

Falling chosen and rejected rewards can occur while the margin improves. To diagnose degradation, inspect absolute likelihood, held-out generation, response length, refusal, task accuracy and data shift. A noisy plateau can be mislabeled preferences, an optimizer/scale issue, insufficient training, an unsuitable reference or already-separated pairs. Do not automatically lower LR by ten or beta below .05 solely from a loss value. LR, beta, batch/token averaging, epochs and parameterization interact.

The negative log-sigmoid gradient tends to zero for an already-correct large positive margin. The old description that DPO pushes harder on easy pairs, and that IPO's squared-loss gradient is bounded, were both wrong. An unregularized separable logistic problem can still seek ever-larger margins over training; that is a different statement from individual easy-example gradient growth.

Preference objectives: restore the mechanisms, not a scoreboard

Hinge preference loss: a finite margin, not a sigmoid probability

TRL 0.11.4 DPOTrainer's tagged hinge branch under the default reference-enabled reverse-KL configuration computes relu(1 − beta*logits), where logits is the unscaled policy-minus-reference chosen/rejected log-ratio difference Delta above. Thus L_hinge=max(0,1−beta*Delta). Beta is positive; the required Delta margin is 1/beta, not one in every parameterization.

Below that margin the derivative with respect to Delta is −beta; above it the derivative is zero. At equality the mathematical subgradient is any value in [−beta,0]; the PyTorch ReLU implementation used here selects zero. By contrast, sigmoid DPO has derivative −beta*sigmoid(−beta*Delta): smooth, nonzero for every finite margin in exact arithmetic, tending to zero gradually. At beta=.1 and Delta=0, hinge has slope −.1 versus sigmoid's −.05. At Delta=12, hinge is satisfied and flat while sigmoid still has a small negative slope. Neither is the IPO squared loss, which can pull an overshot margin back toward its finite target.

Hinge stops spending pair-specific gradient after a margin is satisfied; sigmoid provides a smooth preference-likelihood model and continues a diminishing push. Hinge's kink and abrupt cutoff affect optimization, and its constant below-margin slope does not automatically handle mislabeled comparisons well. A satisfied pair may move again through shared parameters/other pairs. Hinge does not itself output a calibrated preference probability or guarantee noise robustness. In this TRL release, label smoothing is ignored for hinge (with a warning); the ordinary sigmoid branch supports the release's smoothed variant. Keep the current trainer example on loss_type="sigmoid" unless deliberately choosing a different objective.

# CPU hinge and sigmoid comparison, consuming the same Delta convention.
import math
import torch
import torch.nn.functional as F
 
def preference_margin_losses(delta, beta=.1):
    if not math.isfinite(beta) or beta <= 0:
        raise ValueError("finite positive beta required")
    if delta.ndim != 1 or delta.numel() == 0 or not delta.is_floating_point():
        raise ValueError("nonempty floating margin vector required")
    if not bool(torch.isfinite(delta).all()):
        raise ValueError("finite margins required")
    return F.relu(1 - beta * delta), F.softplus(-beta * delta)
 
margins = torch.tensor([0., 10., 12.], dtype=torch.float64, requires_grad=True)
hinge, sigmoid = preference_margin_losses(margins)
gh = torch.autograd.grad(hinge.sum(), margins, retain_graph=True)[0]
gs = torch.autograd.grad(sigmoid.sum(), margins)[0]
torch.testing.assert_close(hinge, torch.tensor([1., 0., 0.], dtype=margins.dtype))
torch.testing.assert_close(gh, torch.tensor([-.1, 0., 0.], dtype=margins.dtype))
assert bool((gs < 0).all())
torch.testing.assert_close(gs[0], torch.tensor(-.05, dtype=margins.dtype))

Practice and answer: At beta=.2, a Delta of six exceeds the target five, so hinge contributes zero gradient; sigmoid still reinforces the preferred margin. A below-target Delta of four has hinge slope −.2. Swapping chosen/rejected negates Delta and can reactivate hinge; setting beta smaller raises the target margin while lowering the active slope, so it is not simply a larger immediate update.

cDPO and IPO

A simple noisy-label/cDPO loss is −(1−epsilon)log sigmoid(z)−epsilon log sigmoid(−z), with epsilon an assumed flipped-label fraction. For a single unconstrained logit its optimum is log((1−epsilon)/epsilon) when epsilon lies strictly between zero and one. Epsilon=.1 gives log 9, a finite preferred margin; epsilon=0 recovers the ordinary logistic tendency toward an unbounded correct logit. The noise model is a choice, not permission to relabel inconvenient data.

IPO, Azar et al. v1 (2023) comes from a general preference-probability objective, not just “more robust DPO.” A common paired form uses squared error of a policy/reference log-ratio difference to a finite target 1/(2*tau). Its derivative in that difference is 2*(difference−target), not bounded. Sequence sum/mean and temperature conventions must follow the selected implementation; TRL's IPO branch uses different log-probability averaging behavior from sigmoid DPO. A finite target can discourage indefinitely growing a correctly separated margin, but does not guarantee better noisy-data scores.

Original IPO benchmark adjudication (10 October 2026; rejected claim). The claim that IPO beats DPO by 1–2 MT-Bench points on noisy synthetic preferences, while being within noise on clean human preferences, is unsupported by Azar et al. v1, §§5.3–5.4 and conclusion. Its illustrative experiments use small context-free bandits, including three-action policies, 18,000 Adam steps and ten seeds; it explicitly leaves language-model preference training to future work. These experiments support studying sampled-preference overfitting, not that MT-Bench effect size or the clean-data equivalence claim. Reject both empirical generalizations. Even the later SimPO v1 Table 4 reports model- and judge-specific IPO/DPO scores, not the claimed controlled noisy-versus-clean experiment. A finite-target loss is a mechanism to test under fixed labels, models and judge settings, not a substitute for evidence of a one-to-two-point gain.

KTO: unpaired desirability

Ethayarajh et al. v1 (2024) use prospect-theory-inspired utility with desirable/undesirable labels rather than same-prompt response pairs. This can use thumbs-up/down data, but production feedback has exposure, selection and user-population bias. Positive/negative weighting and a reference/KL baseline matter. A KTO trainer with unpaired labels is not the same as a historical kto_pair option in a DPO trainer; check release-specific data schema and semantics. Neither collection-cost ratios nor universal quality parity follow from the label format.

ORPO: joint likelihood and odds preference

Hong et al. v1 (2024) combine chosen-response likelihood with an odds-ratio preference term without a separate reference model. If a response score p is defined from its length-normalized log likelihood, its odds are p/(1−p). A preference penalty compares chosen and rejected log odds via negative log-sigmoid; it is not simply an independent penalty on the rejected response. The positive likelihood term retains a direct pull toward chosen data. One stage can simplify a pipeline, but does not imply exactly half the compute of another workflow or make SFT-quality data optional.

Original ORPO result adjudication (10 October 2026; corrected claim). The old “Mistral-ORPO-α (Feb 2024), AlpacaEval 2.0=12.20” conflated checkpoint and date. Hong et al. v1, Table 1 and §6.1, titled Reference-free Monolithic Preference Optimization with Odds Ratio and dated 12 March 2024, reports α=11.33% and β=12.20% for the 7B Mistral checkpoints. The α run uses UltraFeedback; β uses its cleaned version with the same configuration. Both exemplify ORPO on a pretrained Mistral model without a separate preceding SFT phase; ORPO's joint loss still contains supervised chosen-response likelihood.

The paper's §5.2 describes AlpacaEval 2.0 with GPT-4-turbo as evaluator and comparisons against GPT-4 responses. Table 1's values are the reported win-rate percentages, not the later SimPO table's length-controlled metric, an MT-Bench score or a direct α-versus-β win rate. The February attribution is not supported by this version and is corrected to the paper's March date, not asserted as the checkpoint's independently verified release date. These are attributed paper results, not local benchmark measurements; keeping α/β and metric definitions separate is the substantive correction.

SimPO: mean log probability and a margin

Meng et al. v1 (2024) use average response log probability as a reference-free implicit reward plus a target margin:

This changes the objective, not just numerical stability. Length normalization can address a particular source of length effects; it does not remove preference-label verbosity bias or ensure every benchmark improves. Removing reference inference saves that component, not necessarily half of peak memory. The paper's benchmark results are attributed to its experimental setup, not a timeless recommendation that clean data always implies SimPO.

Original SimPO result adjudication (10 October 2026; supported number, corrected comparison). SimPO v1 (23 May 2024), Tables 1, 2 and 4 supports 44.7% length-controlled AlpacaEval 2 win rate for Llama3-Instruct-8B-SimPO. In the same Llama3-Instruct (8B) Table 4 block, DPO is 40.3% LC, so the improvement is 4.4 percentage points, not the original “about seven.” Raw win rates are separately 40.5% versus 37.9%, a 2.6-point difference. The abstract's “up to 6.4” AlpacaEval improvement belongs to a different setup (Mistral-Base: 21.5−15.1); its “up to 7.5” concerns Arena-Hard, not this AlpacaEval LC comparison.

The governing protocol uses 805 AlpacaEval 2 questions, GPT-4-Preview-1106 as both baseline and judge, and reports LC and raw win rates separately. For this Instruct training setup, UltraFeedback prompts receive five responses from the starting SFT/Instruct model at temperature .8, with PairRM selecting the highest/lowest-scored pair in one data-generation pass. It is not simply the base-model UltraChat/SFT pipeline nor a head-to-head 44.7% win probability against DPO. Appendix A specifies temperature .9 for AlpacaEval evaluation in both Llama3 settings, distinct from .8 for training-pair generation, and records the hyperparameter search. The paper calls this its strongest 8B open-source model, not a universal winner over all open model sizes. Thus retain the source-supported 44.7 figure with its checkpoint/protocol, reject the mismatched seven-point interpretation, and do not infer a timeless “best open recipe” or half-peak-memory guarantee.

SPPO and online data refresh

Wu et al. v1 (2024) formulate preference alignment as a constant-sum two-player game and approximate a Nash policy via self-play updates. On-policy response generation and preference scoring supply fresh comparisons, with a different update target from simply rerunning DPO. SPPO is not a synonym for every pipeline that uses newly sampled preference pairs; do not label TÜLU's DPO stage SPPO without source support. Generation/judging costs, judge bias and evaluation leakage remain.

The table is about contracts, not a quality ranking or 2025 market share. Iterative variants can change where the data comes from.

Constitutional AI and RLAIF

Bai et al. v1 (2022) describe a supervised phase of self-critiques/revisions followed by AI-generated preferences, preference-model training and RL. Written principles supply human oversight; “no human labels identifying harmful outputs” is not “no human values or supervision.” Lee et al. v1 (2023) report an RLHF/RLAIF comparison on summarization. It does not establish universal parity on every dialogue task or disclose all modern proprietary recipes.

AI feedback changes label provenance, not the need for auditing. Record judge model/version, rubric/constitution, prompts, sampling, pair ordering and conflicts; evaluate a held-out human/task sample and possible judge self-preference. No fixed API budget guarantees a useful preference dataset, and no paid judging API is used in this review.

Length bias, method choice and worked answers

A dataset whose chosen responses are systematically longer can teach verbosity through both labels and the summed sequence objective. Blindly truncating the chosen response to match rejected length changes meaning and may invalidate the label. Instead audit/relabel length-controlled comparisons, diversify concise preferred examples, evaluate by length strata and consider an explicitly different normalized objective. Adding a “reward penalty” to DPO requires stating how it changes the loss; there is no external reward term to modify invisibly.

Choose better SFT data when the desired behavior is not demonstrated adequately; choose preference training when comparisons add a useful signal; choose online RL when fresh exploration and a reliable task reward justify its complexity. These are hypotheses for controlled baselines, not strict dominance claims. Preference methods can generalize beyond examples but cannot promise competence unsupported by the base/data/reward.

Original MT-Bench anecdote adjudication (10 October 2026; rejected claim). The earlier unnamed-team story claimed PPO 6.8 after three days of tuning, DPO 7.2 in six hours, then PPO 7.5 after another three days, followed by a two-week / 90%-quality-for-10%-effort rule. There is no identified team, checkpoint, preference split, run artifact, MT-Bench judge revision or evaluation configuration for that story. DPO v3, §6 reports sentiment, summarization and dialogue protocols, not this experiment. Its GPT-4 baseline win rates are not those MT-Bench ratings. Reject these exact scores, durations and effort ratios as unsupported evidence, not as a verified historical incident or an approved nonobjective retirement.

The valid decision lesson is to compare fixed starting checkpoints, data/judges, tuning budgets and independent task regressions; separately report measured training time and human engineering time. A method's best tuned result and its first successful trial answer different questions. Without those controls, neither the anonymous story nor the equations establish DPO or PPO dominance.

1. Four terms? Policy chosen, reference chosen, policy rejected, reference rejected, all for the same prompt and exact response mask; only policy terms differentiate.
2. Why Z cancels? It depends only on the shared prompt, so subtracting within-prompt rewards removes the same additive term. The complete Lagrange derivation above is the stretch solution.
3. What is beta doing at initialization? For unscaled margin zero, gradient magnitude is beta/2. A smaller beta does not mean a larger immediate gradient; the fixed-reward optimum and finite-run dynamics are separate questions.
4. Both implicit rewards reach −8: what failed? Not enough information. Inspect margin, absolute log probabilities, data/template/mask, drift and independent generation before blaming LR. Both can decline during a useful relative update.
5. Memory savings? DPO avoids separate RM/value roles in the basic update, but trainable state and activations remain. Count actual bytes/sharing/caches; do not answer “28 versus 56 GB peak.”
6. When PPO instead? Fresh on-policy exploration with trusted scalar or step rewards can justify rollout infrastructure. Iterative DPO is also possible; it does not require discarding all prior training each round.
7. Initial reward accuracy? Exact tie margins imply zero under strict greater-than comparison, while loss is log 2. A different tie convention must be declared.
8. Explain the implicit reward in four sentences. The KL-regularized optimum is an exponential tilt of a reference. Rearrangement expresses reward as a scaled policy/reference log ratio plus a prompt constant. Same-prompt Bradley–Terry differences cancel that constant. Thus policy parameters can fit preference likelihood without a separate reward network, while retaining preference-model and data assumptions.

Spaced review: tomorrow derive Z cancellation and inspect one padded pair; in a week compare IPO/SimPO/ORPO by their actual loss, not names. Evaluation and red-teaming next test gains and regressions not settled by preference accuracy. The preview covers MMLU knowledge tasks, HumanEval code tasks, TruthfulQA truthfulness questions (not a universal hallucination detector), custom domain evaluations, a jailbreak/red-team battery, and benchmark contamination through train/test or tuning leakage. Keep held-out tasks, metric definitions and contamination checks distinct; no evidence supports the old “half of SOTA scores” fraction. The earlier claim that you have already SFT-trained, LoRA-fine-tuned and DPO-aligned a model is not an execution result here: these model-training workflows remain unexecuted references, separate from the CPU teaching checks. Bring back four correct log probabilities, a beta/gradient distinction and a task-specific evaluation plan.

Previous: ← DL S063 · Next: DL S065 →

Original optional resource links

The original optional video/reading destinations are retained here for continuity, not as current API, price, benchmark or publication-date evidence. The versioned primary citations in the teaching above govern the corrected claims. These supplementary destinations were not all fetched or watched; a missing or changed resource does not replace the local derivations, procedures and answers.

• https://www.youtube.com/watch?v=QXVCqtAZAn4
• https://www.youtube.com/watch?v=zjrM-MW-0y0
• https://arxiv.org/abs/2305.18290
• https://huggingface.co/docs/trl/en/dpo_trainer