DL S064 · DPO — RLHF without the RL

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 64 / 80 · Module M10 · ~2 hours

The story

The algebraic trick that killed PPO

Here's the story that made every open-source alignment team drop PPO in 2023.

Rafailov, Sharma, Mitchell et al. (Stanford, May 2023) were sitting with the RLHF objective — maximize E_{y ~ πθ}[r(x, y)] − β·KL(π_θ || π_ref) — and asked a question nobody had bothered to ask before: what does the optimal policy look like for this objective? Not "what does PPO converge to" — what's the actual closed-form solution?

Turns out it's:

$\pi^*(y|x) = \frac{1}{Z(x)}\pi_{ref}(y|x) \cdot \exp\Bigl(\tfrac{1}{\beta} r(x, y)\Bigr)$

Just an exponential tilt of the reference policy by the reward. Boltzmann distribution. First-year statistical mechanics.

Now invert. Solve for r as a function of π*:

$r(x, y) = \beta \log\frac{\pi^*(y|x)}{\pi_{ref}(y|x)} + \beta \log Z(x)$

The reward is just the β-scaled log-ratio of policies. Substitute this expression for r into the Bradley-Terry preference loss (S062) — the Z(x) partition function cancels because Bradley-Terry only cares about reward differences — and you get:

$\mathcal{L}_{DPO}(\theta) = -\mathbb{E}_{(x, y_c, y_r)}\Bigl[\log \sigma\bigl(\beta \log\tfrac{\pi_\theta(y_c|x)}{\pi_{ref}(y_c|x)} - \beta \log\tfrac{\pi_\theta(y_r|x)}{\pi_{ref}(y_r|x)}\bigr)\Bigr]$

No reward model. No rollouts. No value function. No PPO. Just a supervised loss on preference pairs that you can plug directly into any SGD loop.

The moment this paper hit arxiv, every open-source alignment team dropped PPO within 6 months. Zephyr-7B (November 2023) was the first widely-known DPO fine-tune to demolish PPO baselines. Since then: OpenHermes-DPO, Nous-Hermes-DPO, Notus, Tulu-2-DPO, Llama-3-Instruct-DPO variants, Mistral's official Instruct models — all DPO.

Today we derive the loss, implement it in ~40 lines, and run it end-to-end.

Prerequisites

• Session 062 (reward modeling) — DPO's derivation starts from the Bradley-Terry loss.
• Session 063 (RLHF PPO) — DPO's whole point is that it solves the same optimization problem without RL. Skip 063 and you'll wonder what problem DPO is solving.
• Session 058 (SFT) — the reference model is a frozen copy of your SFT'd model, exactly as in PPO.

1 · The derivation, in one page

Start with the KL-regularized RL objective:

$\max_\pi \mathbb{E}_{x, y \sim \pi(\cdot|x)}\bigl[r(x, y)\bigr] - \beta \cdot \text{KL}\bigl(\pi(\cdot|x) \|\ \pi_{ref}(\cdot|x)\bigr)$

This is exactly what PPO is trying to solve. Not an approximation; the same objective.

Step 1: Find the closed-form optimal policy

Take the gradient w.r.t. π(y|x), subject to the constraint Σ_y π(y|x) = 1. Standard Lagrangian gives:

$\pi^*(y|x) = \frac{1}{Z(x)}\pi_{ref}(y|x) \cdot \exp\bigl(\tfrac{1}{\beta} r(x, y)\bigr)$

where Z(x) = Σ_y' π_ref(y'|x)·exp(r(x,y')/β) is a normalization (partition function). This is Boltzmann distribution over responses, tilted by β-scaled reward.

Step 2: Solve for reward as a function of π and π_ref

Take log of both sides:

$\log \pi^*(y|x) = \log \pi_{ref}(y|x) + \tfrac{1}{\beta}r(x, y) - \log Z(x)$

Rearrange:

$r(x, y) = \beta \log\frac{\pi^*(y|x)}{\pi_{ref}(y|x)} + \beta \log Z(x)$

The reward equals the β-scaled log-ratio, up to a Z(x) term that depends only on x, not on y.

Step 3: Plug into Bradley-Terry

The Bradley-Terry preference loss (S062) is:

$\mathcal{L}_{BT} = -\log \sigma\bigl(r(x, y_c) - r(x, y_r)\bigr)$

Substitute the expression for r. The β·log Z(x) terms are the same for chosen and rejected — they cancel:

$r(x, y_c) - r(x, y_r) = \beta \log\frac{\pi^*(y_c|x)}{\pi_{ref}(y_c|x)} - \beta \log\frac{\pi^*(y_r|x)}{\pi_{ref}(y_r|x)}$

Substituting π_θ for π* (since we're training π_θ to be the optimal policy):

$\boxed{\mathcal{L}_{DPO}(\theta) = -\mathbb{E}_{(x, y_c, y_r)}\Bigl[\log \sigma\bigl(\beta \log\tfrac{\pi_\theta(y_c|x)}{\pi_{ref}(y_c|x)} - \beta \log\tfrac{\pi_\theta(y_r|x)}{\pi_{ref}(y_r|x)}\bigr)\Bigr]}$

That's it. Four log-probabilities per training example (θ on chosen, θ on rejected, ref on chosen, ref on rejected), sigmoid, negative log. Same class of loss as reward modeling — pairwise logistic — just with a different quantity inside the sigmoid.

1.1 The intuition

What is the training doing? For each preference pair:

• Increase log π_θ(y_c|x) relative to log π_ref(y_c|x). "Make the chosen response more likely than the reference did."
• Decrease log π_θ(y_r|x) relative to log π_ref(y_r|x). "Make the rejected response less likely than the reference did."

By exactly the amount that would maximize the Bradley-Terry likelihood of the human preference. And β controls how aggressively we do this — small β means "move a lot per pair," large β means "move a little."

2 · DPO in ~40 lines of PyTorch

Here's the entire loss function. Read it, then we'll unpack.

import torch
import torch.nn.functional as F
 
def dpo_loss(
    policy_chosen_logps: torch.Tensor,      # (B,) — sum of log-probs of chosen response tokens under policy
    policy_rejected_logps: torch.Tensor,    # (B,) — same for rejected
    ref_chosen_logps: torch.Tensor,         # (B,) — under reference
    ref_rejected_logps: torch.Tensor,       # (B,) — under reference
    beta: float = 0.1,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    """
    Returns: (loss, chosen_reward, rejected_reward)
    """
    pi_logratios  = policy_chosen_logps - policy_rejected_logps
    ref_logratios = ref_chosen_logps    - ref_rejected_logps
    logits = beta * (pi_logratios - ref_logratios)     # (B,)
 
    loss = -F.logsigmoid(logits).mean()
 
    # For monitoring: the "implicit rewards"
    chosen_reward   = beta * (policy_chosen_logps   - ref_chosen_logps).detach()
    rejected_reward = beta * (policy_rejected_logps - ref_rejected_logps).detach()
 
    return loss, chosen_reward, rejected_reward

That's the whole DPO loss. 15 lines.

2.1 Computing the log-probs

The one non-trivial piece is computing "the sum of log-probs of the response tokens under a model." Given a model, prompt, and response:

def compute_logps(model, prompt_ids, response_ids):
    """
    Sum of log P(y_t | x, y_<t) for all tokens in response.
    """
    input_ids = torch.cat([prompt_ids, response_ids], dim=-1)
    logits = model(input_ids).logits                    # (1, T, V)
    # Shift: at position t, logits[:, t, :] predicts token t+1
    logits = logits[:, :-1, :]
    labels = input_ids[:, 1:]
    # Mask out prompt tokens (we only sum over response tokens)
    prompt_len = prompt_ids.size(1)
    mask = torch.zeros_like(labels)
    mask[:, prompt_len-1:] = 1                          # response tokens only
 
    log_probs = F.log_softmax(logits, dim=-1)
    per_token_logp = log_probs.gather(-1, labels.unsqueeze(-1)).squeeze(-1)
    return (per_token_logp * mask).sum(dim=-1)          # (B,)

Four such computations per DPO step (policy on chosen/rejected, ref on chosen/rejected). The reference computations are torch.no_grad() since ref is frozen. Note: this can be batched to two forward passes total (concatenate chosen+rejected in the batch dim) — TRL does this optimization.

2.2 Memory bill

Compared to PPO:

• Policy: same size, trainable ✓
• Reference: same size, frozen ✓
• NO reward model — save 14GB on a 7B RM
• NO value model — save 14GB on a 7B value head

Net for a 7B setup: DPO ≈ 28GB peak. PPO ≈ 56GB peak. DPO is exactly half the memory of PPO. This is why DPO fits on 1× A100-40GB and PPO needs 1× A100-80GB minimum.

Also: no rollouts. DPO does two forward passes per example (chosen + rejected); PPO does 1 generation forward + 4 forward passes for log-probs/rewards/values per example, plus 2-4 mini-epochs of optimization. Roughly DPO is 5-10× faster per training example than PPO.

3 · Training a DPO model with TRL

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer, DPOConfig
 
SFT_MODEL = "./sft-mistral"   # from S058
 
tok = AutoTokenizer.from_pretrained(SFT_MODEL)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token
 
policy = AutoModelForCausalLM.from_pretrained(SFT_MODEL, torch_dtype="bfloat16")
ref    = AutoModelForCausalLM.from_pretrained(SFT_MODEL, torch_dtype="bfloat16")
 
# UltraFeedback: prompt, chosen response, rejected response
ds = load_dataset("HuggingFaceH4/ultrafeedback_binarized", split="train_prefs")
 
cfg = DPOConfig(
    output_dir="./mistral-dpo-uf",
    beta=0.1,
    num_train_epochs=1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=5e-7,                # 10× lower than SFT!
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    bf16=True,
    max_length=1024,
    max_prompt_length=512,
    logging_steps=25,
)
 
trainer = DPOTrainer(model=policy, ref_model=ref, args=cfg,
                     train_dataset=ds, tokenizer=tok)
trainer.train()

That's the whole thing. Compare to the PPO setup — no reward model to load, no value head, no rollouts, no PPO-epoch loop. It's a normal supervised training run.

On 1× A100-40GB, DPO of Mistral-7B on 60K UltraFeedback pairs, 1 epoch, LoRA r=16: ~3 hours. Full-precision no LoRA: ~6 hours. PPO of the same model on the same data: 24-48 hours.

3.1 The LR sensitivity is critical

Notice: LR = 5e-7. That's 10× lower than SFT, 20× lower than LoRA-SFT. This is not optional.

Why? Because in DPO, each step's gradient depends on the log-ratio of policy to reference. Even a small update to policy log-probs (say, +0.5 nats on some chosen tokens) shifts the log-ratio by 0.5 — which through the sigmoid loss and β=0.1 multiplication produces a nontrivial gradient. A big LR + this feedback = policy diverges from reference within ~50 steps and generation quality collapses.

Empirical LR ranges for DPO:

• Mistral-7B full: 5e-7
• Mistral-7B LoRA: 5e-6
• Llama-3-8B full: 3e-7
• Any model, β=0.5: bump LR 2×

4 · Reading a DPO training log

Three numbers to watch, all logged by TRL:

• loss: starts at exactly log(2) ≈ 0.693 (because at init, policy = reference, so both log-ratios equal, difference is 0, sigmoid(0) = 0.5). Should drop to 0.2-0.4 over training.
• rewards/chosen: the "implicit reward" β·(log π_θ(y_c|x) − log π_ref(y_c|x)). Should grow from ~0 to +2 or +5 over training. This is the policy putting more mass on chosen responses.
• rewards/rejected: same for rejected. Should shrink from ~0 to negative values. Policy is pushing mass away from rejected.
• rewards/accuracy: fraction of pairs where implicit chosen reward > implicit rejected reward. Should climb from 50% to 70-85%.
• rewards/margins: chosen - rejected implicit reward gap. Should grow.

Two failure modes:

• Both rewards drop. If both chosen and rejected implicit rewards go strongly negative, the policy is drifting away from the reference on all responses — losing coherence, hallucinating more, generating shorter. Cause: LR too high. Fix: cut LR.
• Rewards stall or accuracy plateaus at ~55%. DPO isn't learning much. Cause: (a) preference data is too subtle / noisy, (b) β too high (updates too small), (c) reference model already at optimum for this data (rare but possible). Fix: try β=0.05, add data.

5 · DPO variants — a whirlwind tour

Since Rafailov's paper, several DPO variants have shown up. TRL supports most as loss_type in DPOConfig. Worth knowing about:

• Sigmoid (default) — the original DPO loss. Works. Occasional overfitting on noisy pairs.
• IPO (Identity Preference Optimization) — replaces the sigmoid with a squared loss on log-ratio differences. More robust to label noise; harder to overfit. Set loss_type="ipo".
• KTO (Kahneman-Tversky Optimization) — doesn't require pairs, only "was this response good or bad?" binary labels. Much more data-efficient collection. Set loss_type="kto_pair".
• cDPO — allows label noise (fraction ε of preferences are flipped). Robustifies DPO on cheap synthetic data.
• SimPO — removes the reference model entirely. Simpler, sometimes as good, no ref-model memory. Newer, less battle-tested.
• ORPO — one-shot: does SFT and preference optimization in the same loss. Saves you a training stage. Interesting for small datasets.

For your first DPO run: use default sigmoid. If you find you're overfitting, switch to IPO. If you can only get binary labels, use KTO.

6 · When DPO beats PPO, when it doesn't

DPO strictly wins when:

• Your preference dataset is fixed (no fresh preference collection during training).
• You need memory efficiency (single 24-40GB GPU).
• You want reproducibility (DPO is deterministic given the data; PPO's rollouts add stochasticity).
• You want short training time (5-10× faster).

PPO can still edge out when:

• Iterative RLHF — you keep collecting fresh preferences with the current policy. PPO's online setup handles distribution shift naturally; DPO would need repeated retraining from scratch on ever-growing datasets.
• Multi-objective reward — combining helpfulness + safety + conciseness + tool-use into one reward. PPO's r() function is fully general; DPO wants a single preference ranking.
• Frontier scale — at very high quality tiers, the extra expressiveness of PPO's off-policy correction still buys measurable quality. This is empirical, not proven, and only shows up when everything else is world-class.

For 95% of open-source and enterprise use cases in 2024: DPO. That's why Zephyr, Notus, Tulu-2, all recent Mistral Instruct models, and probably 90% of "aligned" open models on HuggingFace since November 2023 are DPO. PPO is now the specialty algorithm.

7 · Diagram — DPO vs PPO side by side

Left: 4 models, generation loop, RL machinery. Right: 2 models, no generation, supervised loss.

8 · The 2024–2025 preference-optimization zoo, in depth

Section 5 gave a whirlwind tour. Since Rafailov et al.'s DPO paper (May 2023), the field spawned a lot of successors. Here's the ones that actually landed on production leaderboards, with when to reach for each.

8.1 IPO — Identity Preference Optimization (Azar et al., DeepMind 2023)

DPO's sigmoid loss can push arbitrarily hard on "easy" preference pairs (where chosen and rejected are already well-separated in the policy). This overfits when the reference model is already opinionated. IPO replaces the sigmoid with a squared loss on the log-ratio difference — the gradient is bounded, so easy pairs contribute little and hard pairs dominate.

Empirically: on noisy synthetic preferences, IPO beats DPO by 1–2 MT-Bench points. On clean human preferences, they're within noise.

Use when: your preferences come from LLM-as-judge or noisy annotators.

8.2 KTO — Kahneman-Tversky Optimization (Ethayarajh et al., 2024)

KTO's genius: you don't need paired preferences. Given a bag of (prompt, response, thumbs-up/down) tuples, KTO applies prospect-theory-inspired loss (asymmetric gain vs loss weighting) to align the policy.

Why it matters: preference pair collection is 5–10× more expensive than single-response feedback. Every chatbot has thumbs-up/down data. KTO makes it directly usable for alignment. Ethayarajh showed KTO matches DPO on paired-data tasks and beats DPO when only binary signals are available.

Use when: you have unpaired binary feedback (production thumbs-up/down logs, safety flags, verifier signals).

Further reading: https://arxiv.org/abs/2402.01306

8.3 ORPO — Odds-Ratio Preference Optimization (Hong et al., 2024)

DPO assumes you already SFT'd. ORPO combines SFT and preference optimization in one loss: L = SFT_loss + λ * odds_ratio_penalty(rejected). One training stage instead of two. Half the compute, similar quality.

Mistral-ORPO-α (Feb 2024) hit AlpacaEval 2.0 = 12.20 with just ORPO on top of Mistral-7B-base — no separate SFT phase.

Use when: compute-tight and want a single-stage recipe; small teams.

Further reading: https://arxiv.org/abs/2403.07691

8.4 SimPO — Simple Preference Optimization (Meng et al., Princeton 2024)

DPO uses a frozen reference model $\pi_{\text{ref}}$ (usually the SFT'd checkpoint) as an anchor. SimPO throws it out entirely. The loss becomes:

$L_{\text{SimPO}} = -\log \sigma\left(\frac{\beta}{|y_w|}\log \pi_\theta(y_w|x) - \frac{\beta}{|y_l|}\log \pi_\theta(y_l|x) - \gamma\right)$

Two changes vs DPO: length normalization ($|y_w|$, $|y_l|$ in denominators — fixes DPO's known length bias) and a target margin $\gamma$ (a bias term the policy has to overcome).

Result: SimPO-tuned Llama-3-Instruct hit AlpacaEval 2.0 = 44.7% (LC win rate), state-of-the-art open at the time of publication. Beat DPO by ~7 pts. Uses half the memory (no ref model).

Use when: you want the current best open recipe and your dataset is clean.

Further reading: https://arxiv.org/abs/2405.14734

8.5 SPPO — Self-Play Preference Optimization (Wu et al., 2024)

SPPO plays the policy against itself: at each iteration, generate $K$ responses per prompt from the current policy, rank them with an RM (or LLM-as-judge), and update to increase probability of top-ranked vs bottom-ranked. This is DPO with on-policy samples — like PPO but without the PPO complexity.

Why it wins: DPO with a fixed preference dataset suffers distribution shift (you're training on preferences over some old model's outputs, not the current model's). SPPO fixes this by resampling every iteration. Downside: you need an RM/judge in the loop, so you're back to spending compute on preference collection.

Tulu-3 uses SPPO-style on-policy preferences as its DPO stage. This is a big part of why it beats Llama-3.1-Instruct.

Further reading: https://arxiv.org/abs/2405.00675

8.6 The 2025 comparison table

8.7 Constitutional AI and RLAIF

Anthropic's Bai et al. (2022) Constitutional AI replaced human preferences with AI-generated preferences guided by a written constitution ("prefer the response that is helpful and honest and non-harmful"). This is called RLAIF (RL from AI Feedback).

By 2025, RLAIF is standard for scaling preference collection. Lee et al. (Google 2024) showed RLAIF matches RLHF on summarization and dialogue tasks. Anthropic's Claude models are trained with heavy RLAIF; OpenAI's newer models use LLM-as-judge as one of many signal sources.

The practical shift: you no longer need $1M in human labeling to train a preference model. You need a good frontier model, a good constitution, and$50 in API credits.

Further reading: https://arxiv.org/abs/2212.08073 · https://arxiv.org/abs/2309.00267

9 · Retention scaffold

Recall

Q1. Write down the DPO loss.

Q2. Why does the partition function Z(x) drop out of the DPO training loss?

Q3. How much memory does DPO save vs PPO for a 7B policy?

Q4. What LR should you use for full-parameter DPO of a 7B model, and why is it 10× lower than SFT?

Q5. Name one situation where PPO is still preferable to DPO.

Stretch prompt

The DPO derivation depends critically on the closed-form solution to the KL-regularized RL problem. Verify the closed form yourself: write out the Lagrangian, take the derivative w.r.t. π(y|x), set to zero, solve. Show that you recover π*(y|x) ∝ π_ref(y|x) · exp(r/β). (This is a rite of passage — every alignment researcher has done this on the back of a napkin at least once.)

In your own words

Explain to a colleague, in 4-5 sentences, why DPO doesn't need a reward model even though it's trained on preference data. Force yourself to name the "reward is a log-ratio" insight and the partition-function cancellation.

Spaced review

• S062 — Reward modeling. DPO's derivation starts from the Bradley-Terry loss you built there.
• S063 — RLHF PPO. DPO solves the same objective as PPO — you need to know what problem they're both trying to solve.
• S058 — SFT. DPO expects an SFT'd model as both policy and reference initialization. DPO on a base model is unstable and usually hurts.

Next session

S065 — Evaluation and red-teaming. You've now SFT'd, LoRA-fine-tuned, and DPO-aligned a model. Is it good? By what measure? Tomorrow we build a real evaluation harness: MMLU for knowledge, HumanEval for code, TruthfulQA for hallucination, custom domain evals, and a jailbreak/red-team battery. We'll also cover benchmark contamination — the reason half of "SOTA" scores don't reproduce.

Bring back tomorrow

• DPO loss: pairwise logistic on β-scaled log-ratios of policy vs reference.
• No RM, no value head, no rollouts. 2 forward passes per training example.
• LR must be small (~5e-7 for 7B full-precision).

Previous: ← DL S063 · Next: DL S065 →