RLHF with PPO — Intuition + Minimal Code

PPO is a method, not an obituary or a safety certificate

Deep Learning & LLMs From Scratch · Session 63 / 80 · M10 · Fine-tuning + alignment.

Suppose an SFT policy can generate several plausible answers and a reward function scores them. Optimizing the reward may improve the behavior you care about, or exploit a weakness in the scoring rule. PPO supplies a practical clipped policy-gradient surrogate; an RLHF pipeline often adds a KL penalty to a frozen reference. Neither proves truthfulness, prevents all forgetting or makes an imperfect verifier unhackable.

The earlier “everything else is bookkeeping” slogan is misleading: saved rollout probabilities versus a frozen reference, response/termination masks, reward placement, advantage/value targets and finite update reuse determine the algorithm. Clipping is not a hard movement bound, and reference KL is not a safety guarantee; those requirements cannot be dismissed as bookkeeping.

Schulman et al., PPO v2 (2017) predates language-model RLHF. InstructGPT v1 (2022) reports SFT, human preference reward modeling and PPO fine-tuning. These establish historical mechanisms, not a claim that every later proprietary model uses PPO or that everybody abandoned it for DPO.

Prerequisites are SFT response masks, reward-model preferences and log probabilities. We derive the loop, compute GAE, implement a complete tiny CPU update, show the exact historical TRL interface and audit modern GRPO/RLVR comparisons. Tiny tabular CPU execution is not a language-model PPO reproduction. No weights, preference dataset, paid API, distributed or GPU run is used here.

Four functions, three policy identities

The policy generates tokens and is optimized. The reference is a frozen distribution used for a drift penalty. The reward function scores a response/trajectory; it can be a frozen learned model, a programmatic verifier or a combination. The value function estimates expected future shaped return from the prompt and prefix and is trained against returns. A scalar head on a shared backbone and a separate value network have different state/memory costs; four roles do not require four equally sized models simultaneously resident.

Keep current policy, saved rollout policy and frozen reference conceptually distinct. At collection time current equals rollout policy; after updates it does not. The reference can remain the original SFT policy across many collections. PPO's ratio compares current to rollout, while RLHF's reference KL compares policy to reference. Substituting the reference for the saved denominator changes the algorithm.

For an illustrative full 7B policy, 16 bytes/parameter of trainable state alone is 112 GB under a specified mixed-precision Adam representation. Two additional frozen BF16 7B networks would add 28 GB; a shared value head adds little parameter state while a separate trainable backbone adds much more. Activations and rollouts remain. The old “PPO peak=56 GB, DPO peak=28 GB” comparison counted only four versus two FP16 weight arrays, not training peaks. LoRA reduces trainable state but does not make four large frozen weight sets vanish.

Generation, scoring and updating

Collect prompts, sample responses under a defined sampler, save rollout log probabilities and values, score complete responses, and compute reference log probabilities at the same prefixes. Construct per-token shaped rewards with a terminal score at the actual last valid action. Compute detached advantages/returns, then perform a bounded number of minibatch updates on that fixed rollout. Discard/recollect before the data becomes too stale.

Sampling transforms matter: if collection uses temperature or truncation, the probabilities used in ratios must match the intended behavior-policy contract. Dropout or mismatched masks can make supposed “old=current” probabilities differ even before the first update. A stored old log probability must not be recomputed after changing parameters.

The earlier “the whole algorithm” description was only a collection/update outline. A working implementation must additionally specify terminal versus truncated bootstrapping, padding-aware GAE and loss reductions, frozen/detached old and reference quantities, actor/value losses (and any entropy term), and update limits plus independent behavioral diagnostics. The derivations and tiny terminal-trajectory loop below make those choices explicit; the outline alone does not.

From policy gradient to the clipped surrogate

For an action sequence sampled from a policy, the score-function identity weights gradients of log probability by return. A state-dependent baseline can reduce variance without changing the expected on-policy gradient under the usual assumptions. In language generation the state is prompt plus prefix, action is the next token, and termination is EOS or a declared boundary.

For token t, define

PPO maximizes the empirical surrogate

Minimize its negative in a normal optimizer. If A=2, rho=1.4 and epsilon=.2, min(2.8,2.4)=2.4: more probability for that beneficial sample gains no more clipped reward. If A=−2 and rho=.6, min(−1.2,−1.6)=−1.6: reducing that bad action's probability further receives no additional improvement. If A=−2 and rho=1.4, the minimum is −2.8, so the harmful move is still penalized. The sign is essential.

Clipping is not a hard bound on probability movement. Shared parameters, other samples and finite optimizer steps can move rho outside the interval. Gradient-norm clipping bounds an update direction's norm and can be used too; it is not the same objective. Ratios equal one immediately before the first update on an exactly matching collection snapshot—not necessarily throughout the first epoch after earlier minibatches have already updated parameters. Monitor the actual clip fraction and update KL rather than asserting an impossible all-epoch invariant.

Cross-entropy versus PPO: the shared derivative is not the shared objective. For a demonstrated token a at a teacher-forced prefix s, supervised CE minimizes −log pi_theta(a|s). Its gradient is −grad log pi_theta(a|s); with vocabulary logits z it is p−one_hot(a). Ordinary SFT draws these prefixes/actions from a fixed demonstration dataset, with nonnegative example weights. It has no rollout advantage or old-policy denominator.

On a fixed PPO rollout, hold old_logp and A detached. Away from a clipping kink, an active, unclipped actor-loss branch has gradient −rho*A*grad log pi_theta(a|s). At collection-time equality rho=1, this is advantage-weighted negative log-probability gradient: positive A reinforces, negative A discourages, and zero A contributes nothing. PPO's scalar surrogate is −rho*A, not −A*log pi globally. Only at that evaluation point, or using the detached local weight rho*A, do those gradients agree. When A is positive and rho exceeds 1+epsilon, or A is negative and rho is below 1−epsilon, the beneficial branch is flat and its actor gradient is zero. Harmful moves on the other side remain active. This signed, ratio-dependent, branch-dependent weighting is unlike ordinary CE.

PPO collects on-policy rollouts, then uses a limited number of updates on the saved batch. The action ratio corrects the local action-probability mismatch at the old sampled states; it does not fully correct the new trajectory/state distribution or turn PPO into unrestricted off-policy learning. Advantages estimate improvement over a baseline, not correctness labels. Reference KL, value loss and optional entropy are separate terms; do not hide them inside the CE comparison. PPO v2, Equations 1–2 and 7 defines these gradient and surrogate roles.

# CPU comparison: local CE gradient, signed PPO weighting, and clipping.
import torch
 
def ce_ppo_gradient_case(advantage, old_probability):
    z = torch.tensor([.4, -.2], dtype=torch.float64, requires_grad=True)
    lp = z.log_softmax(-1)[0]
    old_lp = torch.as_tensor(old_probability, dtype=z.dtype).log().detach()
    ratio = (lp - old_lp).exp()
    ce_grad = torch.autograd.grad(-lp, z, retain_graph=True)[0]
    actor = -torch.minimum(ratio * advantage,
                           ratio.clamp(.8, 1.2) * advantage)
    ppo_grad = torch.autograd.grad(actor, z)[0]
    return ratio.detach(), ce_grad, ppo_grad
 
p0 = torch.tensor([.4, -.2], dtype=torch.float64).softmax(-1)[0].item()
for advantage in [2., -2., 0.]:
    ratio, ce_grad, ppo_grad = ce_ppo_gradient_case(advantage, p0)
    torch.testing.assert_close(ratio, torch.tensor(1., dtype=torch.float64))
    torch.testing.assert_close(ppo_grad, advantage * ce_grad)
ratio, ce_grad, ppo_grad = ce_ppo_gradient_case(2., p0 / 1.4)
torch.testing.assert_close(ppo_grad, torch.zeros_like(ppo_grad))
ratio, ce_grad, ppo_grad = ce_ppo_gradient_case(-2., p0 / 1.4)
torch.testing.assert_close(ppo_grad, -2. * ratio * ce_grad)

Practice and answer: At rho=1 and A=−2, is PPO equivalent to training on the demonstrated token with ordinary CE? No: its local actor gradient is −2 times CE's, so gradient descent lowers rather than raises that sampled action's probability. At rho=1.4 and A=2 the actor gradient is zero, but CE's is generally nonzero. Neither statement bounds all shared-parameter probability changes.

KL shaping and what a sampled log ratio means

For a categorical next-token distribution,

A sampled action gives log pi(a)-log ref(a), which can be negative; its expectation under pi is the KL. In a rollout, pi here is the collection distribution when using saved old probabilities. The KL-regularized reward objective E[R]−beta*KL(policy||reference) discourages cumulative drift; the clipped surrogate regulates incentives within rollout reuse. “Forward” and “reverse” naming depends on which distribution is treated as the target, so write the argument order. This order discourages placing policy mass where the reference has little support; the opposite order is a valid but different objective, not intrinsically “wrong RLHF.”

A common implementation sets per-token reward to −beta*(old_logp−ref_logp) and adds the scalar task reward at the last valid token. GAE and the actor then optimize a surrogate based on those fixed rewards. A directly differentiated current-policy KL loss is not automatically equivalent to detached reward shaping for finite samples, clipping and multiple epochs. Do not add both by accident. Beta depends on reward scale, token/sequence aggregation and objectives; no universal healthy sequence KL of six or fifteen exists.

A KL penalty is a distributional regularizer, not a certified boundary inside which a reward model is trustworthy. It can reduce drift toward unfamiliar outputs but cannot guarantee correct reward judgments even near the reference. Higher reward plus worse independent task/human results establishes proxy failure; high KL alone does not.

GAE by hand and a complete small implementation

Generalized Advantage Estimation, Schulman et al. v6, develops the exponentially weighted bias/variance tradeoff. Let old value be V, terminal next value zero, discount gamma and trace factor lambda. Temporal-difference residual and backward recursion are

Use a terminal mask on continuation and padding. For two valid tokens, rewards [0,1], values [.2,.4], gamma=1 and lambda=.9: deltas are [.2,.6], advantages [.74,.6], and lambda-return targets A+V=[.94,1]. A true terminal state and an artificial truncated trajectory may require different bootstrap policies. Lambda below one trades bias/variance and does not guarantee unbiased advantages for arbitrary value errors. A baseline does not force empirical mean advantage to zero; whitening is a separate optional operation using only valid tokens.

The code below implements terminal, right-padded trajectories and a two-action, two-step tabular environment. State zero precedes the first action; states one/two identify that first action for the second decision. Reward one is issued only for [1,1]. This is a complete executable teaching loop, not missing compute_gae, shuffle(...) or imaginary model methods. It omits truncation bootstrapping and large-model batching deliberately.

import torch
from torch import nn
 
def terminal_gae(rewards, old_values, mask, gamma=1., lam=.95):
    if rewards.ndim != 2 or rewards.numel() == 0 or rewards.shape != old_values.shape or mask.shape != rewards.shape:
        raise ValueError("nonempty matching batch/time shapes required")
    if mask.dtype != torch.bool or not bool(mask[:, 0].all()) or bool((mask[:, 1:] & ~mask[:, :-1]).any()):
        raise ValueError("boolean, nonempty right-padded trajectories required")
    if rewards.device.type != "cpu" or old_values.device != rewards.device or mask.device != rewards.device:
        raise ValueError("this teaching implementation requires CPU tensors")
    if not rewards.is_floating_point() or rewards.dtype != old_values.dtype:
        raise ValueError("matching floating reward/value dtype required")
    if not (0. <= gamma <= 1. and 0. <= lam <= 1.):
        raise ValueError("gamma and lambda must lie in [0,1]")
    if not torch.isfinite(rewards[mask]).all() or not torch.isfinite(old_values[mask]).all():
        raise ValueError("finite rewards and values required at valid actions")
    # Select before arithmetic: NaN * 0 is still NaN.
    rewards = rewards.detach().masked_fill(~mask, 0.)
    old_values = old_values.detach().masked_fill(~mask, 0.)
    adv = torch.zeros_like(rewards)
    following = torch.zeros(rewards.shape[0], dtype=rewards.dtype)
    next_value = torch.zeros_like(following)
    for t in range(rewards.shape[1] - 1, -1, -1):
        live = mask[:, t]
        delta = rewards[:, t] + gamma * next_value - old_values[:, t]
        following = torch.where(live, delta + gamma * lam * following, 0.)
        adv[:, t] = following
        next_value = old_values[:, t]
    return adv.detach(), (adv + old_values).masked_fill(~mask, 0.).detach()
 
def tiny_ppo(iterations=12):
    torch.manual_seed(19)
    logits = nn.Parameter(torch.zeros(3, 2, dtype=torch.float64))
    value = nn.Parameter(torch.zeros(3, dtype=torch.float64))
    reference = logits.detach().clone()
    optimizer = torch.optim.Adam([logits, value], lr=.03)
    metrics = []
    for iteration in range(iterations):
        with torch.no_grad():
            first = torch.distributions.Categorical(logits=logits[0]).sample((32,))
            second = torch.distributions.Categorical(logits=logits[1+first]).sample()
            actions = torch.stack([first, second], dim=1)
            states = torch.stack([torch.zeros_like(first), 1+first], dim=1)
            old_lp = logits[states].log_softmax(-1).gather(-1, actions[..., None]).squeeze(-1)
            ref_lp = reference[states].log_softmax(-1).gather(-1, actions[..., None]).squeeze(-1)
            old_v = value[states].clone()
            score = (actions == 1).all(dim=1).double()
            rewards = -.1 * (old_lp - ref_lp)
            rewards[:, -1] += score
            mask = torch.ones_like(actions, dtype=torch.bool)
            adv, returns = terminal_gae(rewards, old_v, mask)
            adv = (adv - adv.mean()) / adv.std(unbiased=False).clamp_min(1e-8)
        for epoch in range(3):
            new_lp = logits[states].log_softmax(-1).gather(-1, actions[..., None]).squeeze(-1)
            ratio = (new_lp - old_lp).exp()
            if epoch == 0:
                torch.testing.assert_close(ratio, torch.ones_like(ratio))
            actor = -torch.minimum(ratio*adv, ratio.clamp(.8, 1.2)*adv).mean()
            critic = .5 * (value[states] - returns).square().mean()
            loss = actor + .5 * critic
            optimizer.zero_grad()
            loss.backward()
            assert torch.isfinite(logits.grad).all() and torch.isfinite(value.grad).all()
            nn.utils.clip_grad_norm_([logits, value], 1.)
            optimizer.step()
        metrics.append({"reward": score.mean().item(),
                        "sampled_sequence_kl": (old_lp-ref_lp).sum(1).mean().item()})
    torch.testing.assert_close(reference, torch.zeros_like(reference))
    return metrics, logits.detach()
 
advantages, returns = terminal_gae(torch.tensor([[0., 1.]]),
                                    torch.tensor([[.2, .4]]),
                                    torch.tensor([[True, True]]), lam=.9)
torch.testing.assert_close(advantages, torch.tensor([[.74, .6]]))
torch.testing.assert_close(returns, torch.tensor([[.94, 1.]]))
metrics, learned_logits = tiny_ppo()
print("finite tabular PPO updates", len(metrics))

For variable response length, assign terminal score at valid_length−1, never the last padded column. Padding is not an action or a bootstrap value: with rewards [0,1,NaN], old values [.2,.4,NaN] and mask [1,1,0], gamma=1 and lambda=.9 must still give advantages [.74,.6,0] and returns [.94,1,0]. Multiplying the NaN sentinel by zero does not remove it; select/replace padded values before the recurrence. Conversely a NaN on a live action is a data error and must be rejected, not silently erased. The helper detaches rollout rewards and values and validates a boolean right-padded mask; a new rollout segment or time-limit truncation needs its own explicit continuation/bootstrap contract.

In a variable-length actor/critic update, reduce only valid entries, for example per_token_loss[mask].mean(); zeroing pads and taking the full rectangle's mean changes the scale with padding. Whitening likewise uses adv[mask], not padded zeros. The tabular loop has two real actions in every row, so its ordinary means already have this valid-token interpretation. In a GPU extension, create helper tensors on the input device; this CPU reference deliberately uses CPU allocation. Test masked GAE against a separate scalar recurrence, not a second copy of the tensor loop. The local checks do not validate a transformer rollout buffer, learned reward model or CUDA kernels.

Optional entropy bonus: change the loss, not just the dashboard. The original skeleton included ent_coef * entropy_bonus with entropy_bonus = −H. Retain that mechanism explicitly: a minimized loss is actor + vf_coef*critic − ent_coef*H, where ent_coef is finite and nonnegative. For the current categorical distribution at each valid sampled prefix, H = −sum_a p(a)*log p(a). Sum over the vocabulary before averaging over valid response positions; the response mask is not a mask of legal vocabulary actions. The full distribution entropy is not just the surprisal of the one sampled token.

For logits z_j, dH/dz_j = −p_j*(log p_j + H). Thus the entropy contribution to the minimized-loss gradient is ent_coef*p_j*(log p_j + H) (divided by the number of valid positions in a token mean). With positive coefficient, descent tends to flatten a nonuniform categorical distribution; at a uniform distribution the entropy gradient is already zero. This encourages exploration but may conflict with concentrating on a high-reward answer. It does not certify diversity, correctness or safety.

The PPO v2 paper, Equation 9 adds an entropy bonus to the maximized objective; minimizing negates that sign. It is optional: zero coefficient removes that incentive, and the unchanged tiny loop above deliberately uses zero. Reference-KL already changes the objective and often influences entropy, but is not identical to an entropy bonus: KL(p||q)=−H(p)−E_p[log q]. The tagged TRL 0.11.4 legacy loss returns policy loss plus weighted value loss and computes entropy for statistics, not an ent_coef loss term. Do not add an invented ent_coef argument to that historical PPOConfig. The following is an explicit CPU teaching extension, not a claim about that trainer's API.

# CPU entropy extension; no transformer or rollout generation is invoked.
import math
import torch
 
def masked_categorical_entropy(logits, response_mask):
    if logits.ndim != 3 or logits.shape[-1] == 0:
        raise ValueError("batch,time,nonempty-vocabulary logits required")
    if response_mask.shape != logits.shape[:2] or response_mask.dtype != torch.bool:
        raise ValueError("matching boolean response mask required")
    if logits.device.type != "cpu" or response_mask.device != logits.device:
        raise ValueError("CPU logits and mask required")
    if not logits.is_floating_point() or not bool(response_mask.any()):
        raise ValueError("floating logits and at least one valid position required")
    # Select positions BEFORE log_softmax: padded NaN sentinels never enter it.
    live_logits = logits[response_mask]
    if not bool(torch.isfinite(live_logits).all()):
        raise ValueError("finite logits at valid positions required")
    logp = live_logits.log_softmax(-1)
    return -(logp.exp() * logp).sum(-1).mean()
 
def with_entropy_bonus(actor_loss, critic_loss, logits, response_mask,
                       vf_coef=.5, ent_coef=0.):
    if any(not math.isfinite(c) or c < 0 for c in (vf_coef, ent_coef)):
        raise ValueError("finite nonnegative coefficients required")
    entropy = masked_categorical_entropy(logits, response_mask)
    # actor/critic are scalar valid-token means from the PPO update.
    if actor_loss.ndim != 0 or critic_loss.ndim != 0:
        raise ValueError("scalar actor and critic losses required")
    loss = actor_loss + vf_coef * critic_loss - ent_coef * entropy
    return loss, entropy.detach()  # only the metric is detached
 
mask = torch.tensor([[True, False]])
z = torch.nn.Parameter(torch.tensor([[[2., 0.], [float('nan'), float('nan')]]],
                                   dtype=torch.float64))
# Isolate entropy in this fixture. In the tiny loop, pass actor, critic,
# logits[states], mask and chosen coefficients instead of actor + .5*critic.
optimizer = torch.optim.SGD([z], lr=.1)
h_before = masked_categorical_entropy(z, mask).item()
loss, entropy_metric = with_entropy_bonus(z.new_zeros(()), z.new_zeros(()),
                                         z, mask, ent_coef=.05)
optimizer.zero_grad()
loss.backward()
assert not entropy_metric.requires_grad
assert z.grad[0, 0, 0] > 0 > z.grad[0, 0, 1]
torch.testing.assert_close(z.grad[0, 1], torch.zeros(2, dtype=z.dtype))
optimizer.step()
assert masked_categorical_entropy(z, mask).item() > h_before

Practice and answer: Logging entropy.item() under no-grad cannot create an entropy gradient. With ent_coef=0, the actor/value update is unchanged; with .05 the fixture's entropy increases after a step, because −.05*H actually participates in backward. A negative coefficient would encourage concentration instead, which is why this bonus helper rejects it. Padding contributes neither entropy nor gradient. These are CPU gradient checks, not measured language-model exploration or benchmark gains.

TRL 0.11.4: a versioned reference, not mixed APIs

The original example combined a value-head PPOTrainer.step interface with a reward_model= constructor argument belonging to a different style. TRL 0.11.4 legacy PPOTrainer accepts config, policy, reference and tokenizer; .step receives lists of query tensors, response tensors and scalar reward tensors. The release also has a PPOv2 implementation with different model/value/reward setup. Do not mix them.

This unexecuted language-model reference uses TRL 0.11.4, Transformers 4.45.2 and a compatible separately locked PyTorch/CUDA environment. It reads only approved local artifacts. The reward model must be a one-label sequence scorer trained/validated for its declared input template; the illustrative template below is part of that contract, not automatically correct for arbitrary checkpoints.

# Reference only: complete one-batch legacy PPOTrainer lifecycle.
import os
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
 
base = os.environ["LOCAL_SFT_MODEL"]
reward_path = os.environ["LOCAL_REWARD_MODEL"]
tok = AutoTokenizer.from_pretrained(base, local_files_only=True, trust_remote_code=False)
if tok.pad_token_id is None:
    tok.pad_token = tok.eos_token
rm_tok = AutoTokenizer.from_pretrained(reward_path, local_files_only=True, trust_remote_code=False)
policy = AutoModelForCausalLMWithValueHead.from_pretrained(base, local_files_only=True,
                                                         trust_remote_code=False)
ref = AutoModelForCausalLMWithValueHead.from_pretrained(base, local_files_only=True,
                                                      trust_remote_code=False)
ref.requires_grad_(False)
ref.eval()
rm = AutoModelForSequenceClassification.from_pretrained(reward_path, local_files_only=True,
                                                        trust_remote_code=False)
rm.requires_grad_(False)
rm.eval()
if rm.config.num_labels != 1:
    raise ValueError("reward model must expose the agreed scalar score")
config = PPOConfig(batch_size=2, mini_batch_size=1, ppo_epochs=2, learning_rate=1e-5,
                   init_kl_coef=.1, target=6., cliprange=.2, log_with=None)
trainer = PPOTrainer(config=config, model=policy, ref_model=ref, tokenizer=tok)
rm.to(trainer.current_device)
prompts = ["Give one useful study habit.", "Explain why testing a checkpoint matters."]
queries = [tok(p, return_tensors="pt").input_ids[0].to(trainer.current_device) for p in prompts]
responses = trainer.generate(queries, return_prompt=False, max_new_tokens=32,
                             do_sample=True, top_k=0, top_p=1., temperature=1.,
                             pad_token_id=tok.pad_token_id)
texts = [tok.decode(r, skip_special_tokens=True) for r in responses]
scores = []
with torch.no_grad():
    for prompt, response in zip(prompts, texts):
        reward_input = "Prompt: " + prompt + "\nResponse: " + response
        batch = rm_tok(reward_input, return_tensors="pt", truncation=False).to(trainer.current_device)
        scores.append(rm(**batch).logits.reshape(()).detach())
stats = trainer.step(queries, responses, scores)
trainer.log_stats(stats, {"query": prompts, "response": texts}, scores)
trainer.save_pretrained("ppo-one-batch-output")

These are smoke-test hyperparameters, not a trained assistant. Inspect exact prompt template, reward truncation limit, EOS and generated-response masks before use. A saved policy/value artifact is not automatically a complete resumable training checkpoint including optimizer, RNG and data state. Model residency must be budgeted before running an 8B or 70B version.

Reading logs and building a sanity harness

The legacy PPOConfig distinguishes target for adaptive reference-KL control from target_kl used for optional update early stopping. They are not synonyms. Inspect the selected release's log definitions: objective/kl aggregates sampled reference drift, score metrics report the reward proxy, and policy/update KL and clip fraction concern rollout reuse. Sequence-sum versus token-mean units change interpretations. The sign of policy loss does not directly say whether quality improved.

A monitoring harness runs at a recorded interval on held-out prompts, under fixed generation settings: save model revision, reward mean/distribution, independent task score, sampled/reference KL with units, update KL/clip fraction, entropy, response lengths, repeated n-grams, EOS rate and examples approved for inspection. Compare with baseline ranges and flag substantial changes for investigation. A length increase or reward increase is not itself proof of hacking; a joint proxy-quality divergence is stronger evidence. Alerts need context-specific thresholds and should trigger pause/inspection, not an automatic arbitrary beta multiplier.

# CPU-only diagnostic on already-generated permitted strings/scores.
from collections import Counter
 
def audit_outputs(texts, rewards, baseline_mean_length, max_length_ratio=2.):
    if not texts or len(texts) != len(rewards) or baseline_mean_length <= 0:
        raise ValueError("nonempty aligned records and positive baseline required")
    lengths = [len(t.split()) for t in texts]
    grams = Counter()
    for text in texts:
        words = text.split()
        grams.update(tuple(words[i:i+3]) for i in range(max(0, len(words)-2)))
    total = sum(grams.values())
    return {"mean_words": sum(lengths)/len(lengths),
            "mean_reward": sum(rewards)/len(rewards),
            "length_flag": sum(lengths)/len(lengths) > max_length_ratio*baseline_mean_length,
            "top_trigram_share": max(grams.values(), default=0)/max(1, total)}

If reward rises while independent examples collapse to punctuation or canned preambles, stop and preserve the failing checkpoint/metrics. Audit the reward data/template and scorer, evaluate an earlier checkpoint, collect appropriately labeled counterexamples if authorized, and retune LR/KL under controlled evaluation. A length penalty can remove a proxy artifact or introduce a new one. The old first-person incidents, costs and “every serious run fails by step 500” were not verified observations.

PPO, DPO, best-of-N, GRPO and RLVR

Best-of-N generates N responses and selects by a scorer: useful to audit scorer usefulness before training, but it costs inference and can amplify reward-model bias. Failure of best-of-N is a warning, not a theorem that no PPO variant can help. Offline DPO trains on preference pairs without online rollout generation, a value baseline or a separate learned reward network inside its basic update. It can still generalize beyond examples, and iterative DPO can refresh data without restarting training from scratch. PPO accepts flexible scalar/step rewards and on-policy exploration but pays rollout, value-estimation and tuning costs. Neither universally dominates.

GRPO: a different advantage estimator

DeepSeekMath v1 (2024) introduces Group Relative Policy Optimization. For multiple responses to the same prompt, an outcome-reward form uses (R_i−mean(R))/(std(R)+epsilon) as a group-relative advantage, avoiding a conventional learned critic. For rewards [0,0,1,1], population mean/std are .5/.5 and normalized advantages are [−1,−1,1,1]. If all rewards are equal, the outcome-relative signal is zero; normalization cannot invent information.

The group mean includes the sample being evaluated, so do not call the estimator an automatically unbiased independent baseline. Multiple samples, grouping, token normalization and KL estimation all change compute/statistics. Removing a critic saves its state but not necessarily half of total training memory, and a dense critic can still be useful for long, step-reward trajectories. There is no universal quality ordering between PPO and GRPO.

Original result adjudication (10 October 2026; corrected/rejected claim, not missing teaching). The old paragraph attributed G=64 to DeepSeek-R1 and claimed DeepSeekMath-7B gained +5.5 GSM8K / +6.4 MATH over PPO with the same reward model, plus faster wall-clock. DeepSeekMath v1, §4.2 and Table 5 instead specifies 64 training outputs per question for its own GRPO setup (maximum length 1024, batch size 1024, one policy update per exploration stage). Its chain-of-thought Top1 rows compare DeepSeekMath-Instruct 7B: 82.9% GSM8K, 46.8% MATH with DeepSeekMath-RL 7B: 88.2%, 51.7%. The arithmetic is +5.3 and +4.9 percentage points over Instruct, not +5.5/+6.4 over PPO. The tool-integrated rows are a separate protocol and must not be substituted into that comparison; Table 5 also distinguishes Top1 from gray majority-vote results.

Reject the original same-RM PPO gain and wall-clock assertions as unsupported by these governing results: Table 5 is not a controlled PPO-versus-GRPO timing experiment. Dropping critic training is a resource-saving mechanism, not a measured speedup under matched rollout budget/hardware. R1 v1, §2.2.1 and §3 evaluation setup describes a group of size G, but its 64-sample consensus evaluation is not evidence that its training group was G=64. The specific R1 training-size attribution is therefore also rejected as unsupported by that report; the documented G=64 training setting belongs to DeepSeekMath. No benchmark or wall-clock result here is our own run. The lesson is to bind a number to its model revision, comparator, sampling/evaluation protocol and metric before using it to choose an algorithm.

Verifiable rewards and R1-Zero

DeepSeek-R1 v1 (2025) reports R1-Zero's RL without an initial SFT stage, along with readability and language-mixing problems, and R1's multistage cold-start procedure. It does not prove “SFT only teaches format, RL only teaches capability.” Pretraining, demonstrations, sampling and reward all affect learned behavior.

RLVR replaces some learned preference scores with checkable task signals: final-answer agreement, unit-test results or formatting rules. Verifiers can have incomplete tests, parser ambiguity, leaked answers, sandbox vulnerabilities and specification mistakes. Run generated code only in an appropriately restricted sandbox; passing a few tests is not semantic correctness. A format reward is not a reasoning reward.

TÜLU 3 v1 explicitly reports SFT, DPO and RLVR stages plus held-out evaluation and decontamination. That is one disclosed recipe, not evidence that TÜLU, R1 and Llama 3 all share a single five-stage pipeline including a Constitutional AI pass. The old DPO “MATH ceiling,” proprietary-recipe claims and unqualified benchmark improvements are not retained as facts. The useful comparison is which feedback is available, how fresh responses are obtained, what value/advantage estimator is used and how regression is measured.

Optional historical reading alongside the TÜLU paper: Nathan Lambert’s Interconnects 2024 alignment retrospective. This restores the original resource pointer; its current contents and availability have not been freshly verified here, and it is not evidence for a shared proprietary training recipe.

Worked recall and transfer exercises

1. Write PPO-KL. The high-level objective is expected reward minus beta times KL to reference; the update uses the clipped rollout ratio surrogate. If KL is already in shaped rewards, do not automatically add another term to the loss.
2. Four roles? Trainable policy, frozen reference, reward scorer, trainable value function. Weight sharing/offload change implementation memory; the value head alone is small, its separate backbone may not be.
3. Does epsilon=.2 bound a token's change by 20%? No. It caps the sample's beneficial surrogate incentive outside part of the interval. Shared parameters and other updates can move it further.
4. What do reward and KL do in a healthy run? There is no required monotonic curve or universal numeric target. Read proxy scores, drift units, update diagnostics and independent behavior together.
5. Why sampled KL can be negative? One token may be more probable under the reference. The expected log ratio under the policy is nonnegative KL; a sample is not that expectation.
6. What does the GAE example return? Advantages [.74,.6], lambda-return targets [.94,1]. Terminal score belongs at the last real action, and truncation may need a nonzero bootstrap.
7. Diagnose rising reward, KL=45 and punctuation-only outputs. The actual behavioral collapse plus proxy increase is a failure; preserve evidence, audit the reward and reference/masking, evaluate/revert to a validated checkpoint and design a controlled correction. The number 45 alone is not a universal diagnosis.
8. When is online RL worth considering? A reliable evolving reward and a need for on-policy exploration can justify its complexity. Fixed paired data and ordinary SFT infrastructure favor starting with a direct preference baseline. Verifiable rewards remain vulnerable to verifier failures.

Spaced review: tomorrow distinguish old/current/reference without notes and compute the two-token recursion; on day seven inspect a log with declared aggregation units and identify the missing independent metric. DPO next derives an implicit reward parameterization, not proof that PPO was killed. Bring back the incentive-cap distinction, the four roles and an auditable evaluation harness.

Previous: ← DL S062 · Next: DL S064 →

Original optional resource links

The original optional video/reading destinations are retained here for continuity, not as current API, price, benchmark or publication-date evidence. The versioned primary citations in the teaching above govern the corrected claims. These supplementary destinations were not all fetched or watched; a missing or changed resource does not replace the local derivations, procedures and answers.

• https://www.youtube.com/watch?v=hhiLw5Q_UFg
• https://www.youtube.com/watch?v=VPRSBzXzavo
• https://arxiv.org/abs/1707.06347
• https://huggingface.co/blog/rlhf