Gradient Clipping and Accumulation

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 24 / 80 · Module M04 · ~3 hours

0 · Two different gradient-size problems

Clipping and accumulation solve different problems. Clipping changes a gradient before the optimizer uses it. Accumulation combines loss contributions into one optimizer update while keeping only one microbatch's activation graph live at a time. Neither makes NaNs valid or guarantees that an oversized model fits a small GPU.

A rare bad batch and a physically too-large batch are useful failure scenarios, not evidence that every frontier run clips at 1.0 or that a particular laptop can pretrain a model in a weekend. This lesson derives the conditions for equivalence, executes positive and negative CPU controls, and supplies complete but unexecuted CUDA/DDP references.

Prerequisites

Know loss/backward/update, AMP unscaling and scheduler clocks. The programs use synthetic tensors and PyTorch 2.9 CPU; no model or dataset download is needed.

1 · Gradient clipping — choose the constraint deliberately

1.1 Global norm clipping

Imagine concatenating every participating parameter gradient into one vector g. Ideal L2 clipping to c is

with zero handled directly. Multiplication by one positive scalar preserves the direction of a finite nonzero g. PyTorch clip_grad_norm_ mutates gradients in place and returns their pre-clip total norm, using a small denominator epsilon. Therefore the resulting norm need not be exactly c even when clipping fires. Parameters with no gradient do not contribute.

For vanilla SGD without momentum/decay, clipping gives an update norm at most eta*c. With momentum, Adam preconditioning, weight decay or other transformations, the final parameter update is not simply eta times clipped g. Measure the actual update if the operational goal is an update bound. A corrupted/outlier gradient can also point in the wrong direction; shortening it is not proof the direction is trustworthy.

1.2 Value clipping

clip_grad_value_ clamps each coordinate to [-v,v]. For g=(3,4), norm clipping to one gives approximately (.6,.8); value clipping to one gives (1,1), a different direction and total norm square-root two. Value clipping imposes a coordinate constraint; it is not mathematically meaningless or exclusively a backward-compatibility feature. Choose it only when that constraint is intended, rather than confusing it with an L2 bound.

import torch
from torch import nn
 
def parameter_with_grad(values):
    p=nn.Parameter(torch.zeros(len(values),dtype=torch.float64))
    p.grad=torch.tensor(values,dtype=torch.float64);return p
p=parameter_with_grad([3.,4.]);q=parameter_with_grad([3.,4.])
reported=torch.nn.utils.clip_grad_norm_([p],1.,error_if_nonfinite=True)
torch.nn.utils.clip_grad_value_([q],1.)
assert abs(reported.item()-5.) < 1e-12
assert torch.allclose(p.grad,torch.tensor([.6,.8],dtype=p.dtype),atol=1e-6)
assert torch.equal(q.grad,torch.ones(2,dtype=q.dtype))
assert abs(p.grad[0]/p.grad[1]-.75) < 1e-12
assert q.grad[0]/q.grad[1] == 1.
# Correct total norm squares component norms before summing.
a=parameter_with_grad([3.]);b=parameter_with_grad([4.])
manual=torch.sqrt(sum(v.grad.square().sum() for v in [a,b]))
assert manual.item()==5.
wrong=torch.sqrt(sum(v.grad.norm() for v in [a,b]))
assert wrong.item()!=5.
# Nonfinite input is an error, not an outlier repaired by clipping.
r=parameter_with_grad([float("nan"),1.])
try: torch.nn.utils.clip_grad_norm_([r],1.,error_if_nonfinite=True)
except RuntimeError: pass
else: raise AssertionError("nonfinite gradient accepted")
print("global/value geometry, total norm and nonfinite controls passed")

2 · Choosing max_norm and reading its log

A threshold like 1.0 is a starting recipe value, not an architecture law. Loss mean versus sum, supervised-token count, model parameterization, optimizer and precision change gradient scale. Compare a no-clip baseline where safe, log finite pre-clip quantiles and clipping frequency, and assess held-out quality/actual updates under candidate thresholds. Typical CNN/RNN/transformer/diffusion/PPO settings are not interchangeable guarantees.

A stable norm below threshold may be benign but does not prove learning. A growing norm suggests checking data, objective and state; a near-zero norm can mean convergence, masked/frozen parameters, underflow or vanishing gradients. If a plot is exactly at c every step, first determine whether it logged the post-clip norm. The returned pre-clip norm is not capped and can exceed c. It is therefore wrong to diagnose a too-tight threshold from a capped plot without checking what was measured.

Clipping every step is not automatically a bug, nor is it equivalent to one fixed LR reduction under an adaptive optimizer. Inspect the rescale factor and actual parameter changes. Do not silently skip a semantically invalid batch just to make the loss curve continue; preserve enough information to diagnose it and choose an explicit abort/recovery policy.

3 · AMP ordering: finish accumulation, then unscale

FP16 loss scaling multiplies every microbatch loss by the same scale for that effective update. Do not unscale or change the scale mid-window: later gradients would then be added in different units. Once all contributions are present: unscale once, inspect/clip finite gradients, attempt optimizer step, update scaler, zero gradients and advance an accepted-update scheduler only when appropriate.

If you clip Sg to c before dividing by S, the resulting norm is near c/S rather than c. For S=65536 this can almost erase the intended update while leaving a finite loss. BF16 usually operates without loss scaling; it is not immune to invalid forward math or gradient overflow. Save scaler state for continuation.

import torch
p=parameter_with_grad([3.,4.]);p.grad*=1024.
_ = torch.nn.utils.clip_grad_norm_([p],1.);p.grad/=1024.
assert p.grad.norm()<.001
q=parameter_with_grad([3.,4.]);q.grad*=1024.;q.grad/=1024.
_ = torch.nn.utils.clip_grad_norm_([q],1.)
assert q.grad.norm()>.99
# Clipping microgradients separately is nonlinear and not clip(sum).
u=torch.tensor([10.,0.]);v=torch.tensor([-9.,1.])
def ideal_clip(g,c=1.): return g*min(1.,c/g.norm().item())
assert not torch.allclose(ideal_clip(u)+ideal_clip(v),ideal_clip(u+v))
print("clip ordering and per-microclip negative controls passed")

4 · Accumulation — derive the intended objective first

For an additive loss with N supervised elements,

All terms are evaluated at the same parameters before one optimizer step. Equal-size, equally weighted microbatches have $n_j/N=1/k$, giving the familiar divide-by-k rule. Unequal sizes, padding masks, class weights or different reductions require the actual denominator. Mean MSE divides by the number of output elements, not always just examples. Class-weighted index cross entropy uses its weighted denominator, not necessarily a raw token count.

For sum reduction, contributions sum correctly for a sum objective. To recover a mean objective, divide the accumulated numerator by the total appropriate denominator. Omitting this division multiplies raw gradients; for plain SGD that resembles an LR change, but Adam, clipping, epsilon and decay prevent a universal k-times-update equivalence. Logging the divided micro-loss also changes its displayed scale; keep an unscaled numerator/denominator log for comparisons.

4.1 Unequal tails and masked tokens

Do not silently discard the final partial accumulation window. Either flush with its actual denominator or deliberately document a dropping policy. drop_last=True removes a short physical batch but does not necessarily make the number of microbatches divisible by k. An all-ignored group has no supervised mean; define whether to reject or skip it without an optimizer/scheduler update.

The exact CPU program below compares full-batch gradients with unequal masked microbatches, rejects the naive mean-of-means and verifies the resulting AdamW update.

import copy
import torch
from torch import nn
import torch.nn.functional as F
 
torch.manual_seed(24)
x=torch.randn(7,4,dtype=torch.float64)
y=torch.tensor([0,1,-100,2,0,-100,1])
base=nn.Linear(4,3).double();full=copy.deepcopy(base);micro=copy.deepcopy(base)
wrong=copy.deepcopy(base);count=int((y!=-100).sum())
F.cross_entropy(full(x),y,ignore_index=-100,reduction="sum").div(count).backward()
chunks=[slice(0,2),slice(2,5),slice(5,7)]
for s in chunks:
    (F.cross_entropy(micro(x[s]),y[s],ignore_index=-100,reduction="sum")/count).backward()
    (F.cross_entropy(wrong(x[s]),y[s],ignore_index=-100)/len(chunks)).backward()
for a,b in zip(full.parameters(),micro.parameters()):
    assert torch.allclose(a.grad,b.grad,atol=1e-12,rtol=1e-10)
assert not torch.allclose(full.weight.grad,wrong.weight.grad,atol=1e-6)
for model in [full,micro]:
    torch.nn.utils.clip_grad_norm_(model.parameters(),.5)
    torch.optim.AdamW(model.parameters(),lr=.01,weight_decay=.1).step()
assert torch.allclose(full.weight,micro.weight,atol=1e-12)
print("unequal masked gradient and clipped AdamW update equivalence passed")

4.2 Where full-batch equivalence fails

BatchNorm computes each forward's statistics on the physical microbatch, not the effective batch. SyncBatchNorm extends a simultaneous batch across ranks, not sequential microbatches. Contrastive in-batch negatives and other cross-sample losses also change when a batch is split. Dropout may consume randomness differently across shapes/kernels; match masks if testing exact equivalence, or compare distributions. Stochastic augmentation, clipping each microbatch and stepping optimizer/decay too often break the simple proof as well.

import torch
from torch import nn
 
torch.manual_seed(4);x=torch.randn(8,4)
lin=nn.Linear(4,1)
lin(x).square().mean().backward();g=lin.weight.grad.clone()
lin.zero_grad(set_to_none=True)
for mb in x.chunk(4): (lin(mb).square().mean()/4).backward()
assert torch.allclose(g,lin.weight.grad,atol=1e-6)
bn=nn.BatchNorm1d(4,track_running_stats=False)
assert not torch.allclose(bn(x)[:2],bn(x[:2]))
# In-batch contrastive denominator changes even if all positive scores agree.
scores=torch.tensor([[2.,1.,0.,-1.]])
assert not torch.allclose(torch.logsumexp(scores,1),torch.logsumexp(scores[:,:2],1))
print("additive equivalence and BN/contrastive counterexamples passed")

4.3 DDP no_sync and the global denominator

Ordinary DDP averages gradients across participating ranks. With unequal valid-token counts, averaging local mean losses gives each rank equal weight rather than each token. For global denominator N and world size R, backpropagate each rank's summed local numerator times R/N so that DDP's division by R yields the desired global mean gradient.

no_sync() must enclose forward and backward for all but the final microbatch. The final normal backward synchronizes accumulated gradients. All ranks need compatible collective order and matching window boundaries, including tails; an early return on just one rank can hang peers. Reducing eight synchronization rounds to one is an eightfold reduction in the number of such rounds (seven omitted), not a guaranteed sevenfold traffic/latency or throughput gain. Bucketization, overlap and other collectives remain.

5 · Complete accumulation loop with optional AMP

This executable function handles unequal physical batches, ignored targets and the final short window. It uses sum/count normalization, one unscale and one clip per accepted window, and a separate accepted-update clock. CPU tests run with AMP off. Calling the same function on CUDA with dtype FP16/BF16 is an unexecuted GPU reference, not a claim of CUDA parity.

import copy
import itertools
import torch
from torch import nn
import torch.nn.functional as F
 
def windows(iterator,k):
    if k<1: raise ValueError("positive accumulation count required")
    iterator=iter(iterator)
    while True:
        group=list(itertools.islice(iterator,k))
        if not group: break
        yield group
 
def train_accum(model,batches,opt,sched,k=4,amp_dtype=None):
    device=next(model.parameters()).device
    if amp_dtype is not None and device.type!="cuda":
        raise ValueError("this AMP reference expects CUDA")
    scaler=torch.amp.GradScaler(device.type,enabled=(amp_dtype==torch.float16))
    model.train();log=[]
    for group in windows(batches,k):
        total=sum(int((y!=-100).sum()) for _,y in group)
        if total==0: raise ValueError("effective batch contains no supervised targets")
        opt.zero_grad(set_to_none=True);numerator=0.
        for x,y in group:
            x=x.to(device);y=y.to(device)
            with torch.autocast(device.type,dtype=amp_dtype or torch.bfloat16,
                                enabled=(amp_dtype is not None)):
                summed=F.cross_entropy(model(x),y,ignore_index=-100,reduction="sum")
                loss=summed/total
            if not torch.isfinite(loss): raise FloatingPointError("forward loss")
            numerator+=summed.detach().float().item()
            scaler.scale(loss).backward()
        scaler.unscale_(opt)
        finite=all(p.grad is None or torch.isfinite(p.grad).all() for p in model.parameters())
        if finite:
            preclip=torch.nn.utils.clip_grad_norm_(model.parameters(),1.,error_if_nonfinite=True).item()
        elif scaler.is_enabled(): preclip=float("inf")
        else: raise FloatingPointError("nonfinite gradient")
        used=opt.param_groups[0]["lr"];old_scale=scaler.get_scale()
        scaler.step(opt);scaler.update()
        accepted=scaler.get_scale()>=old_scale  # one optimizer, normal scale policy
        if accepted:sched.step()
        log.append(dict(microbatches=len(group),targets=total,mean_loss=numerator/total,
                        preclip=preclip,used_lr=used,accepted=accepted))
    opt.zero_grad(set_to_none=True)
    return log
 
torch.manual_seed(9);base=nn.Linear(4,3).double()
batches=[(torch.randn(n,4,dtype=torch.float64),torch.arange(n)%3) for n in [3,2,4,1,2]]
a,b=copy.deepcopy(base),copy.deepcopy(base)
oa=torch.optim.SGD(a.parameters(),lr=.1);ob=torch.optim.SGD(b.parameters(),lr=.1)
sa=torch.optim.lr_scheduler.StepLR(oa,1,.9);sb=torch.optim.lr_scheduler.StepLR(ob,1,.9)
log=train_accum(a,batches,oa,sa,k=3)
full_groups=[(torch.cat([x for x,y in group]),torch.cat([y for x,y in group]))
             for group in windows(batches,3)]
ref=train_accum(b,full_groups,ob,sb,k=1)
assert [v["microbatches"] for v in log]==[3,2]
assert [v["targets"] for v in log]==[9,3]
assert sa.last_epoch==sb.last_epoch==2
assert all(torch.allclose(x,y,atol=1e-12) for x,y in zip(a.parameters(),b.parameters()))
try: train_accum(a,[(torch.ones(2,4,dtype=torch.float64),torch.full((2,),-100))],oa,sa)
except ValueError:pass
else:raise AssertionError("empty supervised group accepted")
print("complete accumulation loop, tail flush and all-ignored control",log)

The whole group's CPU targets are counted before backward, so normalization is consistent for every contribution. Holding a list of CPU microbatches increases host storage, not simultaneous activation graphs; stream/loader implementations can use a known denominator or a careful two-stage strategy instead. A generic trainer must additionally save model/optimizer/scheduler/scaler/RNG/cursor state at a complete window boundary. The function starts fresh scaler state and is not advertised as a standalone resume API.

5.1 Complete DDP reference — unexecuted

This reference expects an already initialized, authorized process group and a compatible DDP-wrapped model on each rank. It neither launches processes nor creates resources. The complete function handles unequal valid-token counts with a collective denominator and an all-ignored group consistently. All ranks must supply the same number of nonempty windows; equal iteration structure is a precondition, not silently solved by no_sync.

# Distributed reference — unexecuted
import contextlib
import itertools
import torch
import torch.distributed as dist
import torch.nn.functional as F
 
def ddp_accum_epoch(ddp_model,loader,optimizer,scheduler,accum=4):
    if not dist.is_initialized(): raise RuntimeError("initialized process group required")
    device=next(ddp_model.parameters()).device
    iterator=iter(loader);updates=0;ddp_model.train()
    if accum<1:raise ValueError("positive accumulation count required")
    while True:
        group=list(itertools.islice(iterator,accum))
        if not group:break  # precondition: all ranks end together
        count=torch.tensor(sum(int((y!=-100).sum()) for _,y in group),device=device,dtype=torch.float64)
        dist.all_reduce(count,op=dist.ReduceOp.SUM)
        if count.item()==0:raise ValueError("no supervised targets on any rank")
        optimizer.zero_grad(set_to_none=True)
        for j,(x,y) in enumerate(group):
            final=j==len(group)-1
            context=contextlib.nullcontext() if final else ddp_model.no_sync()
            with context:
                logits=ddp_model(x.to(device))
                summed=F.cross_entropy(logits,y.to(device),ignore_index=-100,reduction="sum")
                (summed*(dist.get_world_size()/count)).backward()
        torch.nn.utils.clip_grad_norm_(ddp_model.parameters(),1.,error_if_nonfinite=True)
        optimizer.step();scheduler.step();updates+=1
    optimizer.zero_grad(set_to_none=True)
    return updates

This FP32 reference intentionally separates DDP's global-mean contract from AMP's scale contract. Combining them preserves both: same scale across the window, compatible rank decisions, one unscale after synchronized accumulation, then finite handling/clip/step. Sharded systems may require their own global norm helper rather than clipping a local shard and calling it global.

6 · Effective batch, learning rate and decay

With equal unmasked microbatches, effective_batch = micro_batch * accumulation_count * world_size. For variable-length text, use the sum of supervised tokens, not allocated padding or a nominal product. In a tail or unequal-rank window the product is not the actual denominator.

Holding effective batch, data order, objective, optimizer timing and RNG behavior fixed can reproduce an additive gradient; it does not make all hyperparameters transferable across physical batch changes. Goyal et al.'s linear LR scaling is a bounded large-batch SGD result with warmup, not a universal Adam doubling law or fixed token-count sweet spot. Larger batches change update count for a fixed token budget and interact with schedule and noise.

Isolating the decoupled weight-decay contribution on an updated parameter, successive updates multiply weights by

when terms are small. This is the decay-only factor, not the full AdamW trajectory when momentum or a loss gradient also contributes. In PyTorch, an explicit zero gradient tensor can still trigger decay and moment updates; grad=None instead skips that parameter, including its decay. A frozen or unused parameter therefore does not automatically follow this product.

Halving updates at the same LR and halving lambda reduces shrinkage roughly fourfold; it does not preserve it. To preserve the small-step exponent when total LR sum halves, lambda would need to roughly double, although real optimization/regularization quality still needs validation. Inspect the product rather than repeat a 'halve training, halve decay' rule.

import math
rate,decay,steps=.01,.1,100
original=(1-rate*decay)**steps
wrong=(1-rate*(decay/2))**(steps//2)
approx_match=(1-rate*(decay*2))**(steps//2)
assert abs(original-approx_match)<.0001
assert abs(original-wrong)>.05
# Same sum of LR, close cumulative decay under small-step approximation.
assert abs(math.log(original)+rate*decay*steps)<.0001
print("decay product",original,"wrong halving",wrong,"approx match",approx_match)

7 · Activation checkpointing — save activations, not restart state

Activation checkpointing recomputes selected forward intermediates during backward to reduce retained activation storage. It is unrelated to the disk checkpoint used to resume a run. Use explicit use_reentrant=False under this PyTorch contract. RNG preservation can make dropout recomputation match, but moving tensors to an unseen device or changing global state during recomputation can violate assumptions. Stateful side effects such as BatchNorm running-stat updates need particular care.

A simple chain model illustrates the tradeoff: keep segment boundaries and recompute interiors. With L equal-cost layers segmented every k layers, a rough peak activation count is $L/k+k$, minimized near $k=\sqrt L$. Real graph topology, saved tensors, kernel workspaces and nonreentrant early-stop behavior change the cost. A fixed 30% time penalty or 40–70% total-memory saving is not a guarantee.

Accumulation reduces simultaneous activation-batch storage but not weights/optimizer-state storage; AMP may reduce selected activation/compute tensors without halving FP32 parameters or moments; checkpointing changes retained intermediates. For a 400M-parameter FP32 Adam-like setup, parameters+two moments+gradients alone are about 6.4 GB decimal, before activations, buffers, temporary workspace and allocator reservation. A 24GB card's feasibility still depends on sequence length, architecture, kernels and fragmentation. No generic four-times-larger-model guarantee follows.

import copy
import torch
from torch import nn
from torch.utils.checkpoint import checkpoint
 
torch.manual_seed(10)
a=nn.Sequential(nn.Linear(4,8),nn.ReLU(),nn.Dropout(.3),nn.Linear(8,2)).double()
b=copy.deepcopy(a)
x=torch.randn(5,4,dtype=torch.float64,requires_grad=True)
z=x.detach().clone().requires_grad_()
torch.manual_seed(11);u=a(x);u.square().sum().backward()
torch.manual_seed(11);v=checkpoint(b,z,use_reentrant=False,preserve_rng_state=True)
v.square().sum().backward()
assert torch.equal(u,v)
assert torch.allclose(x.grad,z.grad,atol=1e-12)
assert all(torch.allclose(p.grad,q.grad,atol=1e-12) for p,q in zip(a.parameters(),b.parameters()))
# Disable RNG preservation: forward still matches, recomputed mask can change gradients.
c=copy.deepcopy(a);c.zero_grad();w=x.detach().clone().requires_grad_()
torch.manual_seed(11);bad=checkpoint(c,w,use_reentrant=False,preserve_rng_state=False)
bad.square().sum().backward()
assert torch.equal(u,bad)
assert not torch.allclose(x.grad,w.grad)
print("CPU checkpoint output/gradient parity and RNG negative control passed")

For a GPU checkpointing benchmark, compare the same 10-layer model and seeds with/without checkpointing, warm kernels, reset peak-memory stats before measured steps, synchronize timing and separately report allocated/reserved peaks plus losses/gradient tolerances. Do not infer a GPU memory saving from the executed CPU parity test.

8 · Failure scenarios and tradeoffs

• Loss log changed by k: determine whether the logger recorded normalized micro-loss or the true effective mean. Gradient scale and printed loss are different issues; compare an exact full-batch oracle.
• One reproducible NaN: clipping cannot repair a nonfinite derivative. Locate the first invalid tensor and preserve a reproduction; threshold 1.0 is not insurance against every data bug.
• Wrong AMP order: the numerical control shows a factor-of-1024 update loss from clipping before unscale.
• Repeated DDP communication: wrap both forward and backward in no_sync before the last contribution. Profile actual collectives; do not promise a fixed speedup.
• Dropped tail: flush a two-microbatch final group with its own total count, as the full loop tests. Dropping incomplete data is a deliberate sampling change, not the only correct fix.
• Memory still exceeds capacity: inspect whether weights/state or activations dominate. Accumulation cannot shrink the persistent model; checkpointing cannot remove all optimizer state. A genuinely smaller model/batch or different state strategy may be required.

Choose accumulation when matching an additive effective-batch objective matters; checkpointing when retained activations dominate; a smaller effective batch when its optimization tradeoff is acceptable. There is no mandatory 'checkpoint first' ordering: compare measured recomputation cost, attainable microbatch throughput and objective constraints.

10 · Try it yourself — equivalence before performance

The original batch128 versus four batch32 comparison should first disable batch-coupled/statistical randomness, hold data order fixed and compare every gradient and next parameter update. Add clipping after accumulation to both. The masked unequal-tail CPU oracle above is a stronger accounting test than comparing smooth loss plots. Then reintroduce dropout, BN or contrastive terms one at a time to expose their boundaries.

A third BF16 GPU run may change arithmetic and need not be exactly equal or faster. Compare quality/tolerance, peak memory and synchronized time without prefilled results. The optional ten-layer checkpointing experiment likewise reports measurements, not an expected 40–60% saving. No dataset/GPU acquisition is part of this review.

11 · Recap and worked recall answers

Q1 — Why one norm factor? Positive uniform scaling preserves a finite gradient vector's direction. Value clipping imposes coordinate bounds and can rotate it. Neither guarantees the final adaptive/momentum update has the same direction or a universal safe magnitude.

Q2 — Why unscale first? A threshold applies to the intended gradient units. Clipping Sg to c then dividing gives near c/S. Accumulate at one scale, unscale once, then clip and step.

Q3 — Why divide mean losses? For equal microbatch denominators, the global mean gradient is the mean of microgradients. For unequal denominators, weight each by its valid count over the total or use summed numerators divided by the total. Sum reduction still needs division when the target objective is a mean.

Q4 — What is effective batch? The product of microbatch, accumulation count and ranks only for equal unmasked groups. In general count the global supervised examples/tokens and respect DDP averaging. Match optimizer clocks as well as counts before claiming transfer.

Q5 — Norm exactly at threshold? First distinguish pre- versus post-clip measurement. The API return is pre-clip and is not capped. If genuine gradients require constant clipping, examine scale/objective and actual updates; raising c or lowering LR is a tested hypothesis, not an automatic diagnosis.

Q6 — A 400M model on 24GB? Start with an explicitly hypothetical config: microbatch one at a specified modest sequence length, accumulation to a measured token target, supported BF16 if quality checks pass, selected activation checkpointing, an initial clip candidate 1 and a small LR/warmup sweep. Estimate the 6.4GB FP32 parameter/gradient/moment subtotal plus measured activation/workspace peaks, reserve headroom, and count accepted updates/tokens. Adjust only after a tiny finite-gradient fit and throughput/memory measurements. This is a sizing and validation procedure, not a promise that a named architecture fits or a request to provision hardware.

Sources and execution boundary

PyTorch clipping source v2.9.0, AMP examples 2.9, DDP no_sync source, activation checkpointing, Chen et al. 2016, large-batch SGD, and AdamW support the bounded mechanisms. Exact CPU programs/math/negative controls are executed. CUDA/DDP execution, model-size feasibility and GPU performance remain explicitly unexecuted.

Optional pre-read resources

These inherited resource links are optional, not the primary evidence for this review. Video identity, duration and current availability are not certified.

• Unverified inherited video link — clipping topic
• Unverified inherited video link — accumulation topic
• PyTorch2.9 clip_grad_norm_ reference
• Gradient accumulation with 🤗 Accelerate

Previous: ← DL S023 · Schedulers — Warmup, Cosine, and One-Cycle · Next: DL S025 · Convolution →