QLoRA: NF4 storage, gradients and local integration

QLoRA separates storage, computation and optimization

Goal: derive the NF4/block-scale memory bill, backpropagate through a frozen quantized operator into LoRA, understand paging/precision limits, and specify a complete local-only integration. LoRA reduces trainable state; QLoRA additionally compresses the frozen base. A7B BF16 base alone is14GB decimal, so20GB is not inherently too large for a24GB card—but remaining activations, temporary buffers, optimizer state and runtime overhead decide whether a run fits.

Dettmers et al. report a65B finetune on a single48GB GPU in their specified setup. Their Guanaco/Vicuna evaluation is not universal equality with ChatGPT or a promise that any7B/4K-context/batch configuration fits12/24GB. The paper spans model sizes down to80M; the original claim that NF4 was tuned only for3B+ models and necessarily fails below1B is unsupported.

The useful analogy is a compressed reference library plus a trainable notepad: the library stays frozen, but gradients still propagate through its computation to earlier adapters. Frozen parameters do not stop input gradients.

Memory accounting before optimization

Separate unique logical parameters from packed storage elements. For N quantized weights, A adapter parameters and B bytes of nonquantized base state:

Add adapter weights, adapter gradients, optimizer moments/master copies if present, activations, temporary dequantization/kernel buffers, allocator reservations and framework overhead. Base gradients/Adam moments are absent only if the base is actually frozen and excluded from the optimizer. Activations remain; ordinary attention can grow quadratically in sequence length unless a supported memory-efficient implementation changes storage.

A simple FP32 full-Adam model uses4N bytes weights,4N gradients and8N moments:16N bytes, or112GB for7B, before activations. A mixed-precision implementation may hold BF16 parameters/gradients plus FP32 masters/moments, or FP32 parameters under autocast; enumerate the actual implementation rather than combine mutually inconsistent14/28/56GB rows. LoRA does not make full-parameter SFT “LoRA” and does not remove the frozen-base forward/backward-input work.

For an illustrative32-layer decoder with d=4096, FFN14336,32Q/8KV heads of128 and rank16 adapters on q/k/v/o/gate/up/down, adapter count is 32*16*(8192+5120+5120+8192+3*18432)=41943040. Those adapters alone use about83.89MB in BF16; FP32 moments add335.54MB if both moments are stored in FP32. Different targets/ranks/dtypes change it. A PEFT printed parameter total needs its counting implementation inspected; packed tensor numel is not the logical model size.

NF4: nonuniform reconstruction levels

Four bits select16 values. NF4's codebook is motivated by normal-weight quantiles, includes zero, and is asymmetric: seven negative, zero and eight positive values. It is not “15 equally spaced probabilities from1/32 to31/32 plus zero” or a symmetric INT4 grid. The tagged bitsandbytes0.48.1 source gives the exact table used below.

For each64-weight block, obtain s=max(abs(w)), normalize by s, choose nearest codebook values and store4-bit indices plus a scale. The zero block needs an explicit convention to avoid division by zero. Absolute-max normalization and block size change the effective distribution; raw weights being bell-shaped does not prove NF4 is the exact minimum-MSE quantizer. Equal probability bins do not generally minimize squared distortion—the Lloyd-Max centroid/boundary conditions are different.

NF4 allocates more resolution near zero than a uniform grid, but the original claim of10levels in[-0.2,0.2] is false: this table has five. Neither NF4 nor uniform INT4 universally wins every tensor/task. Compare reconstruction error and downstream operator/output effects; a finite normal-versus-uniform experiment need not flip rankings on every seed or block size.

Double quantization: correct the scale arithmetic

The QLoRA paper's ordinary block scales are32-bit. With64weights/block, scale overhead is 32/64=0.5bits/weight, making4.5bits/weight before other metadata. Double quantization uses8-bit quantized scales with a32-bit second-level scale per256first-level scales:

Ignoring small global offsets/tails, savings are0.373046875bits/weight, consistent with the paper's approximate0.37. If every one of7B weights were quantized, this saves326416015.625bytes≈0.326GB;70B saves≈3.264GB. Actual models leave some weights unquantized and have alignment/metadata overhead. The original lesson mixed FP16 scales,4.5/5.0bpp and100MB/0.5GB claims; those described incompatible accounting schemes.

For one block,4-bit payload is32bytes and an FP32 scale adds4bytes, versus128bytes BF16. Double-quantization amortization is a many-block calculation, not3.5GB exactly for every7B base. The source implementation also stores a scale offset/second quantization state; retain them with the packed data.

Exact CPU teaching quantizer and adapter gradient path

This is a NumPy/PyTorch teaching implementation using the exact NF4 values and real nibble packing. It is not bitsandbytes, a GPU kernel, double-quantized production storage or a quality benchmark. Finite input alone is not enough: a block scale such as 1e40 overflows FP32, while 1e-50 rounds to zero. The program rejects both instead of silently reconstructing infinities or erasing a nonzero block. Representable scales still round; the test is a range guard, not lossless quantization. The low-nibble-first packing here is a declared teaching format, not a bitsandbytes binary interchange contract. The base reconstructed tensor is deliberately cached in full precision for this small CPU gradient demonstration; thus its allocated memory is not a QLoRA memory measurement.

import math
import numpy as np
import torch
from torch import nn
 
NF4=np.array([-1.0,-0.6961928009986877,-0.5250730514526367,-0.39491748809814453,
 -0.28444138169288635,-0.18477343022823334,-0.09105003625154495,0.0,
 0.07958029955625534,0.16093020141124725,0.24611230194568634,0.33791524171829224,
 0.44070982933044434,0.5626170039176941,0.7229568362236023,1.0],dtype=np.float64)
 
def quantize_nf4(w,block=64):
    a=np.asarray(w,dtype=np.float64)
    if block<1 or not a.size or not np.isfinite(a).all(): raise ValueError('quantizer input')
    flat=a.ravel(); padded=np.pad(flat,(0,(-len(flat))%block)).reshape(-1,block)
    scale=np.max(np.abs(padded),axis=1)
    with np.errstate(over='ignore',under='ignore'):
        stored_scale=scale.astype(np.float32)
    if not np.isfinite(stored_scale).all() or np.any((scale>0)&(stored_scale==0)):
        raise ValueError('nonzero block scale must remain finite and nonzero in float32')
    normalized=padded/np.where(scale==0,1,scale)[:,None]
    codes=np.abs(normalized[:,:,None]-NF4[None,None,:]).argmin(-1).astype(np.uint8).ravel()
    n=len(codes)
    if n%2: codes=np.append(codes,np.uint8(7))
    packed=codes[0::2]|(codes[1::2]<<4)
    return {'packed':packed,'scale':stored_scale,'shape':a.shape,
            'length':a.size,'padded':n,'block':block}
 
def dequantize_nf4(q):
    p=q['packed']; codes=np.empty(2*len(p),dtype=np.uint8)
    codes[0::2]=p&15; codes[1::2]=p>>4
    out=NF4[codes[:q['padded']]].reshape(-1,q['block'])*q['scale'][:,None]
    return out.ravel()[:q['length']].reshape(q['shape'])
 
class FrozenNF4LoRA(nn.Module):
    def __init__(self,w,rank=2,alpha=4.):
        super().__init__()
        if rank<1: raise ValueError('rank')
        q=quantize_nf4(w)
        self.register_buffer('base',torch.tensor(dequantize_nf4(q),dtype=torch.float64))
        out_dim,in_dim=w.shape
        self.a=nn.Parameter(torch.randn(rank,in_dim,dtype=torch.float64)*.05)
        self.b=nn.Parameter(torch.zeros(out_dim,rank,dtype=torch.float64))
        self.scale=alpha/rank
    def forward(self,x):
        return x@self.base.T+self.scale*(x@self.a.T)@self.b.T
 
def bits_per_weight(double_quant):
    return 4+8/64+32/(64*256) if double_quant else 4+32/64
 
if __name__=='__main__':
    torch.set_num_threads(1); torch.manual_seed(2)
    rng=np.random.default_rng(2); w=rng.normal(0,.2,(8,8))
    q=quantize_nf4(w); reconstructed=dequantize_nf4(q)
    assert len(q['packed'])==32 and q['scale'].nbytes==4
    assert np.count_nonzero(dequantize_nf4(quantize_nf4(np.zeros((3,7)))))==0
    # Exactly representable codebook entries in a unit-scale block round-trip.
    np.testing.assert_array_equal(dequantize_nf4(quantize_nf4(NF4,16)),NF4)
    assert not np.allclose(NF4,-NF4[::-1]) and ((NF4>=-.2)&(NF4<=.2)).sum()==5
    m=FrozenNF4LoRA(w); x=torch.randn(4,8,dtype=torch.float64,requires_grad=True)
    target=torch.randn(4,8,dtype=torch.float64); old=m.base.clone()
    opt=torch.optim.AdamW(m.parameters(),lr=.03,weight_decay=0)
    loss=(m(x)-target).square().mean(); loss.backward()
    assert m.base.grad is None and x.grad.abs().sum()>0
    assert torch.count_nonzero(m.a.grad)==0 and m.b.grad.abs().sum()>0
    opt.step(); opt.zero_grad(); (m(x)-target).square().mean().backward()
    assert m.a.grad.abs().sum()>0 and torch.equal(old,m.base)
    effective=m.base+m.scale*m.b@m.a
    torch.testing.assert_close(m(x),x@effective.T)
    original_merge=torch.tensor(w,dtype=torch.float64)+m.scale*m.b@m.a
    assert not torch.allclose(effective,original_merge)
    requant=torch.tensor(dequantize_nf4(quantize_nf4(effective.detach().numpy())),dtype=torch.float64)
    assert not torch.allclose(x@effective.T,x@requant.T)
    assert bits_per_weight(False)==4.5 and bits_per_weight(True)==4.126953125
    print('NF4 nibble/zero/gradient/merge controls passed; MSE',float(np.mean((w-reconstructed)**2)),
          '7B double-quant saved GB',(bits_per_weight(False)-bits_per_weight(True))*7e9/8/1e9)

For row-batch input X, output Y=X Qhat(W)ᵀ + s X Aᵀ Bᵀ. If upstream gradient is G, then dB=s Gᵀ X Aᵀ, dA=s Bᵀ Gᵀ X, and dX=G Qhat(W)+s G B A. There is no update to frozen W, but dX is generally nonzero. With B initially zero, dA is initially zero while dB may be nonzero; after B updates, A can learn. The local tests and independent finite differences check this, rather than confuse frozen weights with a broken graph.

Paged optimizers and compute dtype

Paged optimizers use managed-memory paging to handle optimizer-state pressure/spikes in the supported CUDA/backend environment. They do not make memory unlimited, guarantee no OOM or remove activation storage. Paging/staging can add latency and requires host memory/bandwidth; neither5–15% overhead nor universal safety is promised. An adapter-only optimizer can be small enough that paging is not the dominant concern.

NF4 is a storage codebook, not the dtype of all forward/backward arithmetic. Bitsandbytes uses supported dequantization/compute paths; do not infer that every GPU has no4-bit arithmetic or that every kernel materializes a whole dequantized model at once. bnb_4bit_compute_dtype selects a computation contract for those modules, not all norms, adapters, optimizer state or activations.

BF16 is often suitable on supported hardware because of exponent range. FP16 may need gradient scaling; FP32 may be useful for debugging. Check actual backend/device support instead of calling BF16 mandatory on every Ampere device or claiming FP32 halves throughput. Underflow rounds small values to zero; NaNs commonly arise from overflow/invalid operations, not directly from underflow. Find the first invalid value. A wider dtype cannot repair a wrong mask, invalid labels or corrupted state.

What PEFT preparation actually does

The v0.17.1 prepare_model_for_kbit_training implementation first freezes base parameter requires_grad. For applicable quantization types it upcasts eligible non-Params4bit FP16/BF16 parameters to FP32, not merely LayerNorm. Under its gradient-checkpointing conditions it enables input activation gradients via a hook or helper where needed, and enables checkpointing. Non-reentrant checkpointing avoids that particular input-gradient workaround.

This does not automatically unfreeze the embedding weight. The original explanation that gradients cannot pass through frozen4-bit operations was false. Inspect named_parameters after attaching adapters: only intended adapters and any explicitly chosen saved/trainable modules should enter the optimizer. Token additions require a separate embedding-training policy.

Complete local-only QLoRA integration reference

Source contract: bitsandbytes0.48.1, PEFT0.17.1, TRL0.23.1 and Transformers4.57.1. This unexecuted function assumes compatible packages/backend, a supported already-available CUDA GPU, existing local base/tokenizer and local prompt-completion JSONL. The CPU audit does not install them, obtain weights/data, allocate a GPU or claim this stack was runtime-tested together.

# Resource reference: local-only QLoRA; not executed in this audit.
def qlora_local(model_dir,train_jsonl,valid_jsonl,out_dir):
    import os,json
    from pathlib import Path
    os.environ['HF_HUB_OFFLINE']='1'; os.environ['HF_DATASETS_OFFLINE']='1'
    os.environ['TRANSFORMERS_OFFLINE']='1'
    import torch
    from datasets import Dataset
    from transformers import AutoModelForCausalLM,AutoTokenizer,BitsAndBytesConfig
    from peft import LoraConfig,get_peft_model,prepare_model_for_kbit_training,PeftModel
    from trl import SFTConfig,SFTTrainer
    if not torch.cuda.is_available(): raise RuntimeError('supported existing CUDA device required')
    if not torch.cuda.is_bf16_supported(): raise RuntimeError('this reference requires BF16 support')
    root=Path(model_dir)
    if not root.is_dir(): raise ValueError('local base required')
    tok=AutoTokenizer.from_pretrained(str(root),local_files_only=True,trust_remote_code=False)
    if not tok.chat_template or tok.eos_token_id is None: raise ValueError('template/stop contract')
    if tok.pad_token_id is None: tok.pad_token=tok.eos_token
    cfg4=BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type='nf4',
        bnb_4bit_use_double_quant=True,bnb_4bit_compute_dtype=torch.bfloat16)
    model=AutoModelForCausalLM.from_pretrained(str(root),local_files_only=True,
        trust_remote_code=False,quantization_config=cfg4,torch_dtype=torch.bfloat16,
        device_map={'':0})
    model.config.use_cache=False
    model=prepare_model_for_kbit_training(model,gradient_checkpointing_kwargs={'use_reentrant':False})
    adapters=LoraConfig(r=16,lora_alpha=32,target_modules=['q_proj','k_proj','v_proj','o_proj',
        'gate_proj','up_proj','down_proj'],lora_dropout=.05,bias='none',task_type='CAUSAL_LM')
    model=get_peft_model(model,adapters)
    trainable=[n for n,p in model.named_parameters() if p.requires_grad]
    if not trainable or any('lora_' not in n for n in trainable): raise ValueError('trainable inventory')
    def dataset(path):
        rows=[json.loads(s) for s in Path(path).read_text().splitlines() if s.strip()]
        if not rows: raise ValueError('empty dataset')
        for r in rows:
            if not r['prompt'] or not r['completion']: raise ValueError('empty pair')
            if any(m['role']!='assistant' for m in r['completion']): raise ValueError('completion roles')
            ids=tok.apply_chat_template(r['prompt']+r['completion'],tokenize=True,
                                        add_generation_prompt=False)
            if len(ids)>4096: raise ValueError('overlength; curate before training')
        return Dataset.from_list(rows)
    args=SFTConfig(output_dir=str(out_dir),num_train_epochs=1,learning_rate=2e-4,
        per_device_train_batch_size=1,per_device_eval_batch_size=1,gradient_accumulation_steps=1,
        max_length=4096,packing=False,completion_only_loss=True,
        bf16=True,fp16=False,optim='paged_adamw_32bit',report_to='none',
        gradient_checkpointing=True,gradient_checkpointing_kwargs={'use_reentrant':False},
        eval_strategy='epoch',save_strategy='epoch',load_best_model_at_end=True,
        metric_for_best_model='eval_loss',greater_is_better=False,seed=42)
    trainer=SFTTrainer(model=model,args=args,train_dataset=dataset(train_jsonl),
        eval_dataset=dataset(valid_jsonl),processing_class=tok)
    probe=next(iter(trainer.get_train_dataloader()))
    if not (probe['labels'][:,1:]!=-100).any(): raise ValueError('no completion targets')
    before=trainer.evaluate(); trainer.train(); after=trainer.evaluate()
    trainer.save_model(str(out_dir)); tok.save_pretrained(str(out_dir))
    # Adapter remains separate; reload in a separate process using the same base revision/cfg4.
    return {'before':before,'after':after,'trainable_names':trainable,
            'reload_class':PeftModel.__name__,'merged':False}

The target-module list assumes a matching Llama/Mistral-style implementation; fail on incompatible architecture instead of attaching to guessed names. device_map='auto' is not a universal distributed-training placement strategy. This reference is single-device and intentionally starts at batch1/accum1 with packing disabled;4096tokens can still OOM. Increase batch/accumulation only after measuring peak memory and verifying token-denominator handling for unequal masked microbatches. Inspect all prepared target spans/EOS and train/validation provenance as in SFT, not merely the first probe.

Save adapter plus exact base revision/hash, quantization settings, tokenizer/template, package versions and evaluation contract. Adapter-only save_model is not a full base-model restart checkpoint; trainer restart state and base availability remain necessary. Reload with PeftModel.from_pretrained on the exact locally loaded quantized base and test the same fixed probes before considering a merge.

Matched CUDA measurement reference: BF16-LoRA versus NF4-QLoRA

UNEXECUTED GPU REFERENCE — not a benchmark result. The CPU quantizer above cannot measure GPU savings because it caches a full-precision base. This reference instead compares two fresh, local single-device runs with the same base revision, tokenizer, exact input IDs/labels, adapter initialization, optimizer, attention implementation, checkpointing, warmup and measured token budget. Run one mode per fresh process on the same otherwise idle GPU; do not keep both models resident or reuse one model's optimizer for the other. No model loading, GPU allocation or benchmark execution is part of this lesson's verification.

The baseline is BF16 frozen linear weights plus FP32 nonquantized state and FP32 adapters, with BF16 autocast. The NF4 arm changes those same eligible frozen linear weights to NF4 with double quantization and BF16 compute; its remaining parameters follow the same FP32 policy. This explicit baseline avoids attributing PEFT preparation's norm/embedding upcasts to quantization. PyTorch 2.8.0, Transformers 4.57.1, PEFT 0.17.1 and bitsandbytes 0.48.1 are the source contracts, not a claim that this combination was integration-tested. A Llama/Mistral-style model exposing the seven named linear targets and an already available compatible BF16 CUDA device are required.

First isolate the measurement state machine. step(batch) performs exactly one update; tokens(batch) returns the positive number of shifted, unmasked loss targets. Warmup updates are intentional, matched between arms and excluded from timing and token totals. Adam state is thus initialized before the steady-state baseline. Synchronize before resetting peaks and starting the clock, then synchronize after the last measured update before stopping it. Allocated and reserved peaks are process-local allocator metrics, not whole-device VRAM usage; the delta above the post-warmup allocation is not the total footprint.

import math
 
def measure_updates(step, batches, tokens, meter, warmup=2):
    # meter has synchronize/reset/allocated/reserved/peak_allocated/
    # peak_reserved/clock. A CPU fake may test ordering, never GPU performance.
    if type(warmup) is not int or warmup < 1 or len(batches) <= warmup:
        raise ValueError('warmup and nonempty measured suffix required')
    counts = [tokens(b) for b in batches]
    if any(type(n) is not int or n <= 0 for n in counts):
        raise ValueError('positive integer shifted target counts required')
    for b in batches[:warmup]:
        step(b)
    meter.synchronize()
    base_alloc = meter.allocated()
    base_reserved = meter.reserved()
    meter.reset()
    start = meter.clock()
    for b in batches[warmup:]:
        step(b)
    meter.synchronize()
    elapsed = meter.clock() - start
    peak_alloc = meter.peak_allocated()
    peak_reserved = meter.peak_reserved()
    if not math.isfinite(elapsed) or elapsed <= 0:
        raise ValueError('invalid elapsed time')
    if not (0 <= base_alloc <= peak_alloc <= peak_reserved and
            0 <= base_reserved <= peak_reserved):
        raise ValueError('inconsistent allocator counters')
    budget = sum(counts[warmup:])
    return dict(seconds=elapsed, supervised_tokens=budget,
                supervised_tokens_per_second=budget/elapsed,
                warmup_updates=warmup, measured_updates=len(batches)-warmup,
                baseline_allocated_bytes=base_alloc,
                baseline_reserved_bytes=base_reserved,
                peak_allocated_bytes=peak_alloc, peak_reserved_bytes=peak_reserved,
                peak_above_baseline_bytes=peak_alloc-base_alloc)
 
# Resource function only: never called by the CPU checks.
def cuda_lora_arm(model_dir, mode, cpu_batches, device_index=0, warmup=2):
    import os, time, hashlib
    from pathlib import Path
    os.environ['HF_HUB_OFFLINE'] = '1'
    os.environ['TRANSFORMERS_OFFLINE'] = '1'
    import torch
    from transformers import AutoModelForCausalLM, BitsAndBytesConfig
    from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
    if mode not in {'bf16-lora', 'nf4-qlora'}:
        raise ValueError('unknown arm')
    if not Path(model_dir).is_dir() or not torch.cuda.is_available():
        raise ValueError('existing local model and CUDA device required')
    device = torch.device('cuda', device_index)
    torch.cuda.set_device(device)
    if not torch.cuda.is_bf16_supported():
        raise ValueError('BF16 device support required')
    torch.manual_seed(61)
    torch.cuda.manual_seed_all(61)
    targets = ['q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj']
    kwargs = dict(local_files_only=True, trust_remote_code=False,
                  use_safetensors=True, torch_dtype=torch.bfloat16,
                  device_map={'': device_index}, attn_implementation='eager')
    if mode == 'nf4-qlora':
        kwargs['quantization_config'] = BitsAndBytesConfig(
            load_in_4bit=True, bnb_4bit_quant_type='nf4',
            bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
    model = AutoModelForCausalLM.from_pretrained(model_dir, **kwargs)
    names = [n for n, _ in model.named_modules() if n.split('.')[-1] in targets]
    if set(n.split('.')[-1] for n in names) != set(targets):
        raise ValueError('architecture does not expose the declared target set')
    eligible = {n + '.weight' for n in names}
    if mode == 'nf4-qlora':
        model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=False)
        if any(p.requires_grad for p in model.parameters()):
            raise ValueError('base preparation did not freeze all parameters')
        packed = {n for n, p in model.named_parameters() if p.__class__.__name__ == 'Params4bit'}
        if packed != eligible:
            raise ValueError('quantized module set differs from the BF16 baseline contract')
    else:
        for n, p in model.named_parameters():
            p.requires_grad_(False)
            if n not in eligible and p.is_floating_point():
                p.data = p.data.float()
    for n, p in model.named_parameters():
        if n not in eligible and p.is_floating_point() and p.dtype != torch.float32:
            raise ValueError('nonquantized state policy mismatch')
        if mode == 'bf16-lora' and n in eligible and p.dtype != torch.bfloat16:
            raise ValueError('baseline linear weight is not BF16')
    # Both arms: identical non-reentrant checkpointing and no inference cache.
    model.config.use_cache = False
    model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={'use_reentrant': False})
    torch.manual_seed(610)  # same adapter initialization, independent of loader RNG use
    torch.cuda.manual_seed_all(610)
    model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32,
        target_modules=targets, lora_dropout=0.0, bias='none', task_type='CAUSAL_LM'))
    trainable = []
    for n, p in model.named_parameters():
        if p.device != device:
            raise ValueError('CPU offload or mixed device placement is not this comparison')
        if p.requires_grad:
            if 'lora_' not in n:
                raise ValueError('unexpected trainable parameter')
            p.data = p.data.float()
            trainable.append(p)
    if not trainable:
        raise ValueError('missing adapters')
    optimizer = torch.optim.AdamW(trainable, lr=2e-4, weight_decay=0.0)
    batches, counts = [], []
    digest = hashlib.sha256()
    for b in cpu_batches:
        if set(b) != {'input_ids','attention_mask','labels'}:
            raise ValueError('explicit IDs, attention mask and labels required')
        ids, mask, labels = (b[k] for k in ['input_ids','attention_mask','labels'])
        if any(t.device.type != 'cpu' or t.dtype != torch.long for t in [ids, mask, labels]):
            raise ValueError('pretokenized CPU int64 fixture required')
        if ids.ndim != 2 or ids.shape != mask.shape or ids.shape != labels.shape:
            raise ValueError('aligned batch/sequence shapes required')
        if ids.shape[0] < 1 or not 2 <= ids.shape[1] <= 4096:
            raise ValueError('nonempty batch and bounded sequence required')
        if not ((mask == 0) | (mask == 1)).all() or not mask.bool().any(1).all():
            raise ValueError('invalid attention mask')
        if (ids < 0).any() or (ids >= model.config.vocab_size).any():
            raise ValueError('token outside vocabulary')
        if ((labels != -100) & (labels != ids)).any() or ((mask == 0) & (labels != -100)).any():
            raise ValueError('labels must be IDs or masked, including all padding')
        n = int((labels[:, 1:] != -100).sum())
        if n <= 0:
            raise ValueError('no shifted loss targets')
        for k in sorted(b):
            t = b[k].contiguous()
            digest.update((k + repr(tuple(t.shape))).encode())
            digest.update(t.numpy().tobytes())
        batches.append({k: t.to(device) for k, t in b.items()})
        counts.append(n)
    model.train()
    def step(b):
        optimizer.zero_grad(set_to_none=True)
        with torch.autocast('cuda', dtype=torch.bfloat16):
            loss = model(**b).loss
        loss.backward()
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)
        # No item()/printing in the timed region. Finite checks add synchronization;
        # diagnose numerics in a separate, identically specified preflight run.
    class Meter:
        synchronize = staticmethod(lambda: torch.cuda.synchronize(device))
        reset = staticmethod(lambda: torch.cuda.reset_peak_memory_stats(device))
        allocated = staticmethod(lambda: torch.cuda.memory_allocated(device))
        reserved = staticmethod(lambda: torch.cuda.memory_reserved(device))
        peak_allocated = staticmethod(lambda: torch.cuda.max_memory_allocated(device))
        peak_reserved = staticmethod(lambda: torch.cuda.max_memory_reserved(device))
        clock = staticmethod(time.perf_counter)
    by_identity = {id(b): n for b, n in zip(batches, counts)}
    result = measure_updates(step, batches, lambda b: by_identity[id(b)], Meter(), warmup)
    result.update(mode=mode, fixture_sha256=digest.hexdigest(), device=str(device),
                  device_name=torch.cuda.get_device_name(device), torch_version=torch.__version__,
                  trainable_numel=sum(p.numel() for p in trainable))
    return result

This measures steady-state forward/backward/update wall time, not startup, tokenization, host-to-device transfer, validation or checkpoint I/O. All supplied batches are staged before warmup and included in the memory baseline; keep this fixture small and identical. The denominator is supervised shifted tokens, not all attended input tokens. One update per batch, no accumulation or packing, fixed sequence shapes/order and matched masking are part of the comparison. Hash the local base files, tokenizer/template and package/environment manifest separately; the returned digest only binds the actual batches. Verify the eligible module set, frozen dtypes, adapter names/shapes/initial values and trainable counts across arms in preflight; reject unexpected quantized modules, CPU offload, nonfinite losses/gradients or a nonmatching state policy. Do not treat a failed or OOM baseline as an infinite speedup.

Practice and answer: with two warmup updates and three measured batches of 8, 12 and 10 supervised tokens, a fake clock advancing by 2 seconds reports 30/2 = 15 supervised tokens/s. With a post-warmup allocation of 100 bytes and peak 160, report both 160 total peak and 60 above baseline, not “60 bytes of model memory.” These invented counters only test accounting. On actual hardware, alternate arm order across repeated fresh processes, retain every result/failure, report variation and quality under the same evaluation contract. No timing direction or memory ratio is promised; allocator statistics exclude some driver/library allocations and other processes.

Merge changes the operator: test it explicitly

Training/inference with separate adapters uses Qhat(W)+sBA. Merging into the original full-precision W gives W+sBA, which differs by W-Qhat(W) even before any further rounding. Merging into dequantized Qhat(W) can match in exact arithmetic, but production dtype/operation-order rounding still matters. Requantizing the merged weights adds Qhat(Qhat(W)+sBA)-(Qhat(W)+sBA). It is not guaranteed to improve by running GPTQ/AWQ afterward.

PEFT's tagged bitsandbytes merge path warns about rounding differences. Compare unmerged quantized+adapter, merged dequantized and merged-requantized variants with identical prompts, tokenizer, stop rules, loss and downstream metrics. Use paired uncertainty and application tolerances, not an arbitrary one-point threshold. Keeping adapters separate is often a useful baseline, not an automatic quality guarantee.

Alternatives and the quantized-training landscape

HQQ uses half-quadratic optimization and advertises calibration-free quantization; it is its own project/backend integration, not automatically part of bitsandbytes≥0.44. Compare its exact supported training kernels and precision modes. “95% of BF16 perplexity” is not a well-defined universal quality statement.

GPTQ and AWQ are post-training weight-quantization approaches using calibration information/activation-aware salience in their respective methods. A prequantized inference checkpoint is not automatically LoRA-trainable: backward kernels, architecture, quantization state and PEFT/backend support decide. Do not prescribe a fabricated model repository. Locally available artifacts still need identity and compatibility checks.

BitNet b1.58 studies ternary weights {-1,0,+1} (log2(3)≈1.585bits of symbol information), with actual storage/kernel and higher-precision training details. It is not a post-hoc transformation that makes an ordinary Llama3B checkpoint into a matched BitNet. The public2B4T release is a concrete reference; no universal8×memory/4×energy or2026default prediction follows.

FP8 on supported hardware/software is another compute/storage design with scaling/accumulation rules; it is not simply “automatic loss scaling makes every LoRA run1.8× faster.” A model that fits BF16 may still benefit from quantization for batch/context capacity, or may run slower due to dequantization. Measure rather than insist QLoRA always slows down or always has no benefit when it fits.

Decision procedure: first test whether a smaller base meets the target; then inventory available memory and privacy constraints; compare supported BF16-LoRA, QLoRA and other options at a matched token budget. More hardware or sharding are alternatives, not actions this lesson authorizes. Record quality, peak memory, throughput, startup, merge compatibility and serving cost separately. QLoRA is not the only possible30B–70B strategy.

Worked recall and experiment answers

1. Why NF4? Nonuniform codebook matched to a motivating weight distribution, with zero/block scales. Neither theoretical MSE optimality after absmax normalization nor a universal0.1PPL bound follows.
2. **Double quant?**4.5→4.126953125bits/quantized weight under64/256/32/8 assumptions; about0.326GB for7B and3.264GB for70B, before exemptions/tails/offset metadata.
3. Compute dtype? The arithmetic path for quantized linear operations, distinct from packed storage and other state. Check support and finite values; BF16 is a choice, not immunity.
4. Preparation helper? Freezes base parameters, upcasts eligible nonquantized parameters and configures activation/checkpoint gradients under versioned conditions; it does not generally unfreeze embeddings.
5. Merge loss? The base operator/rounding/requantization can change; compare exact deployment variants before release, not just training loss.
6. 7B memory? Enumerate state/dtypes with the formulas, add actual activations/workspaces and measure. No fixed14GB peak guarantee.
7. FP16 NaNs onA100? Inspect first nonfinite, IDs/masks and scaling; supported BF16 may help range, but blindly switching dtype is not a diagnosis.

For the original WikiText stretch, use an existing local model and corpus; freeze tokenizer/stride/context/scored targets, measure mean NLL then PPL for BF16 and quantized bases on identical examples, and report paired deltas. PPL has no “nats” unit; NLL does. Layer sensitivity can be studied by quantizing one layer/group at a time under the same fixture, but effects need not add or vary monotonically with model size. The bounded CPU experiment here tests operators/gradients, not WikiText or Mistral quality.

Sources and execution boundary

QLoRAv1 full paper, bitsandbytes0.48.1 codebook, PEFT0.17.1 preparation, PEFT merge, GPTQ, AWQ, BitNet2B4T card.

Executed: exact CPU NF4 codebook/nibble/zero-block and adapter/merge controls plus independent derivatives/arithmetic. The measurement state machine is checked with a CPU fake clock/counters only; the matched CUDA arms remain unexecuted and supply no benchmark result. Unexecuted: bitsandbytes/PEFT/TRL integration, paged optimizer behavior, GPU allocations,7B/65B fine-tuning and model benchmarks. No downloads of weights/data or package installation occurred. PyTorch 2.8.0 allocator contracts and device synchronization govern the measurement API. Optional reading and next steps are listed below. Review tomorrow: separate four-bit storage from dX and compute the metadata overhead without mixing scale dtypes.

Inherited optional resources and navigation

These pre-read links are optional further reading; their timestamps, video claims and optional pacing are not accepted technical evidence.

• Efficient Fine-Tuning for Llama-v2-7b on a Single GPU
• Mistral Fine Tuning for Dummies (with 16k, 32k, 128k+ Context)
• QLoRA: Efficient Finetuning of Quantized LLMs
• bitsandbytes documentation

Previous: ← DL S060 · Next: DL S062 →