DL S061 · QLoRA — 4-bit + LoRA

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 61 / 80 · Module M10 · ~2 hours

The story

From "needs an A100" to "runs on a gaming laptop"

Yesterday LoRA shrank the trainable footprint of a 7B model from 56GB to about 20GB. Good. But 20GB is still too big for a 4090 or a Colab T4. The bottleneck is the frozen base: 7B params × 2 bytes (bf16) = 14GB, before any activations or LoRA state.

Enter QLoRA (Dettmers et al., May 2023). Three ideas stacked:

1. NF4 quantization — store the frozen base weights in a custom 4-bit format instead of bf16. 4× smaller. Base drops from 14GB to 3.5GB.
2. Double quantization — even the quantization scale factors are quantized. Saves another 0.5GB on a 7B model.
3. Paged optimizers — spill optimizer state to CPU RAM through NVIDIA unified memory so a single OOM spike doesn't kill the run.

Result: full-quality fine-tuning of a 65B parameter model on a single 48GB GPU. Or a 7B model on a 12GB gaming card. The paper's own tagline: "Guanaco, a 65B model matching ChatGPT quality, trained on one GPU in 24 hours."

This is the technique that democratized fine-tuning. Today we understand why it works, what NF4 actually is, and how to configure bitsandbytes + peft + TRL for a real 7B fine-tune on hardware you might actually own.

Prerequisites

• Session 060 (LoRA) — QLoRA = quantize the base + LoRA on top. LoRA is a hard dependency.
• Session 058 (SFT) — the training loop is unchanged.
• Basic quantization vocabulary: know what "int8" vs "int4" vs "fp16" means at the bit level. If not, S067 will formalize it; today we operate at a slightly higher level.

1 · The memory problem, quantified

Recall the memory bill for fine-tuning a 7B model:

The base weights are the big block. Full-fp16 SFT of a 7B was hopeless on consumer hardware even with LoRA — 24GB was tight, and any longer context blew you up. Cutting the base to 4-bit is the whole ballgame.

The trick QLoRA has to solve: quantize the base to 4 bits without destroying the model's quality, then still get useful gradients through it for the LoRA adapters. Both halves are non-trivial.

2 · NF4 — why not just use INT4?

The naive way to quantize a weight matrix to 4 bits: pick min and max, uniformly quantize to 16 levels (INT4), store a scale factor. Symmetric or asymmetric, per-tensor or per-channel — a dozen variants exist and they all lose meaningful quality at 4 bits.

Dettmers's insight: pretrained transformer weights are approximately normally distributed with mean 0. So uniform quantization wastes bit budget on the tails and starves the center. What we actually want is a quantization grid whose 16 levels match the quantiles of a standard normal distribution.

2.1 The NF4 quantile trick

Compute the quantiles of N(0, 1) at 15 evenly-spaced probabilities from 1/32 to 31/32. Add zero as one exact level (crucial for sparsity). That gives 16 levels. Normalize to [-1, 1]. This is NF4.

NF4 levels (16 values, symmetric around 0):
[-1.0, -0.6962, -0.5251, -0.3949, -0.2844, -0.1848, -0.0911, 0.0,
  0.0796, 0.1609, 0.2461, 0.3379, 0.4407, 0.5626, 0.7229, 1.0]

Density is highest near zero (matching the weight distribution) and sparse in the tails.

To quantize a weight w in a block:

1. Divide the weight matrix into blocks of size 64.
2. For each block, compute abs_max = max(|w|).
3. Normalize: w_norm = w / abs_max  ∈ [-1, 1]
4. Find the nearest NF4 level. Store its 4-bit index (0–15).
5. Store abs_max as an fp16 scale factor per block.

Storage per block of 64 weights: 64 × 4 bits (indices) + 16 bits (scale) = 272 bits = 34 bytes. Compare to bf16: 64 × 16 bits = 128 bytes. Compression ratio: 3.76×. (Not exactly 4× because of the scale factor overhead.)

2.2 Why the quantile grid wins

Empirically, NF4 loses ~0.1 – 0.3 perplexity points compared to fp16 on standard LLM tasks. Uniform INT4 loses 0.5 – 2 points. On a 7B model that's a difference between "unusable" and "you might not notice".

The intuition: most of a weight matrix's information is in the near-zero region (small updates from many samples). If your quantization grid is uniform, you have maybe 4 levels covering the "interesting" [-0.2, 0.2] range. If it's NF4, you have 10 levels there. 2.5× more resolution where it matters.

3 · Double quantization — the other 0.5GB

Every block of 64 weights carries a 16-bit scale factor. For a 7B model with all-linear-layers quantized:

• ~6.7B params in linear layers
• 6.7B / 64 = 105M blocks
• 105M × 2 bytes (fp16) = 210 MB of scale factors

210MB isn't 14GB, but on a 65B model it becomes ~2GB, and it's memory you're wasting.

Double quantization: quantize the scale factors themselves. Group them into meta-blocks of 256, and quantize each meta-block's scale to 8-bit with its own fp32 meta-scale.

New cost per meta-block of 256 scales:

• 256 × 8 bits (quantized scales) + 32 bits (meta-scale) = 2080 bits = 260 bytes
• vs 256 × 16 bits = 512 bytes
• Compression: ~2×

Total effective bit rate per weight: 4.0 → ~4.5 bits/param including everything. That's the "4.5 bpp" number you'll see in QLoRA charts.

Memory saved on a 7B: ~100MB. On a 70B: ~1GB. Not free, worth having.

4 · Paged optimizers — the OOM insurance

Even with NF4 base + LoRA, you can OOM if:

• Sequence length spikes (long user in the batch)
• Activation checkpointing not enabled
• Attention memory scales with seq² and you hit an outlier

QLoRA uses NVIDIA's unified memory (available on all datacenter GPUs since Pascal, and consumer GPUs via CUDA) to let the AdamW optimizer state live in CPU RAM and be paged to GPU on demand.

from bitsandbytes.optim import PagedAdamW32bit
optimizer = PagedAdamW32bit(model.parameters(), lr=2e-4)

Normal AdamW stores 2 moments per param in fp32. Even for LoRA-only params (~20MB for a 7B LoRA), that's fine. Paged AdamW becomes critical when:

• You're training a fuller LoRA (r=64+) with more trainable params.
• You have activation memory spikes.

The paging is transparent — you notice slightly higher training latency (~5–15%) in exchange for OOM safety. Almost always worth it for QLoRA runs.

5 · The compute dtype question

Here's a subtlety that trips up most beginners.

The base weights are stored in NF4. But you can't do matmul on NF4 — no GPU hardware supports it. So at each forward pass, bitsandbytes dequantizes NF4 → fp16 or bf16 on the fly, does the matmul, discards the fp16 tensor.

The dequantized dtype is the compute dtype, and you set it explicitly:

• bf16 compute (Ampere+ GPUs: 3090, A100, H100, 4090): dynamic range matches fp32 for most weights; recommended.
• fp16 compute (older GPUs: T4, V100, 2080): smaller range; watch for grad NaNs on longer training.
• fp32 compute (rarely used): safe but slow, halves throughput.

6 · A real QLoRA fine-tune, end to end

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTConfig, SFTTrainer
from datasets import load_dataset
 
MODEL = "mistralai/Mistral-7B-v0.3"
 
bnb_cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
 
tok = AutoTokenizer.from_pretrained(MODEL)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

Load the model quantized:

model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    quantization_config=bnb_cfg,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
model = prepare_model_for_kbit_training(model)   # enables grad checkpointing + input embedding grads

prepare_model_for_kbit_training does three important things: casts LayerNorms to fp32 for stability, enables gradient checkpointing (trades compute for activation memory), and enables gradients on the input embedding (needed because you can't backprop through 4-bit weights back to the input).

Now attach LoRA:

lora_cfg = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_cfg)
model.print_trainable_parameters()
# trainable params: 41,943,040 || all params: 3,793,833,984 || trainable%: 1.1056

Notice the total param count reads ~3.8B instead of 7B — that's because bitsandbytes counts NF4 params at their compressed size. The model still has 7B "logical" parameters.

Train:

ds = load_dataset("tatsu-lab/alpaca", split="train").map(to_chatml)  # from S058
 
cfg = SFTConfig(
    output_dir="./qlora-mistral-alpaca",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    logging_steps=25,
    save_steps=500,
    max_seq_length=2048,
    packing=True,
    optim="paged_adamw_8bit",       # ← paged optimizer!
    gradient_checkpointing=True,
)
 
trainer = SFTTrainer(model=model, args=cfg, train_dataset=ds, tokenizer=tok)
trainer.train()
trainer.save_model()

On an RTX 4090 24GB: ~4 hours, peak VRAM ~15GB. On a Colab T4 16GB: ~14 hours, peak ~13GB (drop batch to 2 and grad-accum to 16).

Post-training you have a ~80MB adapter file. Loading it back for inference:

from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(MODEL, quantization_config=bnb_cfg, device_map="auto")
model = PeftModel.from_pretrained(base, "./qlora-mistral-alpaca")

You're serving a 4-bit quantized base + fp16 LoRA. Latency is ~10% slower than fp16 base + LoRA. For most workloads, worth the memory savings.

7 · QLoRA quality vs full-precision LoRA

The QLoRA paper's headline result: NF4 + LoRA matches fp16 full fine-tuning across 8 tasks. Delta typically < 0.5 points MMLU. On the Guanaco eval set (Vicuna-style automated GPT-4 grading), QLoRA-tuned 33B matched fp16-tuned 65B in preference tests.

For your own runs, expect:

• MMLU delta: -0.5 to -1 point vs full-precision LoRA on same data.
• Perplexity delta: +0.05 to +0.2 (higher = worse) on WikiText.
• Human preference: essentially indistinguishable in blind eval.

The one place quality drops meaningfully: very small models (< 1B). NF4's quantile grid was tuned for weight distributions typical of 3B+ models; sub-1B weights have thinner tails and lose more.

8 · Diagram — the QLoRA data flow

9 · The 2024–2025 quantized-fine-tune landscape

QLoRA (Dettmers et al., May 2023) was the paper that made 65B fine-tuning tractable on a single 48GB GPU. Two years later the field has moved on — sometimes past QLoRA, sometimes around it. Here's the current map.

9.1 QLoRA's actual quality cost, revisited

The original QLoRA paper claimed "no degradation vs 16-bit LoRA." Later ablations (Xu et al. 2024, A Comprehensive Evaluation of Quantization Strategies for Large Language Models) showed the truth is more nuanced:

• On instruction-following (MT-Bench, AlpacaEval): QLoRA loses <0.5 pts.
• On reasoning (MMLU, GSM8K, MATH): QLoRA loses 1–3 pts vs bf16-LoRA.
• On code generation (HumanEval): QLoRA loses 2–5 pts. Code is unforgiving of even small logit shifts.

Moral: for chat/instruction work, QLoRA is basically free. For code and math specialization, prefer bf16-LoRA if you can afford it.

9.2 HQQ — half-quadratic quantization (Badri & Shaji, 2024)

NF4 was designed for weights following a normal distribution. HQQ makes no distributional assumption and instead solves a fast per-block optimization. It quantizes to 4-bit or 2-bit without calibration data, in seconds per billion params. On Llama-2-70B, HQQ-4bit hits 95% of bf16 zero-shot perplexity vs QLoRA-NF4's 93%. Integrated in bitsandbytes>=0.44.

Further reading: https://mobiusml.github.io/hqq_blog/

9.3 GPTQ / AWQ pre-quantized bases

QLoRA quantizes on-the-fly at load. GPTQ (Frantar et al. 2022) and AWQ (Lin et al. 2023) pre-quantize with a calibration set and save 4-bit weight files. Common pattern: download TheBloke/Llama-3.1-8B-Instruct-AWQ, attach LoRA adapters, fine-tune. Faster startup, slightly better base quality (AWQ preserves salient weights).

9.4 BitNet b1.58 — the endgame? (Ma et al., Microsoft 2024)

BitNet trains from scratch with weights in $\{-1, 0, +1\}$ — $\log_2(3) \approx 1.58$ bits. A 3B BitNet matches Llama-3-3B in bf16 but uses 8× less memory and 4× less energy at inference. LoRA-adaptable in fp16. If the trend holds, the 2026 default consumer LLM will be BitNet-scale.

Further reading: https://arxiv.org/abs/2402.17764

9.5 FP8 fine-tuning on Hopper/Blackwell

H100/H200/B200 GPUs support FP8 (e4m3/e5m2) natively. transformer_engine trains LoRA in FP8 with auto loss scaling: 1.6–1.8× speedup on H100, 2.0–2.5× on B200, quality matches bf16. On this hardware, QLoRA is a needless quality tax — use it only when memory is the actual bottleneck.

9.6 The 2025 decision tree

10 · Retention scaffold

Recall

Q1. Why does NF4 outperform uniform INT4 quantization for LLM weights?

Q2. What does double quantization save, and how much?

Q3. What is compute_dtype in a QLoRA config and why does it matter?

Q4. Why do we need prepare_model_for_kbit_training before attaching LoRA to a quantized model?

Q5. What's the classic post-merge failure mode with QLoRA?

Stretch prompt

Design a small experiment to measure the perplexity delta between NF4-quantized Mistral-7B and fp16 Mistral-7B on WikiText-2. Predict: (a) will the delta be uniform across all layers, or larger for some (attention vs MLP, early vs late)? (b) How would you expect the delta to scale with model size?

In your own words

Write 3–4 sentences explaining to a colleague why QLoRA lets you fine-tune a 7B model on a 24GB GPU when fp16 SFT couldn't. Force yourself to name the two big memory contributors it kills and the two contributors it doesn't touch.

Spaced review

• S060 — LoRA. QLoRA = LoRA + quantized base. Zero LoRA understanding = you can't debug QLoRA.
• S058 — SFT. The loss and training loop are unchanged; QLoRA is a pure memory optimization.
• S067 (upcoming) — Quantization deep-dive. NF4 is one point in a bigger design space; we'll see GPTQ, AWQ, and post-training quantization.

Next session

S062 — Reward modeling. SFT teaches format. But format ≠ preference. Two responses that look equally well-formed can be very different in usefulness. Tomorrow we train a reward model on preference pairs (chosen vs rejected) — the Bradley-Terry model, the RM architecture (base + regression head), and the dataset-quality traps that make or break RLHF.

Bring back tomorrow

• The three QLoRA tricks: NF4, double quant, paged optimizers.
• compute_dtype=torch.bfloat16 is the default that works on Ampere+.
• Merging QLoRA has a rounding-error trap; eval post-merge separately.

Previous: ← DL S060 · Next: DL S062 →