DL S067 · Quantization — INT8, INT4, GPTQ, AWQ

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 67 / 80 · Module M11 · ~2 hours

The story

Yesterday (S066) we established: KV cache + fp16 weights = you can't fit two users on an A10. Today: what if the entire model were 4× smaller? Suddenly Llama-2-7B is 3.5 GiB instead of 14 GiB, and you have room to breathe.

The trick is quantization. And here's what's amazing: modern quantization methods lose less than 1 percentage point on MMLU going from fp16 to 4-bit weights. That's a factor of 4 in memory and typically 2× in latency, essentially for free. Nothing else in ML has this ROI.

But "quantization" isn't one thing. There's naive round-to-nearest (RTN), which is trash on LLMs. There's LLM.int8() (bitsandbytes) which handles outlier features but is slow at inference. There's GPTQ which uses second-order info. There's AWQ which is faster and often better. There's the newer QuaRot / SpinQuant family that rotates weights first. And there's BitNet, which trains from scratch with 1.58-bit weights and questions the entire premise of post-training quantization.

Today we derive the core math (scale + zero-point + group size), implement RTN by hand, then compare GPTQ vs AWQ vs bnb vs fp8 vs QuIP# on real numbers. By the end you'll know which one to reach for.

The origin story: Dettmers finds the outliers

August 2022. Tim Dettmers is trying to serve OPT-175B on a research cluster with too few GPUs. He tries the obvious thing: round every weight to INT8, matmul in INT8, done. On models up to 6.7B params it works. At 13B it starts to hurt. At 175B it catastrophically fails — accuracy craters, generation degenerates. Meanwhile, a well-known technique (per-channel INT8) that works fine on ResNet-50 gives the same bad result on OPT.

Dettmers pulls up a histogram of the activations and sees it. In every layer past a certain scale, ~0.1% of feature dimensions have magnitudes 20–100× larger than the median. These outlier features dominate the matmul output. Round-to-nearest quantizes to a scale that accommodates them and destroys resolution for everything else.

The fix he publishes as LLM.int8() (NeurIPS 2022) is basically "detect these columns at runtime, keep them in fp16, quantize the rest." It works. It's also slow because the mixed-precision matmul has no fused kernel. But it opens the floodgates. Within 12 months: GPTQ, AWQ, SmoothQuant, SqueezeLLM, OmniQuant. Within 24 months: BitNet, QuIP#, HQQ, AQLM.

Every one of them is answering the same question Dettmers asked in August 2022: what do you do about the outlier features? Keep that question in your head as we go through the methods; it'll organise everything.

Prerequisites

• Session 066 — memory budget and per-token math.
• Session 060 — QLoRA already uses 4-bit; now you'll know what NF4 actually is.
• Session 016 — precision, underflow, overflow.

1 · The core math

Quantization maps a floating-point range $[\alpha, \beta]$ to an integer range $[q_{min}, q_{max}]$. For INT8 asymmetric: $[q_{min}, q_{max}] = [0, 255]$; for INT8 symmetric: $[-127, 127]$; for INT4: $[-8, 7]$ or $[0, 15]$.

Asymmetric (with zero-point): $\text{scale} = \frac{\beta - \alpha}{q_{max} - q_{min}}, \quad z = \text{round}\left(q_{min} - \frac{\alpha}{\text{scale}}\right)$ $q = \text{clip}\left(\text{round}\left(\frac{x}{\text{scale}}\right) + z, \; q_{min}, q_{max}\right)$ $\hat{x} = \text{scale} \cdot (q - z)$

Symmetric (weights usually): drop the zero-point, use $\text{scale} = \max(|x|) / q_{max}$.

Numeric example. Weight vector $x = [-0.9, -0.1, 0.05, 0.7, 1.2]$, INT8 symmetric.

• $\max|x| = 1.2$, scale $= 1.2 / 127 = 0.00945$
• $q = \text{round}(x / 0.00945) = [-95, -11, 5, 74, 127]$
• Dequant $\hat{x} = q \cdot 0.00945 = [-0.898, -0.104, 0.047, 0.699, 1.200]$
• Error $\|x - \hat{x}\|_\infty = 0.004$. Under 0.5%. Nice.

2 · Per-tensor vs per-channel vs per-group

Applying one scale per tensor to a $4096 \times 4096$ weight matrix is catastrophic — one giant outlier weight burns the entire scale for everyone. Fix: use per-channel scales (one per output row) or per-group (one per group of 128 consecutive weights along the input dim). Per-group with group_size=128 is the default in GPTQ/AWQ/bnb.

Storage cost: for 4-bit weights + fp16 scales at group=128, the effective bits-per-weight is $4 + 16/128 = 4.125$. Basically free.

import torch
 
def quantize_per_group(w, bits=4, group_size=128):
    """w: (out_features, in_features). Returns q (int), scale (fp16)."""
    out_f, in_f = w.shape
    w = w.view(out_f, in_f // group_size, group_size)  # (out, G, gs)
    max_abs = w.abs().amax(dim=-1, keepdim=True)       # (out, G, 1)
    qmax = 2 ** (bits - 1) - 1                          # 7 for INT4
    scale = max_abs / qmax
    q = torch.round(w / scale).clamp(-qmax - 1, qmax).to(torch.int8)
    return q, scale.squeeze(-1)  # scale: (out, G)
 
def dequantize(q, scale, group_size=128):
    out_f, G = scale.shape
    return (q.view(out_f, G, group_size).float() * scale.unsqueeze(-1)).view(out_f, G * group_size)

That's real Round-To-Nearest per-group. It's fine for CNNs and mediocre for LLMs. Why?

3 · The outlier problem

Dettmers et al. (LLM.int8, 2022) discovered: in LLMs beyond ~6.7B params, a small number of feature dimensions have activations that are ~20× larger than the rest. RTN quantization of those channels blows the scale, and the other 99% of features lose all resolution.

Every modern method is really a strategy for the outlier problem:

• LLM.int8() (bitsandbytes): detect outlier columns at runtime, keep them in fp16, quantize the rest to INT8. Mixed-precision matmul. Zero calibration, works out of the box. Slow at inference.
• GPTQ (Frantar et al., 2022): quantize column by column, at each step solve a small linear system to compensate remaining columns for the introduced error. Uses the inverse Hessian $H^{-1} = (2 X^T X)^{-1}$ from a calibration set. Slow to quantize (30 min for 7B), fast at inference.
• AWQ (Lin et al., 2023): observe that only ~1% of weight channels matter (the ones multiplied by outlier activations). Rescale weights so those channels have smaller magnitude before quantization; absorb the inverse scale into the previous layer. Fast to compute (5 min for 7B), fast at inference, often beats GPTQ.
• fp8 (E4M3/E5M2): H100+ hardware supports fp8 matmul natively. No outlier drama, but you need Hopper.
• NF4 (QLoRA): a 4-bit code optimized for normally-distributed weights. Info-theoretically optimal for that distribution. Used inside bitsandbytes 4-bit.

4 · GPTQ intuition

Frantar's insight: quantization introduces error $\delta_j$ on the $j$-th column. The output error on the linear layer is $\|X \delta\|^2$. If we quantize columns one at a time, after quantizing column $j$ we can adjust remaining columns $j+1..n$ to compensate. The optimal adjustment is a closed-form solution involving the inverse Hessian of the layer's input covariance:

$\text{Update: } W_{j+1:} \mathrel{-}= \frac{W_j - \hat{W}_j}{[H^{-1}]_{jj}} \cdot H^{-1}_{j, j+1:}$

You need a small calibration set (~128 samples) to estimate $H = 2 X^T X$. Then it's per-layer, embarrassingly parallel across layers. The AutoGPTQ library does this in ~30 min for a 7B model.

Practical usage:

from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
 
quant_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer, group_size=128)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", quantization_config=quant_config)
model.save_pretrained("llama-2-7b-gptq-4bit")

Or just download a pre-quantized one from TheBloke on HuggingFace and skip the 30 minutes.

5 · AWQ intuition

Lin et al. observed that on Llama-7B, quantizing just 1% of the weight channels — the ones aligned with outlier activation channels — in fp16 and the rest in INT3 recovers nearly all quality. But mixed precision is a pain to serve. Their trick: for each salient channel $j$, multiply the weight column by scalar $s_j > 1$ and divide the activation column by $s_j$. Elementwise math is unchanged, but weights are now smoother and quantize better.

You find $s_j$ per-channel by grid search minimizing $\|W \cdot X - Q(W \cdot s) \cdot (X / s)\|$ on a tiny calibration set. Takes minutes. Very cheap.

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
 
model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
 
quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized("llama-2-7b-awq")

6 · bitsandbytes — the "just make it fit" button

If you don't want to think, load with:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
 
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", quantization_config=bnb_config)

That's the QLoRA config from S060. Zero calibration, works everywhere, ~30% slower at inference than GPTQ/AWQ. Perfect for training and prototyping, not ideal for production serving.

7 · Benchmarks that matter

Numbers from the AWQ paper + our own runs on A100, Llama-2-7B, WikiText-2 perplexity (lower better):

Takeaways:

• Per-tensor RTN is unusable. Group-wise RTN is fine.
• GPTQ/AWQ recover essentially all quality.
• AWQ is faster because its kernel (INT4 GEMM) is simpler than GPTQ's.
• bnb is slowest but most flexible.

Latency picture: AWQ ~2.7× faster than fp16 at batch=1 on A100. On memory-bound decode, roughly speedup ≈ compression_ratio.

The 2024–2025 quantization scoreboard (updated)

The 2022–2023 methods above are now the baseline. Here's what shipped since:

fp8: the boring win

Hopper's E4M3 fp8 (4-bit exponent, 3-bit mantissa) is post-training quantizable with almost zero quality loss. NVIDIA's TransformerEngine does it automatically. vLLM 0.5+ ships --quantization fp8 as a one-liner. On Llama-3-70B, fp8 weights + fp8 KV halve memory and double throughput vs bf16 with a MMLU delta of <0.1.

If you have H100/H200, fp8 is the answer 90% of the time. GPTQ/AWQ are for A100 and older.

BitNet 1.58: the "train it quantized" gambit

Ma et al., "The Era of 1-bit LLMs" (Microsoft, 2024) trained a 3B model where every weight is in $\{-1, 0, +1\}$ — log₂(3) ≈ 1.58 bits. Matmul becomes addition; no multiplier needed. At 3B scale they match fp16 baselines on perplexity. At 70B (BitNet b1.58 70B, 2024) the ppl gap closes further.

The catch: post-training BitNet-ification of a normal LLM fails badly. You have to pretrain with the quantization in the forward pass. Nobody has spent Llama-3-scale compute on it yet, so the 70B+ regime is untested. If a lab did it, inference cost would drop 10× overnight. Watch this space.

QuIP# and 2-bit territory

Tseng et al. (QuIP#, 2024) push post-training quantization down to 2 bits per weight by first rotating weights with a random orthogonal matrix (incoherence processing) then encoding with an E8 lattice codebook. Llama-2-70B at 2-bit QuIP# loses ~0.5 perplexity — shockingly good. The 70B model fits on a single 24GB 4090.

Marlin kernels (IST-DASLab, 2024) provide fused INT4/INT2 matmuls on Ampere+ that hit >90% of memory bandwidth peak. If you serve AWQ or GPTQ in 2025, you're probably going through Marlin whether you know it or not — vLLM auto-dispatches.

W8A8, W4A16, W4A8: what do these codes mean?

Modern papers name their scheme with W<weight_bits>A<activation_bits>:

• W16A16: bf16 baseline. Nothing quantized.
• W8A16 / W4A16: weight-only quantization (GPTQ, AWQ, bnb-4bit). Activations stay bf16. Best quality, easy to implement.
• W8A8 (SmoothQuant, LLM.int8): both weights and activations INT8. Needs INT8 hardware (T4, A100, H100). 2× compute throughput on top of memory savings.
• W4A8: exotic. Only pays off with fp8 activations on H100. TensorRT-LLM supports it.
• W4A4: extreme; QuaRot (2024), Atom (2024). Massive throughput but 1–2 ppl points.

Rule of thumb: weight-only wins in memory-bound (single-request, long context). Weight-and-activation wins in compute-bound (large batches, short context). vLLM's autoconfig picks reasonably.

Further reading — the 2025 frontier:

• Ma et al., BitNet b1.58 (2024).
• Tseng et al., QuIP# (ICML 2024).
• Xiao et al., SmoothQuant (ICML 2023).
• IST-DASLab, Marlin: fast INT4 kernels (2024).
• NVIDIA, TransformerEngine fp8 recipe.
• Egiazarian et al., AQLM: 2-bit quantization via additive quantization (2024).

8 · When quantization silently hurts

9 · Mermaid: decision tree

Recall

Stretch: derive the AWQ scaling equivalence: for a linear layer $y = Wx$, show that $y = (W \cdot \text{diag}(s)) \cdot (\text{diag}(1/s) \cdot x)$ is mathematically equivalent. Where do you push diag(1/s)? Answer: into the previous layer's weights (LayerNorm or the prior Linear), so activations arriving here are already pre-divided.

Stretch 2: BitNet b1.58 uses ternary weights $\{-1, 0, +1\}$. Show that matmul can be done with only additions (no multiplies). Estimate the energy-per-token improvement vs bf16 GEMM assuming FP16 multiply-adds are 10× more energy than integer adds.

In your own words: the outlier problem in LLM quantization is __________________________.

Spaced review: S060 QLoRA (NF4 double-quant), S016 numerical stability, S066 KV cache (memory is why we're doing this).

Next session (S068): we go beyond quantization to pruning (removing weights entirely) and distillation (training a small model to mimic a big one). When do you reach for each?

Bring back tomorrow:

• The scale/zero-point formula.
• The outlier framing of quantization methods.
• Your fp16-vs-INT4 latency + PPL numbers on a model you actually use.

import torch
# W: any linear weight, or capture an activation tensor with a forward hook
per_tensor = W.abs().max()
per_channel = W.abs().amax(dim=1)          # one scale per output row
print('max/median channel ratio:', (per_channel.max()/per_channel.median()).item())

def q(x, scale, bits):
    n = 2**(bits-1) - 1
    return torch.round(x/scale*n).clamp(-n, n) / n * scale

for bits in (8, 4):
    e_t = (q(W, per_tensor, bits) - W).abs().mean()
    e_c = (q(W, per_channel[:,None], bits) - W).abs().mean()
    print(bits, 'per-tensor', e_t.item(), 'per-channel', e_c.item())

Previous: ← DL S066 · Next: DL S068 →