S106 · Tokenization — BPE, WordPiece, SentencePiece

Module M13: NLP & Transformers · Session 106 of 130 · Track: LLM 🔤

What you'll be able to do after this session

• Explain Tokenization to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S104 — Embeddings — word2vec, GloVe, Contrastive Learning

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Byte Pair Encoding Tokenization — HuggingFace — BPE in plain English with pictures.
• 🎥 Deep dive (30–60 min) · Let's build the GPT Tokenizer — Andrej Karpathy — Karpathy builds a real BPE tokenizer from bytes up.
• 🎥 Hands-on demo (10–20 min) · Hugging Face Tokenizers walkthrough — Load a tokenizer, encode, decode, inspect vocab.
• 📖 Canonical article · Hugging Face — Summary of the tokenizers — Definitive comparison of BPE, WordPiece, and SentencePiece.
• 📖 Book chapter / tutorial · Neural Machine Translation of Rare Words with Subword Units (Sennrich 2016) — The original BPE-for-NLP paper.
• 📖 Engineering blog · Google Research — Transformer, a novel neural network architecture — Frames why subword tokenization is essential.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: How text becomes numbers.

Language models eat integers, not text. So the very first step of every LLM pipeline is a tokenizer — a deterministic function that maps a string to a sequence of integer IDs (and back). This sounds boring, but it's arguably the most impactful design decision in an LLM after "which architecture". The tokenizer determines your vocabulary size, your context length in effective characters, how well you handle non-English scripts, whether you can spell, and even the flavour of your model's mistakes.

The naive approach — one token per word — fails spectacularly. Any word not in your vocabulary (typos, names, new slang, chemistry terms) becomes an unknown token, and the model has zero information about it. The naive fix — one token per character — makes sequences 5× longer, tanking compute cost, and forces the model to reinvent morphology from scratch.

The winning idea is subword tokenization: build a vocabulary of common character sequences of varying lengths. Common words ("the", "hello") get their own token; rare words ("unbelievability") get broken into "un" + "believ" + "ability". The two dominant algorithms are Byte-Pair Encoding (BPE, used by GPT-2/3/4, Llama, Mistral) and WordPiece (used by BERT). SentencePiece is a wrapper/implementation that operates directly on raw bytes so it's language-agnostic (no need to pre-tokenise on whitespace) — used by T5, Llama, and most modern models.

Gotcha to carry forever: Whitespace and leading spaces are part of the token. "hello" and " hello" (with leading space) are usually different token IDs in GPT tokenizers. This is why prompt engineering sometimes cares about exact spacing, and why counting "words" in a prompt to estimate tokens is always wrong.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

How BPE builds a vocabulary. Start with characters as your vocab. Count all adjacent pairs across your corpus. Merge the most frequent pair into a new token. Repeat until you reach the target vocab size.

Worked example — encoding a word with a hypothetical BPE. Suppose the vocab contains ["un", "believ", "able", "believable"]. The tokenizer greedily finds the longest matches:

• Try "unbelievable" → not in vocab
• Match "un" (id=42), remaining = "believable"
• Match "believable" (id=857), remaining = ""
• Output: [42, 857] — 2 tokens for a 12-character word.

If "believable" weren't in the vocab, we'd fall back to [un, believ, able] — 3 tokens.

Worked example — actual GPT-4 tokenization. Try it at platform.openai.com/tokenizer:

• "Hello, world!" → 4 tokens: Hello , world !
• "antidisestablishmentarianism" → roughly 6 tokens.
• "你好世界" (Chinese "hello world") → 8 tokens.
• Same string in Hindi or Tamil often uses even more tokens — a real fairness issue in per-token pricing.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# tokenizer_lab.py — poke at a real GPT-4 tokenizer and BERT
# pip install tiktoken transformers
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")           # GPT-3.5 / GPT-4
 
samples = [
    "Hello, world!",
    " Hello, world!",                                # note leading space
    "antidisestablishmentarianism",
    "The quick brown fox jumps over the lazy dog.",
    "你好世界",                                       # Chinese
    "नमस्ते दुनिया",                                    # Hindi
    "print('hi')",                                   # code
]
for s in samples:
    ids = enc.encode(s)
    pieces = [enc.decode([t]) for t in ids]
    print(f"{len(ids):3d} tokens | {s!r}")
    print("           pieces:", pieces)
 
# Round-trip must be lossless
assert enc.decode(enc.encode("Hello, world!")) == "Hello, world!"
 
# ---- BERT WordPiece for comparison --------------------------------------
from transformers import AutoTokenizer
bert = AutoTokenizer.from_pretrained("bert-base-uncased")
print("\nBERT tokens for 'unbelievable':", bert.tokenize("unbelievable"))
print("BERT tokens for a URL:", bert.tokenize("Visit https://openai.com/gpt-4"))
 
print("\nGPT-4 vocab size:", enc.n_vocab)
print("BERT vocab size :", bert.vocab_size)

Checklist while it runs:

1. Confirm "Hello" and " Hello" produce different token IDs — leading-space is significant.
2. Watch how many more tokens Hindi/Chinese take compared to equivalent English — that's the tokenizer fairness issue.
3. Notice BERT prints ##able in "un ##believ ##able" — ## marks a continuation subword.
4. Confirm the round-trip assertion — decode(encode(x)) == x for any UTF-8 string.
5. Try a long emoji sequence ("😀" * 100) and see how many tokens it eats.

Try this modification: Train your own tiny BPE on your favourite short text using tokenizers from Hugging Face — 20 lines will give you a 500-token vocabulary you can inspect end-to-end.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Tokenizers cause more subtle production bugs than any other component of an LLM pipeline, precisely because they look trivial.

Cost and latency are token-denominated. OpenAI, Anthropic, and every hosted LLM bill per input+output token, and inference latency scales roughly linearly with token count. A prompt that tokenises to 3× more tokens in Hindi than English is literally 3× more expensive — a documented fairness concern that led to research on "token efficiency" and multilingual tokenizers (SentencePiece with larger vocabularies for CJK/Devanagari scripts).

Retraining vocabulary. Once you ship a model, its tokenizer is frozen forever — the embedding table has one row per token ID. Changing the vocab means retraining the model. This is why Llama-3 came with a much bigger vocabulary (128K) than Llama-2 (32K) — Meta traded some efficiency for better multilingual coverage, but at the cost of a full retraining.

Common bugs:

• Tokenizer-model mismatch. Loading a GPT-2 tokenizer with a Llama model gives you gibberish token IDs → garbage outputs. Always load them as a pair (same from_pretrained repo).
• Truncation surprises. You truncate a 10,000-token document to 4,096 and lose the answer that was at position 5,000. RAG pipelines must measure in tokens, not characters or words.
• Whitespace normalisation. Some tokenizers strip repeated whitespace, some preserve it. Matters for code models: def foo():\n pass vs def foo(): pass differ by several tokens.
• BOS/EOS/pad confusion. BERT expects [CLS] + tokens + [SEP]. GPT often doesn't need a BOS. Passing wrong special tokens silently degrades performance by 5–10 points.

War story: SolidGoldMagikarp. In 2023, researchers discovered that certain tokens (" SolidGoldMagikarp", " petertodd") had been included in GPT-2/3's vocabulary during tokenizer training but almost never appeared in the actual training corpus. The model had never learned what these tokens meant, so prompting with them produced bizarre hallucinations. The lesson: your tokenizer and training corpus must be aligned. Karpathy's tokenizer video covers this in detail.

(e) Quiz + exercise · 10 min

1. Explain in one sentence what a tokenizer does.
2. Why did the industry converge on subword tokenization instead of word-level or character-level?
3. What is the core algorithm of BPE, in three steps?
4. Name two reasons the same sentence in Hindi typically produces more tokens than in English.
5. Why is a tokenizer effectively frozen once a model is trained?

Stretch (connects to S104 — Embeddings): For a given LLM, the token embedding table is (vocab_size × hidden_dim). Given vocab_size=50257 and hidden_dim=768 (GPT-2 small), how many parameters is that, and what fraction of the model's ~124M total is it?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Tokenization? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S107): Attention Intuition — Why RNNs Failed, Why Attention Won
• Previous (S105): Transfer Learning & Fine-Tuning Classical DL
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.