DL S045 · SentencePiece and Modern Tokenizers
Why Llama uses SentencePiece unigram, not BPE. Understand the unigram LM training algorithm, the tokenizers library ecosystem, and which tokenization scheme to pick for your project.
🎯 Understand the SentencePiece unigram algorithm well enough to know when to pick it over BPE, and be able to train + deploy either using the tokenizers library.
Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 45 / 80 · Module M08 · ~1.7 hours
The story
BPE is the tokenizer we hear about most. It's not the tokenizer Llama uses. Llama 1/2/3, T5, and many multilingual models use SentencePiece with a unigram language model algorithm. Different math, different tradeoffs, better multilingual handling, worse code handling.
This session unpacks why. We cover the unigram LM training algorithm (which optimizes a proper probabilistic objective, unlike BPE's greedy heuristic), the SentencePiece library's practical advantages (whitespace-as-a-symbol, reversible tokenization, single training config), and how the Hugging Face tokenizers library gives you both BPE and unigram behind one API. By the end you'll know which tokenizer to pick for a new project and how to train it in ~20 lines.
- Explain SentencePiece's unigram LM training algorithm in outline (initialize huge vocab, EM to score tokens, prune, repeat).
- Contrast BPE (greedy merge, deterministic encoding) with unigram (probabilistic scoring, multiple valid tokenizations).
- Explain what 'reversible' tokenization means and why SentencePiece handles whitespace as ' ▁'.
- Choose between BPE and unigram for a new project given constraints (multilingual? code-heavy? small vocab? big?).
- Train a tokenizer using the huggingface tokenizers library in ~20 lines.
- Recognize a few tokenization-related bugs and their fingerprints.
Prerequisites
- S044 — BPE, which unigram is compared against throughout.
1 · The two things called 'SentencePiece'
Confusing naming. SentencePiece is a library from Kudo/Google that implements TWO tokenization algorithms:
- SentencePiece BPE — the same BPE algorithm from S044, but wrapped in SentencePiece's whitespace-as-symbol convention.
- SentencePiece Unigram — a different algorithm based on a unigram language model.
When people say "Llama uses SentencePiece", they mean SentencePiece with the BPE algorithm. When people say "T5 uses SentencePiece", they usually mean SentencePiece unigram. Very easy to talk past each other. Ask which.
Today we focus on the unigram LM algorithm because that's the mathematically distinct thing — the "SentencePiece BPE" is essentially GPT-style BPE with different plumbing.
2 · Unigram LM tokenization — the algorithm
The unigram LM tokenizer assumes each token has an independent probability, and the probability of a sentence is the product of its tokens' probabilities:
Training goal: find a vocabulary and per-token probabilities that maximize the total corpus probability.
Concretely (rough sketch, real algorithm is iterative EM):
- Initialize with a HUGE candidate vocabulary — every substring up to some length that appears in the corpus at least once. Millions of candidates.
- E-step: for each word in the corpus, find its most probable segmentation given current token probabilities (Viterbi over the possible segmentations).
- M-step: re-estimate token probabilities from the segmentations produced in the E-step.
- Prune: remove ~10% of the least-useful tokens (those whose removal barely hurts the corpus log-likelihood).
- Repeat E/M/prune until vocab size reaches target.
Result: a vocab (~32k tokens) AND a per-token log-probability. Encoding a new string is a Viterbi over possible segmentations weighted by the learned probabilities.
3 · BPE vs unigram — the head-to-head
- Determinism: BPE encodes any string to exactly one token sequence. Unigram has multiple valid options; default picks the Viterbi-best.
- Training objective: BPE is greedy heuristic (locally optimal pair). Unigram is probabilistic (globally optimal vocab under a unigram-LM likelihood).
- Rare-word handling: BPE tends to split rare words into character-level fragments. Unigram tends to keep slightly longer subword chunks.
- Code handling: BPE (with GPT-2's regex pre-tokenizer) handles code better — punctuation and identifiers get consistent boundaries. Unigram is worse on code because whitespace-based Viterbi doesn't respect syntactic boundaries.
- Multilingual: unigram is typically better for high-resource multilingual corpora (T5, mT5) — smoother probability distribution across scripts.
- Trainable vocab size: both scale to ~256k. Unigram training is 2-5× slower than BPE at large vocab.
4 · SentencePiece's whitespace trick
SentencePiece treats whitespace as a first-class character by replacing it with ▁ (U+2581) before tokenization. This makes tokenization reversible — you can concatenate the tokens back to the original string without ambiguity.
Detokenize by joining tokens and replacing ▁ with a space. No language-specific whitespace handling needed.
Compare to Hugging Face's byte-level BPE (used in GPT-2), where the same idea is achieved by using Ġ for leading-space bytes. Different symbol, same principle.
5 · Training a tokenizer with Hugging Face tokenizers
Modern practice: don't hand-roll. Use tokenizers (the fast Rust-backed library from Hugging Face). Handles all four algorithms (BPE, unigram, WordPiece, char) with a unified API.
Swap models.Unigram() for models.BPE() and trainers.UnigramTrainer for trainers.BpeTrainer and you have a BPE tokenizer with the same API. Twenty lines total.
Grab any 5MB text file (project Gutenberg book, English Wikipedia dump, your own writing). Train two tokenizers with vocab_size=8000 — one models.BPE(), one models.Unigram(). Then encode a fixed test paragraph with each and compare len(enc.tokens) / word_count. Typical result on English prose: BPE lands around 1.3 tokens/word, Unigram around 1.25. Now encode the same paragraph five times through Unigram with subword-regularization sampling enabled (tokenizer.encode(text, sampling=True)) and observe: you get different tokenisations each time. That's the property BPE fundamentally cannot offer.
6 · Picking a tokenizer for a new project
Decision tree:
- English only, code-heavy (documentation, coding assistant) → BPE (GPT-2 style with regex pre-tokenizer). Handles punctuation and code identifiers well.
- English only, prose-heavy (chatbot, general LLM) → either BPE or unigram is fine. Modern LLMs (GPT-4, Claude, Llama) all use BPE variants.
- Highly multilingual (100+ languages) → unigram (T5, XLM-R, Bloom). Smoother handling of scripts with different byte distributions.
- Domain-specific (medical, legal, scientific) → BPE trained on your domain corpus for best coverage of jargon.
- Prefer determinism (production serving) → BPE. Unigram's multiple-valid-tokenizations is nice for training regularization, annoying for reproducibility.
7 · Tokenization budget — how many tokens per language?
- GPT-4 tokenizer on English: ~0.75 tokens per word (1 word ≈ 1.3 tokens).
- GPT-4 tokenizer on Chinese: ~1-2 tokens per Chinese character.
- GPT-4 tokenizer on French, German: ~1.5-2 tokens per word (worse than English).
- Llama-2 tokenizer on English: ~1 token per word (slightly worse than GPT-4).
- Gemma 3 tokenizer (256k vocab) on any language: ~0.5-0.8 tokens per word — big vocab wins.
Practical implication: cost per character of an API call VARIES 3-5× across languages. English is cheap; non-Latin scripts are expensive. This is starting to change with 256k-vocab models but is still true across most APIs in 2024.
8 · Pitfalls
Loading a Llama tokenizer (SP-BPE) with HuggingFace GPT-2 tokenizer code will not error but will produce wrong tokens. Different whitespace conventions (▁ vs Ġ), different merge orders, different byte remappings. Always use AutoTokenizer.from_pretrained(model_id) — never assume tokenizers are interchangeable.
Sometimes fine-tuning datasets get retokenized with a slightly different tokenizer config (different unicode normalization, different special tokens). Loss will plateau at a suspiciously high value because embeddings for changed tokens are now random. Always verify: run a few strings through both, check the token IDs match exactly.
For English, len(text) / 4 is a rough estimate. For code, closer to len(text) / 3. For Chinese, closer to len(text) / 1.5. For emoji-heavy text, could be len(text) (every emoji = 3-4 bytes = 3-4 tokens with byte-level BPE). Always tokenize before enforcing a token budget.
<|endoftext|>, <|im_start|>, <pad> are meant to be structural. If a user's input contains these strings as raw text and your prompt template doesn't escape them, the model may see them as boundary markers and behave weirdly (e.g., truncate generation at a stray <|endoftext|> in user text). Sanitize user input against your special-token list before templating.
9 · The end of tokenization?
There's active research on byte-level models (no tokenization at all — feed raw UTF-8 bytes) and latent-space tokenization. Meta's MegaByte and 2024's Mamba variants have shown byte-level can work if you use a hierarchical model or a state-space model that handles long sequences cheaply.
The pitch: no tokenization means no tokenization pathologies (§S044). But byte-level sequences are 4-5× longer than BPE for English, which explodes attention costs unless you have a sub-quadratic architecture.
For the next few years, BPE / unigram will dominate. But by 2028, "we don't have tokenization" is a plausible frontier feature.
Retention scaffold
Recall questions
1. What algorithm does 'SentencePiece Unigram' train with?
An EM-style loop: initialize a huge candidate vocab, use Viterbi to find the most probable segmentation per word, re-estimate token probabilities, prune the least-useful ~10% of tokens, repeat until vocab size reached.
2. Give one advantage of unigram over BPE.
Any of: probabilistic training objective (vs greedy heuristic); multiple valid segmentations enable subword regularization; better on high-resource multilingual corpora; typically slightly longer subword chunks for rare words.
3. What does the ▁ character represent in SentencePiece?
A leading space (whitespace-as-first-class-symbol). Enables reversible tokenization — you can join tokens and swap ▁ for space to reconstruct the original.
4. Why is GPT-4's English tokenization cheaper per character than its Chinese tokenization?
BPE learns tokens for frequent English word-pieces during training. Chinese characters get 1-2 tokens each (byte-level), so per-character cost is higher. Big vocabs (Gemma 3's 256k) narrow this gap.
5. Name a scenario where you'd pick BPE over unigram.
Code-heavy or English-only prose with a strong preference for determinism/reproducibility. Also: matching an existing pretrained model (GPT-2/3/4, Llama) where the tokenizer is fixed.
Modern picks (2024–2025) — who ships what, and why
A quick decision cheatsheet based on what actually shipped in 2024–2025:
| Model | Tokenizer | Vocab | Why they chose it |
|---|---|---|---|
| GPT-4 / 4o | cl100k_base / o200k_base BPE | 100k / 200k | continuity with GPT-3.5, English-optimized |
| Llama 3 / 3.1 | tiktoken-style BPE (byte-level) | 128k | Meta switched away from SentencePiece unigram at Llama 3 — cited inference-latency wins and better code handling |
| DeepSeek-V3 | BPE | 129k | Chinese-heavy pretraining corpus, deep pretokenization regex |
| Mistral 7B / Mixtral | SentencePiece BPE | 32k | inherited from Llama-1 lineage |
| Mistral Nemo (Tekken) | Byte-level BPE (custom) | 131k | new 2024 tokenizer, ~30% better on non-English + code |
| Llama 1 / 2 | SentencePiece unigram | 32k | multilingual reach |
| Gemma 3 | SentencePiece unigram | 256k | Google's T5/mT5 lineage; multilingual + tool tokens |
| T5 / mT5 / PaLM | SentencePiece unigram | 32k / 256k | Google's default |
| Qwen 2.5 | BPE (byte-level) | 152k | Chinese + English + code, Alibaba tuned |
The 2024 story: BPE won at the frontier, unigram won at Google. Meta's Llama-3 switch from unigram → BPE is the loudest signal that inference-time determinism and code quality matter more than multilingual elegance for most Western deployments. Google keeps unigram alive because their T5 stack, mT5 evals, and PaLM lineage are all built on it.
And then there's the 2024 curveball: tokenmonster (Forsythe, github.com/alasdairforsythe/tokenmonster). It uses a completely different training loss — it optimizes vocabulary directly against a language-model objective, not against pair-count greed (BPE) or corpus likelihood (unigram). On matched vocab sizes it beats both by 5–15% compression on English + code. No frontier lab has adopted it (switching tokenizers on an existing model is expensive), but it's the strongest evidence yet that both BPE and unigram are local optima, not global ones.
A colleague at a big lab wanted to swap Llama 2's 32k SentencePiece unigram for a fresh 128k byte-level BPE (better on their Hindi + code fine-tuning data). Sounds simple. It isn't. Every learned embedding, every LM-head weight, every KV-cache assumption in production serving is pinned to the old vocab. The migration path is: freeze base model, add new tokens as extra embedding rows initialized from a mean of related old-token embeddings, train briefly to "warm up" the new rows, then continue fine-tuning. Even so, quality on already-tokenized old inputs degrades ~1–2 points. The generalized lesson: tokenizer is a production commitment. Pick early, get it right, don't revisit unless you're pretraining a fresh model.
Stretch prompt
Train both BPE and unigram tokenizers on the same 10 MB corpus with vocab_size=8000. Tokenize the same held-out paragraph with each. Compare: (a) total token count, (b) average token length in chars, (c) tokens that DIFFER between them. You'll typically find unigram produces slightly fewer tokens (better compression) but BPE has more predictable boundaries.
In your own words
"If a colleague asks 'why does Llama use SentencePiece instead of GPT-2's BPE?', what's my 3-sentence answer?"
Spaced review
- S044 — BPE, half the story.
- S041 — char-level tokenizer, the trivial baseline.
Next-session teaser
M08 continues into pretraining data curation. S046 shows how you'd build a SlimPajama-style 1B-token pretraining dataset from Common Crawl. Real numbers, real filters, and the deduplication step that keeps your model from memorizing the same Wikipedia article 500 times.
Bring back tomorrow
- SentencePiece is a library; can wrap either BPE or unigram.
- Unigram: probabilistic EM training, multiple valid tokenizations, better multilingual.
- BPE: greedy merges, deterministic, better on code.
- Tokenizer choice = production constraint. Pick early, commit fully, never mix.