mlintermediate 100m read

DL S045 · SentencePiece and Modern Tokenizers

Why Llama uses SentencePiece unigram, not BPE. Understand the unigram LM training algorithm, the tokenizers library ecosystem, and which tokenization scheme to pick for your project.

🧠SoftwareM08 · Tokenization + data + scaling laws· Session 045 of 130 100 min

🎯 Understand the SentencePiece unigram algorithm well enough to know when to pick it over BPE, and be able to train + deploy either using the tokenizers library.

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 45 / 80 · Module M08 · ~1.7 hours

The story

BPE is the tokenizer we hear about most. It's not the tokenizer Llama uses. Llama 1/2/3, T5, and many multilingual models use SentencePiece with a unigram language model algorithm. Different math, different tradeoffs, better multilingual handling, worse code handling.

This session unpacks why. We cover the unigram LM training algorithm (which optimizes a proper probabilistic objective, unlike BPE's greedy heuristic), the SentencePiece library's practical advantages (whitespace-as-a-symbol, reversible tokenization, single training config), and how the Hugging Face tokenizers library gives you both BPE and unigram behind one API. By the end you'll know which tokenizer to pick for a new project and how to train it in ~20 lines.

You will be able to

Explain SentencePiece's unigram LM training algorithm in outline (initialize huge vocab, EM to score tokens, prune, repeat).
Contrast BPE (greedy merge, deterministic encoding) with unigram (probabilistic scoring, multiple valid tokenizations).
Explain what 'reversible' tokenization means and why SentencePiece handles whitespace as ' ▁'.
Choose between BPE and unigram for a new project given constraints (multilingual? code-heavy? small vocab? big?).
Train a tokenizer using the huggingface tokenizers library in ~20 lines.
Recognize a few tokenization-related bugs and their fingerprints.

Prerequisites

S044 — BPE, which unigram is compared against throughout.

Pre-read (~30 min)

🔬 Deep dive 30 min

Let's build the GPT Tokenizer (SentencePiece section)

Andrej Karpathy

Skip to the ~1h30m mark where Karpathy switches from BPE to SentencePiece. He compares them live with the same corpus.

💡 Intuition 9 min

Unigram Tokenization

Hugging Face

Nine-minute intro to the unigram algorithm. Concrete, no math scary.

📖 Reading 15 min

SentencePiece: A simple and language independent subword tokenizer

Kudo & Richardson 2018

The library paper. Sections 2 and 3 explain the two modes (BPE and unigram) and the reversibility trick.

📖 Reading 20 min

Subword Regularization: Improving NMT Models with Multiple Subword Candidates

Kudo 2018

The unigram-LM tokenization paper. Explains WHY you'd want multiple tokenizations of the same string (regularization) — a subtle but real advantage over BPE.

1 · The two things called 'SentencePiece'

Confusing naming. SentencePiece is a library from Kudo/Google that implements TWO tokenization algorithms:

SentencePiece BPE — the same BPE algorithm from S044, but wrapped in SentencePiece's whitespace-as-symbol convention.
SentencePiece Unigram — a different algorithm based on a unigram language model.

When people say "Llama uses SentencePiece", they mean SentencePiece with the BPE algorithm. When people say "T5 uses SentencePiece", they usually mean SentencePiece unigram. Very easy to talk past each other. Ask which.

Today we focus on the unigram LM algorithm because that's the mathematically distinct thing — the "SentencePiece BPE" is essentially GPT-style BPE with different plumbing.

BPE grows a vocab, unigram prunes one

🌍 Real world

💻 Code world

2 · Unigram LM tokenization — the algorithm

The unigram LM tokenizer assumes each token has an independent probability, and the probability of a sentence is the product of its tokens' probabilities:

P(\text{tokens}) = \prod_i p(t_i)

Training goal: find a vocabulary and per-token probabilities that maximize the total corpus probability.

BPE is a greedy stenographer; unigram is a probabilistic editor

🌍 Real world

💻 Code world

Concretely (rough sketch, real algorithm is iterative EM):

Initialize with a HUGE candidate vocabulary — every substring up to some length that appears in the corpus at least once. Millions of candidates.
E-step: for each word in the corpus, find its most probable segmentation given current token probabilities (Viterbi over the possible segmentations).
M-step: re-estimate token probabilities from the segmentations produced in the E-step.
Prune: remove ~10% of the least-useful tokens (those whose removal barely hurts the corpus log-likelihood).
Repeat E/M/prune until vocab size reaches target.

Result: a vocab (~32k tokens) AND a per-token log-probability. Encoding a new string is a Viterbi over possible segmentations weighted by the learned probabilities.

3 · BPE vs unigram — the head-to-head

Practical differences

Determinism: BPE encodes any string to exactly one token sequence. Unigram has multiple valid options; default picks the Viterbi-best.
Training objective: BPE is greedy heuristic (locally optimal pair). Unigram is probabilistic (globally optimal vocab under a unigram-LM likelihood).
Rare-word handling: BPE tends to split rare words into character-level fragments. Unigram tends to keep slightly longer subword chunks.
Code handling: BPE (with GPT-2's regex pre-tokenizer) handles code better — punctuation and identifiers get consistent boundaries. Unigram is worse on code because whitespace-based Viterbi doesn't respect syntactic boundaries.
Multilingual: unigram is typically better for high-resource multilingual corpora (T5, mT5) — smoother probability distribution across scripts.
Trainable vocab size: both scale to ~256k. Unigram training is 2-5× slower than BPE at large vocab.

4 · SentencePiece's whitespace trick

SentencePiece treats whitespace as a first-class character by replacing it with ▁ (U+2581) before tokenization. This makes tokenization reversible — you can concatenate the tokens back to the original string without ambiguity.

Detokenize by joining tokens and replacing ▁ with a space. No language-specific whitespace handling needed.

Compare to Hugging Face's byte-level BPE (used in GPT-2), where the same idea is achieved by using Ġ for leading-space bytes. Different symbol, same principle.

5 · Training a tokenizer with Hugging Face `tokenizers`

Modern practice: don't hand-roll. Use tokenizers (the fast Rust-backed library from Hugging Face). Handles all four algorithms (BPE, unigram, WordPiece, char) with a unified API.

Swap models.Unigram() for models.BPE() and trainers.UnigramTrainer for trainers.BpeTrainer and you have a BPE tokenizer with the same API. Twenty lines total.

Try itTrain both a BPE and a Unigram tokenizer on the same 5MB corpus and compare tokens/word.

Grab any 5MB text file (project Gutenberg book, English Wikipedia dump, your own writing). Train two tokenizers with vocab_size=8000 — one models.BPE(), one models.Unigram(). Then encode a fixed test paragraph with each and compare len(enc.tokens) / word_count. Typical result on English prose: BPE lands around 1.3 tokens/word, Unigram around 1.25. Now encode the same paragraph five times through Unigram with subword-regularization sampling enabled (tokenizer.encode(text, sampling=True)) and observe: you get different tokenisations each time. That's the property BPE fundamentally cannot offer.

💡 Hint · Use the code above; keep vocab_size=8000 for speed.

6 · Picking a tokenizer for a new project

Decision tree:

English only, code-heavy (documentation, coding assistant) → BPE (GPT-2 style with regex pre-tokenizer). Handles punctuation and code identifiers well.
English only, prose-heavy (chatbot, general LLM) → either BPE or unigram is fine. Modern LLMs (GPT-4, Claude, Llama) all use BPE variants.
Highly multilingual (100+ languages) → unigram (T5, XLM-R, Bloom). Smoother handling of scripts with different byte distributions.
Domain-specific (medical, legal, scientific) → BPE trained on your domain corpus for best coverage of jargon.
Prefer determinism (production serving) → BPE. Unigram's multiple-valid-tokenizations is nice for training regularization, annoying for reproducibility.

7 · Tokenization budget — how many tokens per language?

Approximate tokens-per-word by tokenizer + language

GPT-4 tokenizer on English: ~0.75 tokens per word (1 word ≈ 1.3 tokens).
GPT-4 tokenizer on Chinese: ~1-2 tokens per Chinese character.
GPT-4 tokenizer on French, German: ~1.5-2 tokens per word (worse than English).
Llama-2 tokenizer on English: ~1 token per word (slightly worse than GPT-4).
Gemma 3 tokenizer (256k vocab) on any language: ~0.5-0.8 tokens per word — big vocab wins.

Practical implication: cost per character of an API call VARIES 3-5× across languages. English is cheap; non-Latin scripts are expensive. This is starting to change with 256k-vocab models but is still true across most APIs in 2024.

8 · Pitfalls

War story Mixing SentencePiece and BPE assumptions

Loading a Llama tokenizer (SP-BPE) with HuggingFace GPT-2 tokenizer code will not error but will produce wrong tokens. Different whitespace conventions (▁ vs Ġ), different merge orders, different byte remappings. Always use AutoTokenizer.from_pretrained(model_id) — never assume tokenizers are interchangeable.

War story Fine-tuning with a different tokenizer than pretraining

Sometimes fine-tuning datasets get retokenized with a slightly different tokenizer config (different unicode normalization, different special tokens). Loss will plateau at a suspiciously high value because embeddings for changed tokens are now random. Always verify: run a few strings through both, check the token IDs match exactly.

War story Trusting `len(text)` as a token count estimate

For English, len(text) / 4 is a rough estimate. For code, closer to len(text) / 3. For Chinese, closer to len(text) / 1.5. For emoji-heavy text, could be len(text) (every emoji = 3-4 bytes = 3-4 tokens with byte-level BPE). Always tokenize before enforcing a token budget.

War story Special tokens leaking through generation

<|endoftext|>, <|im_start|>, <pad> are meant to be structural. If a user's input contains these strings as raw text and your prompt template doesn't escape them, the model may see them as boundary markers and behave weirdly (e.g., truncate generation at a stray <|endoftext|> in user text). Sanitize user input against your special-token list before templating.

9 · The end of tokenization?

There's active research on byte-level models (no tokenization at all — feed raw UTF-8 bytes) and latent-space tokenization. Meta's MegaByte and 2024's Mamba variants have shown byte-level can work if you use a hierarchical model or a state-space model that handles long sequences cheaply.

The pitch: no tokenization means no tokenization pathologies (§S044). But byte-level sequences are 4-5× longer than BPE for English, which explodes attention costs unless you have a sub-quadratic architecture.

For the next few years, BPE / unigram will dominate. But by 2028, "we don't have tokenization" is a plausible frontier feature.

Common misconception

✗ What most people think

"SentencePiece is just a fast implementation of BPE — a library that does the same thing as a hand-rolled BPE trainer, only in C++ and with a nicer interface."

✓ What is actually true

SentencePiece is a different framing of the problem, and its central design decision is that it does not pre-tokenize on whitespace. Standard BPE pipelines split text into words first, then learn merges within words, which quietly builds in the assumption that spaces delimit words — false for Japanese, Chinese, Thai, and awkward for code. SentencePiece treats the raw string as a sequence of characters, encodes the space itself as a visible symbol (the meta-character), and learns over the whole stream. That makes encoding lossless and reversible: decoding is string concatenation with the meta-character mapped back to a space, with no detokenization heuristics guessing where spaces belonged. It also offers unigram as an alternative algorithm to BPE, which is a genuinely different objective, not a faster version of the same one.

Why the myth is so sticky

Because in a normal English pipeline the two really do behave almost identically, and the interfaces look interchangeable — you call train, you get a vocabulary, you encode and decode. The difference is invisible until you hit a case where whitespace is not a reliable delimiter, or where you need the decode to be exactly the input. And the "just a fast library" reading is reinforced by the fact that SentencePiece can run BPE, so you can use it as exactly that and never notice the rest. The belief breaks the first time a round-trip through your tokenizer changes whitespace — a leading space eaten, a double space collapsed — which is catastrophic for code and invisible for prose.

Prove it to yourself

Check losslessness, which is the property the whitespace decision buys:

cases = [
    'hello world',
    '  leading and  double  spaces  ',
    'def f(x):\n    return x  # indented code',
    'no spaces at all in some scripts',
]
for s in cases:
    assert sp.decode(sp.encode(s)) == s, repr(s)

# A whitespace-pretokenizing pipeline fails the second and third
# cases: it must GUESS where spaces go when rejoining, and
# indentation-sensitive text is exactly where guessing is fatal.

From first principles

Start with the question

Why does the unigram algorithm start with a large vocabulary and prune it down, when BPE starts small and grows? Growing seems the natural direction — why go the other way?

1
Unigram defines a probabilistic model: each vocabulary piece has a probability, a segmentation's probability is the product of its pieces, and the objective is the likelihood of the corpus under the best segmentation.
forced by · it is a generative model of text as a sequence of independent pieces, unlike BPE which has no probabilistic model at all
2
Under that objective, the value of a piece depends on what else is in the vocabulary. A piece is useful only insofar as no combination of other pieces covers the same text as cheaply — so utility is defined relative to the whole set, not in isolation.
forced by · the segmentation is a competition among all pieces that could cover a span, so removing one changes the value of others
3
Growing greedily therefore cannot evaluate a candidate correctly: at the moment you consider adding a piece, the pieces that would compete with it may not exist yet, so its measured contribution is against an incomplete field.
forced by · greedy addition commits to a decision using information that will be invalidated by later additions
4
Pruning has the opposite property. Start with a large candidate set — all frequent substrings — so that every piece is evaluated in the presence of all its competitors, and remove the ones whose deletion costs the least likelihood.
forced by · the loss from removing a piece can be computed exactly, since you can re-segment using everything that remains
5
Therefore the direction follows from the objective. BPE's objective is local — the most frequent adjacent pair — so it can be optimised greedily upward. Unigram's objective is global over the vocabulary, so it must be optimised by removing from a superset.
forced by · a criterion defined relative to the full set can only be evaluated when the full set is present

⇒ Therefore

Therefore the growth direction is not an implementation preference — it is forced by whether the objective is local or global.

This predicts a capability BPE structurally cannot have, and it is worth knowing: because unigram assigns probabilities to segmentations rather than producing one deterministically, it can sample alternative segmentations of the same string. That is subword regularisation — training on varied segmentations of identical text as an augmentation, which makes a model more robust to the segmentation it meets at inference. BPE has no distribution over segmentations to sample from, so it needs a bolt-on (dropout on merges) to approximate this. If you ever see a tokenizer used as a source of augmentation, it is a unigram model, and now you know it could not have been otherwise.

Mental modelThe raw stream, spaces included

Do not picture text as a list of words to be broken into pieces. Picture one unbroken character stream in which the space is simply another character — rendered visible as a meta-character so it can be merged into pieces like anything else. Tokens are then substrings of that stream, and a token typically begins with the space that preceded its word.

Decoding is then trivially exact: concatenate the pieces, turn the meta-characters back into spaces, done. No rules about when to put a space before a comma, no language-specific detokenizer, no guessing.

No whitespace pre-tokenization. That is the design decision everything else follows from, and it is what makes the tokenizer language-agnostic and lossless.
The leading space belongs to the token. This is why the same word encodes differently at the start of a line than mid-sentence, and it is a frequent source of confusion when hand-constructing prompts or comparing token counts.
Two algorithms, not one: BPE (greedy frequency merges, grow upward) and unigram (likelihood-based, prune downward). Unigram supports sampling alternative segmentations; BPE does not.
Normalisation runs before tokenization and is part of the model file. Unicode normalisation form, case handling, and whitespace treatment are baked in — so two tokenizers with identical vocabularies can still disagree.

🔔 Fires when you see

Fire this model the moment you see: a decode that does not reproduce the input exactly · leading-space surprises in token counts · tokenizing a language without whitespace word boundaries · code or indentation-sensitive text · subword regularisation or segmentation sampling · a tokenizer that needs a separate detokenizer step.

The tradeoff

BPE or unigram for a new tokenizer?

BPE

+ you gain deterministic and simple to reason about — one segmentation per string, produced by replaying an ordered merge list; fast to train and fast to encode; and it is what the widely-used models use, so tooling, token counters, and community intuition all assume it

− you pay no probabilistic model, so no principled way to sample alternative segmentations for augmentation; and the greedy merge order can lock in early decisions that a global objective would not have made

pick when you want compatibility with existing tooling and mental models, or the tokenizer must be trivially auditable — which makes it the safe default

Unigram

+ you gain a proper probabilistic model over segmentations, enabling subword regularisation, which measurably helps in low-resource and morphologically rich settings where a single fixed segmentation is a brittle commitment; and the global pruning objective tends to produce a cleaner vocabulary with fewer degenerate pieces

− you pay slower and more involved to train, since it requires repeated expectation-maximisation over a large candidate set; nondeterministic segmentation is a complication for caching and for anything that assumes a stable token count; and less community tooling assumes it

pick when you are training multilingual or low-resource models where segmentation robustness matters, or you specifically want subword regularisation as an augmentation

Byte-level fallback with either

+ you gain guarantees that any input is representable, so no character can ever become an unknown token and every encode-decode round trip is exact regardless of what appears at inference time

− you pay unseen characters decompose into raw bytes and cost several tokens each, so coverage is guaranteed but efficiency on out-of-distribution scripts is poor; and it consumes 256 vocabulary slots

pick when always, unless you can prove your inference-time input alphabet is a subset of your training alphabet — and you cannot

What a senior engineer actually does

Use BPE with byte fallback unless you have a specific reason for unigram, and the specific reason is nearly always multilinguality or a desire for subword regularisation. The algorithm choice matters far less than two other things: whether the training corpus for the tokenizer matches your actual data distribution, and whether the pipeline is lossless.

Check losslessness explicitly with a round-trip assertion over awkward inputs — leading spaces, repeated spaces, indented code, mixed scripts — before you commit to any tokenizer. A tokenizer that silently normalises whitespace will not show up in your loss curve, will not show up in prose benchmarks, and will quietly make your model unable to produce correctly indented code. It is the cheapest test on this page and the one most often skipped.

Retention scaffold

SentencePiece / Unigram — click to reveal · click to reveal

★ = stretch question

Recall questions

1. What algorithm does 'SentencePiece Unigram' train with?

An EM-style loop: initialize a huge candidate vocab, use Viterbi to find the most probable segmentation per word, re-estimate token probabilities, prune the least-useful ~10% of tokens, repeat until vocab size reached.

2. Give one advantage of unigram over BPE.

Any of: probabilistic training objective (vs greedy heuristic); multiple valid segmentations enable subword regularization; better on high-resource multilingual corpora; typically slightly longer subword chunks for rare words.

3. What does the ▁ character represent in SentencePiece?

A leading space (whitespace-as-first-class-symbol). Enables reversible tokenization — you can join tokens and swap ▁ for space to reconstruct the original.

4. Why is GPT-4's English tokenization cheaper per character than its Chinese tokenization?

BPE learns tokens for frequent English word-pieces during training. Chinese characters get 1-2 tokens each (byte-level), so per-character cost is higher. Big vocabs (Gemma 3's 256k) narrow this gap.

5. Name a scenario where you'd pick BPE over unigram.

Code-heavy or English-only prose with a strong preference for determinism/reproducibility. Also: matching an existing pretrained model (GPT-2/3/4, Llama) where the tokenizer is fixed.

Modern picks (2024–2025) — who ships what, and why

A quick decision cheatsheet based on what actually shipped in 2024–2025:

Model	Tokenizer	Vocab	Why they chose it
GPT-4 / 4o	`cl100k_base` / `o200k_base` BPE	100k / 200k	continuity with GPT-3.5, English-optimized
Llama 3 / 3.1	tiktoken-style BPE (byte-level)	128k	Meta switched away from SentencePiece unigram at Llama 3 — cited inference-latency wins and better code handling
DeepSeek-V3	BPE	129k	Chinese-heavy pretraining corpus, deep pretokenization regex
Mistral 7B / Mixtral	SentencePiece BPE	32k	inherited from Llama-1 lineage
Mistral Nemo (Tekken)	Byte-level BPE (custom)	131k	new 2024 tokenizer, ~30% better on non-English + code
Llama 1 / 2	SentencePiece unigram	32k	multilingual reach
Gemma 3	SentencePiece unigram	256k	Google's T5/mT5 lineage; multilingual + tool tokens
T5 / mT5 / PaLM	SentencePiece unigram	32k / 256k	Google's default
Qwen 2.5	BPE (byte-level)	152k	Chinese + English + code, Alibaba tuned

The 2024 story: BPE won at the frontier, unigram won at Google. Meta's Llama-3 switch from unigram → BPE is the loudest signal that inference-time determinism and code quality matter more than multilingual elegance for most Western deployments. Google keeps unigram alive because their T5 stack, mT5 evals, and PaLM lineage are all built on it.

And then there's the 2024 curveball: tokenmonster (Forsythe, github.com/alasdairforsythe/tokenmonster). It uses a completely different training loss — it optimizes vocabulary directly against a language-model objective, not against pair-count greed (BPE) or corpus likelihood (unigram). On matched vocab sizes it beats both by 5–15% compression on English + code. No frontier lab has adopted it (switching tokenizers on an existing model is expensive), but it's the strongest evidence yet that both BPE and unigram are local optima, not global ones.

War story When you can NOT change the tokenizer

A colleague at a big lab wanted to swap Llama 2's 32k SentencePiece unigram for a fresh 128k byte-level BPE (better on their Hindi + code fine-tuning data). Sounds simple. It isn't. Every learned embedding, every LM-head weight, every KV-cache assumption in production serving is pinned to the old vocab. The migration path is: freeze base model, add new tokens as extra embedding rows initialized from a mean of related old-token embeddings, train briefly to "warm up" the new rows, then continue fine-tuning. Even so, quality on already-tokenized old inputs degrades ~1–2 points. The generalized lesson: tokenizer is a production commitment. Pick early, get it right, don't revisit unless you're pretraining a fresh model.

Stretch prompt

Train both BPE and unigram tokenizers on the same 10 MB corpus with vocab_size=8000. Tokenize the same held-out paragraph with each. Compare: (a) total token count, (b) average token length in chars, (c) tokens that DIFFER between them. You'll typically find unigram produces slightly fewer tokens (better compression) but BPE has more predictable boundaries.

In your own words

"If a colleague asks 'why does Llama use SentencePiece instead of GPT-2's BPE?', what's my 3-sentence answer?"

Spaced review

S044 — BPE, half the story.
S041 — char-level tokenizer, the trivial baseline.

Next-session teaser

M08 continues into pretraining data curation. S046 shows how you'd build a SlimPajama-style 1B-token pretraining dataset from Common Crawl. Real numbers, real filters, and the deduplication step that keeps your model from memorizing the same Wikipedia article 500 times.

Bring back tomorrow

SentencePiece is a library; can wrap either BPE or unigram.
Unigram: probabilistic EM training, multiple valid tokenizations, better multilingual.
BPE: greedy merges, deterministic, better on code.
Tokenizer choice = production constraint. Pick early, commit fully, never mix.

Previous: ← DL S044 · Next: DL S046 →