The Algorithm, in Four Lines
- Split every word into characters, marking the word end (shown here as </w>).
- Count every adjacent symbol pair across the whole corpus.
- Merge the single most frequent pair everywhere it occurs, and record that merge.
- Repeat until you have the number of merges you asked for.
That is the entire method. Vocabulary size is simply base characters plus the number of merges — which is why it is an exact dial rather than something you discover after the fact.
Merge Order Is the Model
The merge list is ordered, and that order matters at encoding time: to tokenize a new word you replay the merges from rank 1 downward, applying each wherever it fits. The "How it got there" panel shows this replay step by step — the word starts as loose characters and progressively fuses.
Notice that the earliest merges are almost always extremely common fragments (e+s, t+h), while later merges assemble whole frequent words. The vocabulary discovers morphology — prefixes, suffixes, stems — without ever being told that such things exist.
Building a vocabulary by merging
Byte-pair encoding starts from individual characters and repeatedly merges the most frequent adjacent pair, recording each merge as a new vocabulary entry.
Worked through on a tiny corpus — the words "low", "lower", "newest", "widest", with the counts they appear:
- Start with characters:
l o w e r n s t i d - The most frequent adjacent pair is
e + s → merge into es - Next most frequent:
es + t → est - Next:
l + o → lo - Next:
lo + w → low
Continue until the vocabulary reaches the target size — 30,000 or 100,000 entries in practice. The merges are recorded in order, and applying them in that same order is how new text is tokenised.
The outcome is a vocabulary that spends its budget where the text actually is. Frequent words become single tokens; rare words remain as fragments; and because the individual characters (or bytes) are always in the vocabulary, nothing is ever unrepresentable.
That last property is why the <UNK> token has essentially disappeared from modern models.
What it produces on real text
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2")
tok.tokenize("tokenization") # ['token', 'ization']
tok.tokenize("Tokenization") # ['Token', 'ization'] -- case matters
tok.tokenize("antidisestablishmentarianism")
# ['ant', 'id', 'ises', 't', 'ablish', 'ment', 'arian', 'ism']
tok.tokenize(" hello") # ['Ġhello'] -- Ġ marks a leading space
Three observations from those lines.
Common words are single tokens; long or rare ones fragment into pieces that were frequent in the training corpus.
Leading spaces are part of the token. GPT-style tokenisers treat " hello" and "hello" as different tokens, which is why a prompt ending in a space can change the output noticeably.
Case matters. "Token" and "token" are different entries, so capitalisation consumes vocabulary and affects tokenisation.
Byte-level BPE, and why it matters
Plain BPE over characters has a gap: a character never seen in training cannot be represented. With Unicode's 149,000+ code points, that gap is real.
Byte-level BPE operates on UTF-8 bytes rather than characters. There are only 256 possible bytes, all of which are in the base vocabulary, so any text in any script — plus emoji, plus binary noise — is representable.
The cost is efficiency for non-Latin scripts. A Chinese character is three UTF-8 bytes, so it costs at least one token and often more; the same sentence in Chinese and in English can differ severalfold in token count.
| Text | Approximate tokens |
|---|
| English prose | ~1.3 per word |
| Code | ~2 per word, more with symbols |
| Chinese / Japanese | ~1 per character |
| Cyrillic, Arabic | ~2–3 per word |
| Emoji | 1–4 each |
That table is a real cost and latency difference when calling an API, and it is why multilingual models often use larger vocabularies to compensate.
Learning a vocabulary by merging the commonest pair
BPE starts from single characters and repeatedly merges whichever adjacent pair is commonest, until the vocabulary is the size you asked for. This trains one on a small corpus, tokenises with it, and shows why token counts are the thing that actually costs you money.
Experiments to try
- Set merges to 0. Every word shatters into individual characters — the tokens-per-word figure is at its worst.
- Drag the slider up slowly and watch "Tokens / Word" fall. Each merge buys compression, with the earliest merges buying the most.
- Tokenize lowest after training on the repetitive sample. It typically splits into a stem plus est — a suffix the algorithm found purely from frequency.
- Now type a word the corpus has never seen, such as zyxwv. It falls back to characters instead of failing — this graceful degradation is BPE's most important property.
- Switch to the code-like sample. Different merges appear entirely, because the vocabulary is a direct fingerprint of the training data — which is exactly why a model tokenizes unfamiliar domains inefficiently.
Where that leaves you
BPE is a compression algorithm repurposed as a vocabulary builder: merge what is frequent, leave the rest in pieces. This lab runs the genuine training loop on whatever corpus you paste in — the merge table you see is the actual product of counting pairs, not a stored example. Real tokenizers add byte-level fallbacks and pre-tokenization rules, but the core loop is exactly this.
The variants
| Algorithm | Merge criterion | Used by |
|---|
| BPE | Most frequent adjacent pair | GPT-2, GPT-3, Llama |
| WordPiece | Pair that most increases corpus likelihood | BERT, DistilBERT |
| Unigram | Start large, prune the least useful tokens | T5, XLNet, mBART |
| SentencePiece | A wrapper implementing BPE or Unigram on raw text | T5, Llama, many multilingual models |
WordPiece differs from BPE only in the merge criterion: rather than raw frequency, it picks the merge that most improves the likelihood of the corpus under a unigram model. In practice the vocabularies are similar. It marks continuations with ## — "token", "##ization".
Unigram works backwards: begin with a large candidate vocabulary and iteratively remove the tokens whose loss hurts least. It can produce several valid segmentations of the same word with probabilities, which enables subword regularisation during training.
SentencePiece is worth understanding as a different kind of thing — not an algorithm but an implementation that treats input as a raw byte stream with no pre-tokenisation on whitespace. That matters for languages that do not separate words with spaces, and it makes detokenisation exactly reversible.
What tokenisation decides for the model
Several well-known model behaviours trace directly back to this step.
Poor spelling and character counting. "strawberry" may be two or three tokens, none of which is a letter. Asking how many r's it contains asks the model to reason about something it cannot see.
Inconsistent arithmetic. "1234" might be one token or several depending on the tokeniser and the surrounding text, so digits are not consistently represented. Some newer models tokenise digits individually to improve this.
Context limits in tokens, not words. A 128,000-token window is roughly 96,000 English words, and far fewer for code or other scripts.
Sensitivity to trailing spaces. Because the space is part of the token, a prompt ending with a space tokenises differently from one that does not.
The tokeniser must ship with the model. Token id 1,547 means one thing in one vocabulary and something else in another. A mismatch produces fluent nonsense rather than an error.
Questions people ask
How large should the vocabulary be? 30,000–50,000 for a single language; 100,000–250,000 for multilingual models. Larger means shorter sequences and a bigger embedding table.
Can I train my own tokeniser? Yes, with tokenizers or SentencePiece — worth it for a specialised domain or language, and it means the model must be trained or substantially adapted with it.
Why does my word split oddly? Because those fragments were the frequent merges in the training corpus. It looks arbitrary and is entirely determined by corpus statistics.
Is BPE reversible? Yes — concatenating the tokens reconstructs the original text exactly, which is why byte-level variants are careful about spaces.
Do I need to lowercase first? No, for a cased tokeniser. It was trained on cased text and lowercasing loses information.
How do I count tokens before an API call? With the model's own tokeniser — tiktoken for OpenAI models, the transformers tokeniser otherwise. Estimating from word count is unreliable.
Recap in one screen
- BPE starts from characters and repeatedly merges the most frequent adjacent pair until the vocabulary is full.
- Frequent words become single tokens; rare ones fragment; characters remain as a fallback, so nothing is unknown.
- Byte-level BPE works on UTF-8 bytes, so any text is representable — at more tokens for non-Latin scripts.
- WordPiece, Unigram and SentencePiece are variations on the same idea with different merge or pruning criteria.
- Tokenisation explains poor spelling, inconsistent arithmetic, token-based context limits and trailing-space sensitivity.