The Curious 80 / 10 / 10 Split
BERT masks about 15% of tokens, but does not always insert a literal [MASK]. Of the chosen positions:
- 80% become [MASK] — the standard fill-in-the-blank case.
- 10% are replaced by a random word — forcing the model to notice when a token does not fit, rather than trusting whatever it sees.
- 10% are left unchanged — but still predicted, so the model keeps a useful representation of every token, not just masked ones.
The reason is a train/test mismatch: [MASK] never appears during fine-tuning or inference. If every masked slot were a literal mask token, the model would learn features that only activate on a symbol it will never see again.
Masking Rate Is a Trade-off
Mask too little and each sentence teaches the model almost nothing — training is slow because most positions carry no loss. Mask too much and you destroy the very context needed to make a prediction. Around 15% is the long-standing sweet spot, though later work has shown much higher rates can work with large enough models. Drag the rate slider and watch the sentence degrade: at 50% the remaining context often cannot pin the answer down.
Filling in the blanks
Masked language modelling hides some tokens and trains the model to recover them from both sides of the gap.
"The cat sat on the [MASK]" → predict "mat"
The difference from causal modelling is the direction of information flow. Here the model sees everything except the masked positions, so a token can be predicted from the words after it as well as before. That bidirectional context is what makes the resulting representations strong for understanding tasks — and it is exactly why the model cannot generate, since there is no notion of "next".
BERT's specific recipe masks 15% of tokens, and the composition is deliberate:
| Treatment | Share of masked tokens |
|---|
Replaced with [MASK] | 80% |
| Replaced with a random token | 10% |
| Left unchanged | 10% |
The last two look odd and solve a real problem. [MASK] never appears at fine-tuning or inference time, so a model trained only on it would be mismatched with how it is used. Including corrupted and unchanged tokens forces the model to build a good representation of every position, not just the masked ones.
Why 15%, and the efficiency cost
Too little masking and each pass teaches almost nothing. Too much and there is not enough context left to make prediction possible.
15% became the convention, and later work suggests the optimum is task- and scale-dependent — some studies find 40% works better for larger models.
There is a real efficiency consequence either way:
| | Masked LM | Causal LM |
|---|
| Predictions per forward pass | 15% of tokens | 100% of tokens |
| Sees | Both directions | Past only |
| Can generate | No | Yes |
A causal model extracts a training signal from every position in one pass; a masked model from roughly one in seven. That is a substantial difference in training efficiency, and it is one of several reasons the field shifted towards causal modelling for large-scale pretraining.
ELECTRA addressed it directly with a different objective — a small generator replaces tokens, and the main model classifies every position as original or replaced, giving a signal at all positions.
What it produces, and what it is for
The output of masked pretraining is a model whose hidden states are excellent contextual representations. That is the product; the mask-filling task was only the means.
Those representations are what get used:
- Embeddings for search, similarity and clustering — particularly after contrastive fine-tuning, as in Sentence-BERT.
- Token classification — named entities, part of speech, where bidirectional context genuinely helps.
- Sentence classification — sentiment, intent, topic, via the
[CLS] position or pooled states. - Span extraction — extractive question answering, predicting start and end positions.
For all of these, an encoder fine-tuned on a few thousand labelled examples is often as accurate as prompting a much larger model, and orders of magnitude cheaper to run. A DistilBERT classifier answers in milliseconds on a CPU.
Exploration guide
- Mask a single strongly-constrained token such as mat. The top candidate should dominate — the surrounding words leave little freedom.
- Now mask an adjective like soft. Confidence spreads across several plausible words: the context permits a whole family of answers.
- Load the "bank / river" preset and mask bank. The words after it are what disambiguate the sense — exactly the information a left-to-right model would not yet have.
- Mask two adjacent tokens. Accuracy drops sharply, because MLM predicts each blank independently and cannot coordinate between them — a genuine known limitation of the objective.
About the model in this lab
Predictions come from a small context-scoring model built live in your browser from the demo corpus shown in the vocabulary count — it ranks candidate words by how often they co-occur with the surviving context. It is a genuine calculation, not a canned list, but it is far simpler than a transformer: treat the shape of the results as the lesson, not the exact percentages.
In one line
MLM turns any raw text into supervised training data by deleting parts of it. Because the blank is surrounded rather than trailing, the model learns deep bidirectional understanding — ideal for classification, retrieval and question answering, and unsuitable for generation. That generative job belongs to causal models, which predict strictly forward.
The variants that improved on BERT
| Model | What changed |
|---|
| RoBERTa | More data, longer training, dynamic masking, no next-sentence prediction |
| ALBERT | Parameter sharing across layers, factorised embeddings |
| DeBERTa | Disentangled attention for content and position |
| ELECTRA | Replaced-token detection instead of masking |
| DistilBERT | Distilled from BERT: 40% smaller, 60% faster, ~97% of the quality |
RoBERTa is the instructive one. Same architecture, same objective, and substantially better results — from training longer on more data with larger batches, masking dynamically (a different mask each epoch rather than one fixed at preprocessing), and dropping BERT's next-sentence-prediction objective, which turned out to be unhelpful.
That is a useful general lesson: BERT's original recipe was not optimal, and a large part of the subsequent improvement came from training procedure rather than architecture.
Next-sentence prediction asked whether two segments were adjacent in the original text. It was intended to teach discourse relationships and mostly taught topic matching, which the masked objective already provided.
Masked versus causal, in practice
The split looked permanent for several years — encoders for understanding, decoders for generation. What changed is scale: a large decoder, prompted appropriately, does classification and extraction well enough that the bidirectional advantage stopped being decisive for most applications.
So current practice is roughly:
| Need | Reach for |
|---|
| Embeddings for search or clustering | A masked-pretrained encoder, contrastively fine-tuned |
| Cheap high-volume classification | A fine-tuned small encoder |
| Token-level labelling | An encoder |
| Generation of any kind | A decoder |
| Zero-shot or few-shot on an unknown task | A decoder |
| Lowest possible inference cost | An encoder |
The rows favouring encoders share a shape: a fixed task with labelled data and a cost or latency constraint. That describes a great many production systems, which is why BERT-family models remain very widely deployed despite receiving far less attention.
Practical notes
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")
text = "The capital of France is [MASK]."
ids = tok(text, return_tensors="pt")
logits = model(**ids).logits
pos = (ids.input_ids[0] == tok.mask_token_id).nonzero(as_tuple=True)[0]
top = logits[0, pos].topk(5).indices[0]
print(tok.convert_ids_to_tokens(top)) # ['paris', 'lyon', 'marseille', ...]
Three things worth knowing when fine-tuning one:
Continued pretraining helps on specialised domains. Running the masked objective on your own unlabelled corpus before supervised fine-tuning often gives a measurable gain, and needs no labels.
The [CLS] representation is not a good sentence embedding out of the box — it was not trained for similarity. Use a model fine-tuned contrastively, or masked mean pooling.
512 tokens is the usual limit. Longer documents need chunking, or a long-context variant such as Longformer.
Why 15 percent, and what the 80/10/10 split is really for
Masked language modelling has two numbers in it that look arbitrary -- a 15 percent masking rate and an 80/10/10 split of what to do with each masked token. Both are answers to specific problems, and both are visible in the arithmetic.
Questions people ask
Why can BERT not generate text? It has no notion of "next" — its objective assumes surrounding context exists. Iterative mask-filling works poorly.
Is masked language modelling obsolete? Not for embeddings and cheap classification, where encoders remain the practical choice. For general-purpose models, causal pretraining dominates.
Why the 80/10/10 split? Because [MASK] does not exist at fine-tuning time; the random and unchanged cases prevent the model relying on the token's presence.
Should I continue pretraining on my domain? If you have a substantial unlabelled domain corpus, usually yes — it is cheap and often helps.
What is [CLS] for? A position whose final representation is used as a sequence summary for classification.
Can I use a decoder for embeddings? Yes, by pooling hidden states, and purpose-trained embedding models generally do better because they are trained with a similarity objective.
Recap in one screen
- Hide about 15% of tokens and predict them from both directions; the representations are the real product.
- The 80/10/10 masking split exists because
[MASK] never appears at fine-tuning time. - Bidirectional context is why encoders excel at understanding and cannot generate.
- Causal modelling trains on every position per pass, which is a large efficiency advantage at scale.
- Encoders remain the right tool for embeddings, token labelling and cheap high-volume classification.