Choosing n
| n | Behaviour |
|---|
| 1 | No word order at all; smallest feature set |
| 2 | Captures negation and common phrases; usually worth it |
| 3 | Some benefit, considerably sparser |
| 4+ | Rarely justified for features; used in language models with heavy smoothing |
For classification features, (1, 2) is the standard choice and the marginal gain from trigrams is usually small relative to the extra columns.
For a count-based language model, trigrams to 5-grams with Kneser-Ney smoothing was the state of the art for years, and it needed large corpora to estimate.
The sparsity wall
The counts grow catastrophically. With a 50,000-word vocabulary there are 2.5 billion possible bigrams, 1.25 × 1014 trigrams, and 3 × 1023 5-grams. No corpus covers a meaningful fraction of them.
So most n-grams have a count of zero, and a zero count means a probability of zero — the model declares a perfectly ordinary sentence impossible because it happens not to have seen one four-word span before. Worse, one zero anywhere makes the probability of the whole sentence zero.
The fixes are all forms of smoothing. Add-one (Laplace) smoothing adds a pseudocount to everything, which is simple and crude. Backoff falls back to a shorter n-gram when the longer one is unseen. Kneser-Ney, the best of the classical methods, discounts observed counts and redistributes the mass according to how many distinct contexts a word appears in — capturing that Francisco is common but only ever after San.
Counting sequences of words
An n-gram is a contiguous run of n items. For "the cat sat on the mat":
| n | Name | Grams |
|---|
| 1 | Unigram | the, cat, sat, on, the, mat |
| 2 | Bigram | the cat, cat sat, sat on, on the, the mat |
| 3 | Trigram | the cat sat, cat sat on, sat on the, on the mat |
Why bother beyond unigrams? Because word order carries meaning that a bag of individual words discards. "Not good" and "good not" have identical unigrams; as bigrams they are distinguishable. Sentiment classification improves measurably from bigrams for exactly this reason.
The cost is the vocabulary. With 10,000 distinct words there are up to 100 million possible bigrams and a trillion trigrams. Almost all are absent from any real corpus, so the feature matrix becomes enormous and extremely sparse.
That trade — more context against exponentially more features — is the whole subject.
As a language model
Before neural networks, next-word prediction was done by counting. An n-gram language model estimates:
P(wᵗ | w₁ … wᵗ₋₁) ≈ P(wᵗ | wᵗ₋ₙ₊₁ … wᵗ₋₁)
The approximation is the Markov assumption: only the previous n−1 words matter. For a trigram model, the probability of the next word depends on the two before it and nothing further back.
Estimation is division: count how often "sat on the" was followed by "mat", divide by how often "sat on the" appeared at all.
Two problems arise immediately, and the fixes are classic:
Unseen sequences get probability zero, which zeroes the probability of the whole sentence. Smoothing fixes it — add-one (Laplace) is the simplest, and Kneser-Ney is the one that actually works well, because it accounts for how many different contexts a word appears in.
Longer n means better context and sparser counts. Backoff and interpolation handle it by falling back to shorter n-grams when the longer one has no data.
Why neural models replaced them
Two limitations are structural, not fixable by better smoothing.
No generalisation across similar words. A trigram model that has seen "the black cat" learns nothing about "the dark cat" — the two contexts are entirely separate counts. An embedding-based model places "black" and "dark" near each other, so evidence transfers.
No long-range dependency. A 5-gram model cannot connect a pronoun to a noun seven words earlier. Increasing n makes the counts too sparse to estimate long before the range becomes useful.
Neural language models solved both — distributed representations generalise across words, and recurrence or attention reaches arbitrary distances. That is the whole reason the field moved.
Interactive Exploration Guide
- Start at n = 1. Set the n slider to 1 and read the output. With no context the model produces frequency-ordered nonsense.
- Step up to 2 and 3. Raise n and watch local fluency appear. Pairs and triples look like real language, even though the sentence as a whole still drifts.
- Push to 6. Set n to 6. Now most contexts have been seen once or not at all, so the model either repeats the training text verbatim or has nothing to offer — overfitting and sparsity in the same picture.
- Scroll through the corpus. Use the view slider and watch which sequences have high counts. Almost all the probability mass sits on a small number of common patterns; the tail is enormous and nearly empty.
Traps worth knowing
- No smoothing. Any unseen n-gram makes the whole sentence probability zero. Smoothing is not optional.
- Multiplying raw probabilities. Multiplying hundreds of small numbers underflows to zero in floating point. Sum log-probabilities instead.
- Raising n to fix quality. It usually makes things worse by increasing sparsity. Better smoothing beats a larger n.
- Forgetting sentence boundaries. Without explicit start and end tokens the model cannot represent which words begin or end a sentence.
In one line
An n-gram model estimates the next word by counting how often each continuation followed the previous n−1 words, which makes it fast, interpretable, and trainable in a single pass. It is defeated by sparsity: the number of possible n-grams grows exponentially with n, so most sequences are never observed and smoothing is mandatory. Neural language models replaced it precisely because embeddings let them generalise across similar contexts rather than requiring each one to have been seen.
Where n-grams are still used
They have not disappeared, because counting is cheap, interpretable and needs no training:
- Features for classical classifiers. TF-IDF over unigrams and bigrams, plus logistic regression, remains a strong baseline for text classification — and it trains in seconds.
- Character n-grams for language identification, authorship attribution and handling misspellings. Character 3-grams are remarkably effective at identifying a language from a short string.
- Autocomplete and query suggestion, where a count-based model over query logs is fast and adequate.
- Spelling correction and fuzzy matching, using character n-gram overlap.
- Evaluation metrics. BLEU and ROUGE, still standard for translation and summarisation, are n-gram overlap measures.
- Plagiarism and duplicate detection, via shared n-gram fingerprints.
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer(ngram_range=(1, 2), # unigrams and bigrams
min_df=3, # drop very rare grams
max_features=50_000,
sublinear_tf=True)
X = vec.fit_transform(docs)
min_df and max_features are what keep this tractable — without them, bigrams alone can produce millions of columns, almost all appearing once.
For character n-grams, analyzer="char_wb", ngram_range=(3, 5) is a strong setting for noisy or multilingual text.
Count, predict, and watch it fall apart
An n-gram model is a table of counts. Building one shows both why it works at all and the two walls it hits -- sparsity and no generalisation.
Questions people ask
Are n-grams obsolete? As language models, largely. As features, as evaluation metrics and for character-level tasks, no.
Why does adding bigrams help sentiment analysis? Because "not good" is the signal, and unigrams cannot represent it.
What is the Markov assumption? That only the last n−1 items matter. It is false and useful.
How do I handle unseen n-grams? Smoothing and backoff for a language model; min_df and a fixed vocabulary for features.
Should I use word or character n-grams? Words for clean text in a known language; characters for noisy text, misspellings, multiple languages and short strings.
Is BLEU an n-gram metric? Yes — it measures overlap of n-grams up to 4 between a candidate and reference, with a brevity penalty.
Recap in one screen
- An n-gram is a run of n consecutive items; going beyond unigrams reintroduces word order.
- Feature count grows explosively with n, so
min_df and a capped vocabulary are essential. - As a language model, n-grams assume only the last n−1 words matter, and need smoothing for unseen sequences.
- They cannot generalise across similar words or reach long distances — which is why neural models replaced them.
- Still used for classical features, character-level tasks, autocomplete, and BLEU/ROUGE evaluation.