Stemming: The Fast and Crude Approach
Stemming is a process that reduces words to their "stem" or root form by chopping off prefixes and suffixes. It uses a set of simple, rule-based heuristics and does not care if the resulting stem is a real dictionary word.
- Method: Algorithmic, rule-based (e.g., remove "ing", "ed", "s").
- Speed: Very fast, as it doesn't need to look up words in a dictionary.
- Result: Often produces non-words. For example, "studies" might become "studi".
- Use Case: Ideal for applications where speed is critical and perfect accuracy isn't necessary, such as search engine indexing or large-scale text analysis.
Lemmatization: The Smart and Accurate Approach
Lemmatization is a more sophisticated process that aims to find the true root form of a word, known as the lemma. It uses a dictionary and morphological analysis to consider the context of the word and its part of speech.
- Method: Dictionary-based, considers the word's meaning and part of speech.
- Speed: Slower than stemming because it involves dictionary lookups.
- Result: Always returns a valid dictionary word. For example, "better" is correctly identified as having the lemma "good", and "are" becomes "be".
- Use Case: Best for applications requiring high accuracy, such as chatbots, question-answering systems, and machine translation, where understanding the correct meaning of words is essential.
"Run", "runs", "running" and "ran" are four strings and one concept. To a bag-of-words model they are four unrelated features, each with a quarter of the evidence. Reducing them to a common form pools that evidence.
There are two methods, and they differ in how much they know.
Stemming chops suffixes according to rules. Fast, crude, and it produces strings that are not always words.
Lemmatisation looks the word up, using its part of speech, and returns the real dictionary form.
| Word | Porter stemmer | Lemmatiser |
|---|
| running | run | run |
| runs | run | run |
| ran | ran | run |
| studies | studi | study |
| better | better | good |
| was | wa | be |
| caring | care | care |
Two rows show the difference clearly. "Ran" and "better" need knowledge, not rules — a stemmer has no way to know that "ran" relates to "run" or that "better" is a form of "good". And "studi" and "wa" show what rule-based chopping produces: fragments that work as index keys and read as errors.
When each is right
Stemming is appropriate when the output is a lookup key nobody reads. Search indexing is the canonical case: if "studi" is the key for both "study" and "studies", a query for either matches, and the user never sees the stem.
Lemmatisation is appropriate when the output is shown to someone, or when correctness matters — topic modelling where you display the top terms, linguistic analysis, anything with an audit trail.
| | Stemming | Lemmatisation |
|---|
| Method | Suffix rules | Dictionary plus part of speech |
| Output | Sometimes not a word | Always a real word |
| Speed | Very fast | Slower — needs POS tagging |
| Handles irregulars | No | Yes |
| Needs language resources | No | Yes, per language |
from nltk.stem import PorterStemmer, WordNetLemmatizer
PorterStemmer().stem("studies") # 'studi'
WordNetLemmatizer().lemmatize("studies") # 'study'
WordNetLemmatizer().lemmatize("better", pos="a") # 'good'
That last line matters: without the part of speech, a lemmatiser assumes noun and gets adjectives and verbs wrong. lemmatize("running") returns "running" unless you tell it the word is a verb. In practice, run a POS tagger first — spaCy does this automatically.
Both techniques exist to solve a problem transformers do not have.
A subword tokeniser already splits "running" into "run" + "ning", so the shared stem is represented and the suffix is available as separate information. The model sees both the root and the inflection, and it can use the inflection — tense, number and aspect all carry meaning.
Stemming before a transformer therefore does two harmful things: it destroys grammatical information the model was trained to use, and it creates strings ("studi", "wa") that are not in the vocabulary and tokenise into odd fragments.
The rule is simple. Stem or lemmatise for classical bag-of-words models. Never for transformers.
Chopping suffixes against looking words up
A stemmer applies rules and can produce nonsense. A lemmatiser consults a dictionary and needs to know the part of speech. Both are implemented here on the same words, including the cases where each one fails.
Things to try
Use the interactive tool above to see the difference firsthand:
- Analyze the Default Text: Click the "Normalize Text" button with the default sentence. Look at the results in the "Transformation Matrix".
- Notice how for "foxes", both stemming and lemmatization produce "fox".
- For "jumping", stemming gives "jump" and lemmatization also gives "jump". In many cases, they agree.
- But for "studying", stemming produces "studi" (not a real word), while lemmatization correctly returns "study".
- Test Irregular Verbs: Clear the text and type "He went to the best restaurants. They were good." Click "Normalize".
- Stemming fails on these: "went" stays "went", "best" stays "best", "were" becomes "wer".
- Lemmatization shines: "went" becomes "go", "best" becomes "good", and "were" becomes "be". It understands the underlying lemma.
- The "Caring" vs. "Cars" Test: Type "caring for cars".
- Stemming will likely reduce both to "car". It loses the meaning and context.
- Lemmatization will correctly identify "caring" -> "care" and "cars" -> "car".
Key Takeaway: Speed vs. Accuracy
The choice between stemming and lemmatization is a trade-off. Stemming is fast and good enough for many applications, but it can be inaccurate. Lemmatization is more accurate and provides meaningful root words, but it comes at a higher computational cost. Choose the tool that best fits the needs of your specific NLP task.
The stemmers you will meet
Porter (1980) is the classic for English — five phases of suffix rules, well understood, moderately aggressive. Still the default in many pipelines.
Snowball (Porter2) is Porter's own improvement, and it covers many languages. The better default when you have a choice.
Lancaster is more aggressive and produces shorter stems, sometimes conflating words that should stay distinct.
For lemmatisation, WordNet (via NLTK) is the traditional English option, and spaCy is the practical one — it tags parts of speech and lemmatises in one pass, across many languages, and is considerably faster.
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The studies were better than we ran previously")
[t.lemma_ for t in doc]
# ['the', 'study', 'be', 'well', 'than', 'we', 'run', 'previously']
Note "be" from "were" and "run" from "ran" — the irregulars a stemmer cannot reach.
What it buys, measured
The benefit is vocabulary reduction, and it is substantial for classical models.
On a typical English corpus, stemming reduces the distinct term count by roughly 30–50%. That means more occurrences per feature, a smaller and better-conditioned matrix, and less overfitting on rare forms.
Whether it improves accuracy is less certain and task-dependent. For topic classification and search recall it usually helps. For sentiment it can hurt, because it flattens distinctions ("caring" and "cared" carry different implications). For anything where tense or number is the signal, it clearly hurts.
The honest guidance: treat it as a hyperparameter and test it, rather than applying it because a tutorial did.
Common mistakes
- Lemmatising without a part of speech, so verbs and adjectives are treated as nouns and left unchanged.
- Using an English stemmer on other languages. Suffix rules are language-specific, and applying English ones to German or Turkish produces nonsense.
- Stemming before a transformer. Destroys information and creates out-of-vocabulary fragments.
- Stemming the query but not the index (or vice versa) in a search system, so nothing matches.
- Assuming lemmatisation is always better. It is more correct and slower, and for a search index the stem is fine.
- Applying it to named entities, where "Rogers" becoming "Roger" changes the referent.
Questions people ask
Which should I use? Lemmatisation when output quality matters or is shown to users; stemming for search indexes and quick baselines; neither for transformers.
Is lemmatisation much slower? Meaningfully so, because it needs POS tagging and dictionary lookups — roughly an order of magnitude in NLTK, less with spaCy's pipeline.
Do I need stop-word removal too? For classical bag-of-words models, often yes. For transformers, no.
What about other languages? Snowball covers many for stemming; spaCy has lemmatisers for a wide range. Morphologically rich languages benefit more from lemmatisation than English does.
Does it help search? Yes, for recall — matching "running" to a query for "run". Modern search engines do this, alongside embedding-based retrieval.
Why does the Porter stemmer produce non-words? Because it applies suffix rules without any dictionary. The stems are index keys, not words.
Recap in one screen
- Both reduce inflected forms to a common form so evidence pools across them.
- Stemming chops with rules — fast, and it produces non-words and misses irregulars.
- Lemmatisation uses a dictionary and part of speech — correct, slower, and it needs the POS to be right.
- Use stemming for index keys, lemmatisation for anything readable, neither before a transformer.
- Vocabulary shrinks by 30–50%; accuracy gains are task-dependent, so test rather than assume.