If you have ever searched for \u201crunning shoes\u201d on Google and seen results for \u201crun,\u201d \u201crunner,\u201d and \u201cruns,\u201d you have already seen NLP preprocessing at work. Techniques like stemming and lemmatization help computers understand that different word forms can share the same meaning.<\/p>\n\n\n\n
Human language is full of variation, so the same word can appear in many forms depending on tense, number, or grammar. By reducing these variations to a common base form, text normalization helps machine learning models work more efficiently and recognize patterns more accurately.<\/p>\n\n\n\n
In this article, we will walk through what stemming and lemmatization each do, how they differ, the specific tools Python provides for each, and how to decide which one to use for a given NLP task. Whether you are building a search engine, a chatbot, or a sentiment analysis model, understanding this distinction will directly affect the quality of your results.<\/p>\n\n\n\n
\n Stemming and lemmatization are text preprocessing techniques used in natural language processing (NLP) to reduce words to their base forms. Stemming removes word endings using fixed rules and may produce non-dictionary root forms, while lemmatization uses vocabulary and grammatical context to return the correct dictionary base form, known as the lemma. Stemming is generally faster but less accurate, whereas lemmatization is slower but provides more precise and meaningful results.\n <\/p>\n\n <\/div>\n\n<\/div>\n\n\n\n
Before looking at each technique individually, it helps to understand where they fit in the broader NLP pipeline.<\/p>\n\n\n\n
Stemming is the simpler and faster of the two techniques. The idea is straightforward: strip the suffixes and prefixes from a word until you are left with something close to its root. No dictionary lookup is involved, and no understanding of grammar is required. The algorithm just applies a set of rules and chops off endings.<\/p>\n\n\n\n
import nltk<\/p>\n\n\n\n
from nltk.stem import PorterStemmer<\/p>\n\n\n\n
stemmer = PorterStemmer()<\/p>\n\n\n\n
words = [“running”, “happiness”, “studies”, “troubling”, “easily”]<\/p>\n\n\n\n
for word in words:<\/p>\n\n\n\n
print(f”{word} \u2192 {stemmer.stem(word)}”)<\/p>\n\n\n\n
Because stemming works entirely through pattern-matching rules, it makes errors in two specific ways that are worth understanding: overstemming and understemming.<\/p>\n\n\n\n
It is also called false positives, which occur when a stemming algorithm reduces separate inflected words to the same word stem even though they are not related. <\/p>\n\n\n\n
For example, the Porter Stemmer algorithm stems “universal,” “university,” and “universe” to the same word stem. Though they are etymologically related, their meanings in the modern day are from widely different domains. Treating them as synonyms will reduce relevance in search results. Overstemming occurs when words are over-truncated, and in such cases, the meaning of the word may be distorted or have no meaning.<\/p>\n\n\n\n
It is also called “false negatives” and is when a stemming algorithm reduces inflected words to different word stems when they should map to the same one. The words “alumnus,” “alumni,” and “alumnae” all refer to the same concept of a former student, but most stemming algorithms fail to reduce them to a common root. <\/p>\n\n\n\n
A search for “alumni event” might miss documents that use “alumnae” even though they are clearly about the same thing. Understemming occurs when two words are stemmed from the same root, but are not given different stems even though they should be.<\/p>\n\n\n\n
Lemmatization<\/a> takes a fundamentally different approach. Instead of applying blind rules to chop off word endings, it looks up words in a dictionary and uses grammatical knowledge to find their actual base form, which is called the lemma.<\/p>\n\n\n\n import nltk<\/p>\n\n\n\n from nltk.stem import WordNetLemmatizer<\/p>\n\n\n\n nltk.download(‘wordnet’)<\/p>\n\n\n\n nltk.download(‘averaged_perceptron_tagger’)<\/p>\n\n\n\n lemmatizer = WordNetLemmatizer()<\/p>\n\n\n\n # Without POS tag (defaults to noun)<\/p>\n\n\n\n print(lemmatizer.lemmatize(“running”)) # \u2192 running<\/p>\n\n\n\n # With correct POS tag (verb)<\/p>\n\n\n\n print(lemmatizer.lemmatize(“running”, pos=”v”)) # \u2192 run<\/p>\n\n\n\n print(lemmatizer.lemmatize(“better”, pos=”a”)) # \u2192 good<\/p>\n\n\n\n print(lemmatizer.lemmatize(“studies”, pos=”v”)) # \u2192 study<\/p>\n\n\n\n Lemmatization performs morphological analysis, the study of how words are formed and structured, which lets it return a word\u2019s valid base form (lemma). Because it accounts for grammatical and morphological relationships, lemmatization usually produces more accurate and meaningful results than stemming, which simply chops off affixes without checking whether the result is a real word.<\/p>\n\n\n\n Morphological analysis enables the lemmatizer to understand relationships like that between \u201cwent\u201d and \u201cgo.\u201d Even though those two words share no letters in common, a lemmatizer recognizes \u201cwent\u201d as the past tense of \u201cgo\u201d and returns \u201cgo.\u201d A stemmer would typically leave \u201cwent\u201d unchanged, since there\u2019s no obvious suffix to remove.<\/p>\n\n\n\n\n
\n
Morphological Analysis: The Engine Behind Lemmatization<\/strong><\/h2>\n\n\n\n
\n
\n
\n