Every day, billions of emails are filtered for spam, millions of news articles are automatically categorised, and thousands of customer support messages are routed to the right team \u2014 all without a human reading them first. Behind many of these systems sits a surprisingly simple yet remarkably effective algorithm: Naive Bayes.<\/p>\n\n\n\n
Of the Naive Bayes family, the multinomial variant is the workhorse of text classification. It is purpose-built for data where features represent counts or frequencies \u2014 exactly the kind of data produced when text is converted into a bag-of-words representation.<\/p>\n\n\n\n
Multinomial Naive Bayes is fast, interpretable, and competitive with far more complex models on short-text classification tasks. It scales to millions of documents without breaking a sweat, and its probabilistic foundations make its predictions transparent and explainable.<\/p>\n\n\n\n
This guide covers everything: from the probabilistic theory underpinning Naive Bayes, through the multinomial model’s mechanics and Laplace smoothing, to practical implementation with sklearn’s MultinomialNB, evaluation, and a comparison of the key Naive Bayes variants.<\/p>\n\n\n\n
\n Multinomial Naive Bayes is a probabilistic text classification algorithm that applies Bayes\u2019 theorem while assuming that features, usually word counts or term frequencies, are conditionally independent given the class label. It represents each document as a multinomial distribution over a vocabulary and estimates the probability of observing specific word frequencies within each class. By combining these likelihoods with prior class probabilities, the algorithm predicts the class with the highest posterior probability. Multinomial Naive Bayes is widely used in document classification, spam filtering, sentiment analysis, and other NLP classification tasks.\n <\/p>\n\n <\/div>\n\n<\/div>\n\n\n\n
Multinomial Naive Bayes<\/a> is grounded in Bayes’ theorem, one of the most fundamental results in probability theory. Understanding the theorem is the key to understanding what the algorithm<\/a> is actually computing.<\/p>\n\n\n\n Bayes’ theorem states:<\/p>\n\n\n\n P(Class | Document) = [P(Document | Class) \u00d7 P(Class)] \/ P(Document)<\/p>\n\n\n\n In the context of text classification, each term has a precise meaning:<\/p>\n\n\n\n \u2022 P(Class | Document) \u2014 Posterior probability: <\/strong>the probability that a document belongs to a given class, given its content. This is what the classifier computes and maximises.<\/p>\n\n\n\n \u2022 P(Document | Class) \u2014 Likelihood: <\/strong>the probability of observing this document’s word pattern if it truly belongs to the class. This is where word frequencies come in.<\/p>\n\n\n\n \u2022 P(Class) \u2014 Prior probability: <\/strong>the baseline probability of a class before seeing the document. Estimated from the proportion of training documents in each class.<\/p>\n\n\n\n \u2022 P(Document) \u2014 Marginal likelihood: <\/strong>the overall probability of this document across all classes. It is constant across all classes for a given document, so it is ignored during classification we simply compare numerators.<\/p>\n\n\n\n The classifier selects the class with the highest posterior probability the class for which the combination of prior probability and document likelihood is greatest.<\/p>\n\n\n\n The word “naive” in Naive Bayes refers to a bold simplifying assumption: all features in this case, all words in a document, are assumed to be conditionally independent given the class label.<\/p>\n\n\n\n In plain language, knowing that an email contains the word “free” tells you nothing additional about whether it also contains the word “money”, once you already know the email is spam. Each word is treated as an independent piece of evidence.<\/p>\n\n\n\n This assumption is almost never true in natural language. Words co-occur in patterns “machine” frequently precedes “learning”; “free” often accompanies “offer”. Real documents have rich dependency structures that the naive assumption completely ignores.<\/p>\n\n\n\n Despite its obvious inaccuracy, the naive independence assumption produces surprisingly strong classifiers for several reasons:<\/p>\n\n\n\n \u2022 Classification robustness: <\/strong>Naive Bayes only needs to correctly rank classes it does not need calibrated probabilities. Even with correlated features, the ranking is often correct.<\/p>\n\n\n\n \u2022 Bias towards the right direction: <\/strong>Correlated features tend to reinforce each other consistently across both classes, so their double-counting errors partially cancel out.<\/p>\n\n\n\n \u2022 High-dimensional advantage: <\/strong>In text classification, there are often thousands of features (vocabulary words). The independence assumption makes the parameter estimation tractable; instead of estimating millions of joint probabilities, the model estimates one probability per word per class.<\/p>\n\n\n\n For short documents and moderate vocabulary sizes, exactly the conditions of spam detection, news categorisation, and sentiment analysis, Naive Bayes consistently delivers accuracy that rivals far more complex models.<\/p>\n\n\n\n The multinomial model is specifically designed for discrete count data, making it the natural choice for text, where documents are commonly represented as word count vectors.<\/p>\n\n\n\n The bag-of-words (BoW) representation converts a document into a fixed-length vector of word counts, discarding word order and grammar. The process is:<\/p>\n\n\n\n \u2022 Tokenisation: <\/strong>Split the text into individual tokens (words or subwords).<\/p>\n\n\n\n \u2022 Vocabulary construction: <\/strong>Build a vocabulary of all unique tokens across the training corpus.<\/p>\n\n\n\n \u2022 Count vectorisation: <\/strong>For each document, count how many times each vocabulary word appears. The result is a vector of feature counts, one dimension per vocabulary word.<\/p>\n\n\n\n A document “free money free offer” with vocabulary {free, money, offer, meeting} becomes the count vector [2, 1, 1, 0]. This vector is the input to the multinomial Naive Bayes classifier.<\/p>\n\n\n\n Given a class label c, the multinomial model estimates the likelihood of observing a document’s word counts as:<\/p>\n\n\n\n P(Document | Class = c) = product of P(word_i | Class = c) raised to the power of count(word_i)<\/p>\n\n\n\n Where P(word_i | Class = c) is the probability of word i appearing in a document of class c, estimated from training data as:<\/p>\n\n\n\n P(word_i | c) = count(word_i in class c documents) \/ total word count in class c documents<\/p>\n\n\n\n Because multiplying many small probabilities together causes numerical underflow, the computation is performed in log-space the product of probabilities becomes a sum of log-probabilities. This is both numerically stable and computationally efficient.<\/p>\n\n\n\n The prior probability P(Class = c) is estimated simply as the proportion of training documents belonging to class c:<\/p>\n\n\n\n P(c) = number of documents in class c \/ total number of training documents<\/p>\n\n\n\n In a balanced dataset, all class priors are equal. In an imbalanced dataset such as a spam filter where spam is rarer than legitimate mail the prior reflects this imbalance, naturally making the classifier more conservative about assigning rare classes.<\/p>\n\n\n\n \n Gmail\u2019s original spam filtering system<\/strong>, launched by Google<\/strong> in 2004<\/strong>, heavily relied on Naive Bayes text classification<\/strong>. Despite the simplicity of the algorithm, it achieved remarkably high spam detection accuracy with very low false-positive rates by statistically learning which words and patterns were strongly associated with spam emails. Variants of Naive Bayes and related probabilistic filtering techniques still influence modern production spam detection systems<\/strong> because of their speed, efficiency, and strong performance on large-scale text classification tasks.\n <\/p>\n<\/div>\n\n\n\n Laplace smoothing, also called additive smoothing, addresses one of the most critical failure modes in multinomial Naive Bayes: the zero-probability problem.<\/p>\n\n\n\n Consider a word that appears in the test document but never appeared in any training document of a given class. Its estimated conditional probability is zero. Because probabilities are multiplied together (or summed in log-space), a single zero probability makes the entire class likelihood zero regardless of how strong the evidence from all other words is.<\/p>\n\n\n\n This is not a quirk of bad data. It is inevitable: any real-world deployment will encounter vocabulary words not seen in training for every class. Without correction, the classifier becomes brittle and unreliable on novel vocabulary.<\/p>\n\n\n\n Laplace smoothing adds a small pseudocount (alpha, typically 1) to every word count before computing probabilities:<\/p>\n\n\n\n P(word_i | c) = [count(word_i in class c) + alpha] \/ [total words in class c + alpha \u00d7 vocabulary size]<\/p>\n\n\n\n The effect:<\/p>\n\n\n\n \u2022 No word ever has a zero probability; every vocabulary word gets at least a pseudocount of alpha.<\/p>\n\n\n\n \u2022 Words that genuinely appear frequently in a class still dominate the smoothing effect is small relative to large true counts.<\/p>\n\n\n\n \u2022 The alpha parameter controls the strength of smoothing. Alpha = 1 is Laplace smoothing; alpha < 1 is Lidstone smoothing, which applies less aggressive redistribution.<\/p>\n\n\n\n Laplace smoothing is not just a numerical fix it is a form of regularisation. By redistributing a small portion of probability mass to unseen events, it produces a model that is less overconfident and more robust to vocabulary variation between training and test distributions.<\/p>\n\n\n\n Python’s scikit-learn provides a clean, efficient implementation of multinomial Naive Bayes through the MultinomialNB class. The full pipeline from raw text to evaluated classifier is concise and readable.<\/p>\n\n\n\n MultinomialNB expects non-negative integer or float feature counts not raw text strings. sklearn offers two primary vectorisers:<\/p>\n\n\n\nThe Naive Independence Assumption<\/strong><\/h2>\n\n\n\n
Why Does It Work Despite Being Wrong?<\/strong><\/h3>\n\n\n\n
The Multinomial Model: Feature Counts<\/strong><\/h2>\n\n\n\n
From Text to Feature Counts: Bag of Words<\/strong><\/h3>\n\n\n\n
Computing the Class Likelihood<\/strong><\/h3>\n\n\n\n
Prior Probability Estimation<\/strong><\/h3>\n\n\n\n
Laplace Smoothing: Handling Unseen Words<\/strong><\/h2>\n\n\n\n
The Zero-Probability Problem<\/strong><\/h3>\n\n\n\n
The Smoothing Solution<\/strong><\/h3>\n\n\n\n
Implementing Multinomial Naive Bayes with Sklearn<\/strong><\/h2>\n\n\n\n
Step 1: Text Vectorisation<\/strong><\/h3>\n\n\n\n