Bayes’ Theorem and Updating Beliefs

Bayes’ theorem reverses a conditional probability. It lets you go from “how likely is this evidence if the claim is true” to “how likely is the claim now that I have seen this evidence”.

The Formula

P(A | B) = P(B | A) × P(A) / P(B)

• P(A): the prior, your starting probability for A before seeing any evidence.
• P(B | A): the likelihood, how probable the evidence B is if A is true.
• P(B): the overall probability of seeing the evidence B, whether or not A is true.
• P(A | B): the posterior, the updated probability of A after seeing B.

The denominator is usually found by adding both routes to the evidence:

P(B) = P(B | A) × P(A) + P(B | not A) × P(not A)

Example: 20% of incoming emails are spam. The word “free” appears in 60% of spam emails and in 5% of non-spam emails. An email contains “free”. What is the probability it is spam?

1. P(spam) = 0.20 and P(not spam) = 0.80
2. P(“free”) = 0.60 × 0.20 + 0.05 × 0.80 = 0.12 + 0.04 = 0.16
3. P(spam | “free”) = (0.60 × 0.20) / 0.16 = 0.12 / 0.16 = 0.75

The word “free” raises the spam probability from 20% to 75%.

Why the Prior Matters

A strong test result can still mislead when what you're testing is rare. The same evidence applied to a low prior yields a lower posterior than the same evidence applied to a high one, which is why you should never leave out a prior. Naive Bayes classifiers use this exact update, combining the evidence from many words to score each email.