Point Estimation and Maximum Likelihood

A point estimate is a single number calculated from a sample to represent an unknown population value. The sample mean estimates μ, the sample proportion estimates p, and the sample variance estimates σ².

Good Estimators

• Unbiased: on average across many samples, it lands on the true value. Dividing by n − 1 in the sample variance is what makes it unbiased.
• Low variance: its values do not swing widely from sample to sample.
• Consistent: it gets closer to the true value as the sample grows.

Maximum Likelihood Estimation

Maximum likelihood estimation (MLE) picks the parameter value that makes the observed data most probable.

L(θ) = P(data | θ)

• θ (theta): the unknown parameter being estimated, such as a probability or a mean.
• L(θ): the likelihood, which measures how well a particular θ explains the data you actually saw.
• MLE: the value of θ that gives the largest L(θ).

Example: A coin is flipped 10 times, and lands heads 7 times. Find the MLE of p, the probability of heads.

1. The likelihood is L(p) = C(10, 7) × p⁷ × (1 − p)³.
2. Try a few values: L(0.5) ≈ 0.117, L(0.6) ≈ 0.215, L(0.7) ≈ 0.267 and L(0.8) ≈ 0.201.
3. The likelihood is highest at p = 0.7, which matches calculus: setting the derivative of the log-likelihood to zero gives p = 7 / 10.

The MLE is simply the observed proportion. Logistic regression and many other models are trained by finding the parameters that maximize the likelihood of the training data.