P-Values, Significance Levels and Errors

A test needs a rule for deciding when the evidence is strong enough. That rule combines the p-value and the significance level.

P-Value

The p-value is the probability of getting a result at least as extreme as the one observed, assuming H₀ is true. A small p-value means the data would be unusual if H₀ were true. It is not the probability that H₀ is true.

Significance Level

The significance level, α (alpha), is the cutoff chosen in advance, commonly 0.05. The decision rule is:

• If p ≤ α, reject H₀.
• If p > α, fail to reject H₀.

Example: A two-tailed z-test gives z = 2.1. Decide at α = 0.05.

1. P(Z < 2.1) ≈ 0.9821, so the upper tail is 1 − 0.9821 = 0.0179.
2. A two-tailed test counts both tails, so p = 2 × 0.0179 = 0.0358.
3. Since 0.0358 < 0.05, reject H₀.

Type I and Type II Errors

• α: the chance of rejecting H₀ when it is true. Setting α = 0.05 accepts a 5% risk of a false alarm.
• β (beta): the chance of missing a real effect.
• Power = 1 − β: the chance of detecting a real effect. Larger samples and larger true effects raise power.

Lowering α reduces false positives but increases β, so these two errors trade off.