Hypothesis Testing and Confidence Intervals

What is hypothesis testing?

Hypothesis testing is a framework for evaluating whether data provides evidence against a stated null hypothesis.

It does not:

• Prove that a new feature works
• Prove that a difference matters to the business
• Tell you the probability that your idea is correct

Typical steps:

1. State the null (H₀) and alternative (H₁) hypotheses
2. Choose a significance level (α), often 0.05
3. Choose an appropriate test
4. Collect data and calculate the test statistic and p-value
5. Interpret the result alongside the effect size and confidence interval

Null and alternative hypotheses

Define hypotheses before looking at results. Otherwise it's tempting to pick whichever comparison looks best (p-hacking).

Common tests:

P-values and significance

• A p-value measures how surprising the observed result (or something more extreme) would be if the null hypothesis and model assumptions were true.
• It is not the probability that the null hypothesis is true.
• If p < α (e.g., 0.05), the result is called "statistically significant".

• Statistical power is the probability of detecting a real effect of a given size. Small samples have low power.
• Running many tests increases the chance of false positives. Adjust or pre-register comparisons.

Confidence intervals

• A confidence interval gives a range of plausible values for an effect under a specified procedure.
• It shows both direction and precision.
• Width depends on:
  • Sample size (bigger → narrower)
  • Variability (more variable → wider)
  • Confidence level (99% → wider than 95%)

Common mistakes

• Significant ≠ important. A tiny effect can be significant with huge samples.
• Not significant ≠ no effect. The test may simply lack power.
• Reporting only "significant / no t significant". Report:
  • Effect size
  • Confidence interval
  • Sample size and context
  • Key assumptions
• Stopping a test early as soon as it looks significant inflates false positives.

Try it: A test estimates that a change improved conversion by 0.4 percentage points, with a confidence interval from −0.1 to 0.9 points. Explain why the result is uncertain.