Hypothesis Testing and A/B Testing

Hypothesis testing helps you decide if a pattern in your data is real or just chance. It is the base of many business experiments.

a. Understanding Hypothesis Testing

A hypothesis test is a method to check a claim using data. You start with a null hypothesis, which says there is no effect or no difference. The alternative hypothesis says there is one.

For example, the null hypothesis might say that a new website design has no effect on sales. You collect data and test whether the results are strong enough to reject that idea.

You also choose a significance level before the test, usually 0.05. This is the amount of risk you accept of finding an effect that is not really there.

b. T-Tests and Chi-Square Tests

A t-test compares the averages of two groups. For example, you can check if customers who saw a new offer spent more than those who did not. SciPy provides this in a single function.

from scipy import stats
t_stat, p_value = stats.ttest_ind(group_a, group_b)
print(p_value)

A chi-square test checks if two categorical columns are related. For example, it can test if the preferred product differs by city.

table = pd.crosstab(df["city"], df["product"])
chi2, p, dof, expected = stats.chi2_contingency(table)
print(p)

Choose the test that matches your data. Use a t-test for comparing numeric averages and a chi-square test for comparing categories.

c. Understanding P-Values and Significance

The p-value tells you how likely you are to see results at least this strong if the null hypothesis were true. A small p-value means the results are unlikely to be due to chance.

If the p-value is below your significance level, such as 0.05, you reject the null hypothesis and call the result statistically significant. If it is above, you do not have enough evidence to claim a difference.

A p-value does not tell you how big or important an effect is. A tiny difference can be significant in a very large sample. Always look at the size of the effect as well as the p-value.

d. Running and Interpreting an A/B Test

An A/B test compares two versions of something, like two webpage designs. Users are randomly split into group A, which sees the original, and group B, which sees the new version. You then compare a metric such as the conversion rate.

Plan the test before you start. Decide the metric, the significance level, and how many users you need. Do not stop the test early just because the results look good, as this raises the risk of false results.

After the test, use the right statistical test and check the p-value. Report the result in plain words, such as "Version B increased sign-ups by 12 percent, and the result is statistically significant." Also share the size of the change and any limits of the test.