Statistical Tests and A/B Testing
Statistical tests help determine whether observed differences or relationships provide meaningful evidence under a specified probability model.
For example, an online store may compare checkout conversion rates, while a manufacturer may analyse differences in product weight across production lines.
Statistical tests evaluate hypotheses using sample observations. A/B testing uses controlled experiments to estimate the effect of one version or intervention compared with another.
What Are the Main Statistical Tests?
| Test | Purpose | Example |
|---|---|---|
| Z-Test | Tests means or proportions under suitable conditions | Delivery time |
| T-Test | Compares population means | Customer spending |
| Chi-Square | Tests association between categorical variables | Device and purchase outcome |
| ANOVA | Compares means across multiple groups | Production centres |
| Mann-Whitney U | Compares independent groups using ranks | Satisfaction ratings |
| Wilcoxon Signed-Rank | Tests paired differences using ranks | Before-and-after measurements |
| Fisher's Exact | Tests association in small contingency tables | Rare categorical outcomes |
The correct test depends on data types, group structure, independence, and statistical assumptions.
What Is a Z-Test?
A Z-test compares a standardised statistic with the standard normal distribution under suitable assumptions.
For a sample mean with known population standard deviation:
\[ Z=\frac{\bar{x}-\mu_0}{\sigma/\sqrt{n}} \]
Example
Suppose average delivery time is claimed to be 30 minutes.
A sample of 100 deliveries has:
\[ \bar{x}=32,\quad \sigma=8 \]
Then:
\[ Z=\frac{32-30}{8/\sqrt{100}}=2.5 \]
The two-sided p-value is approximately 0.0124.
This provides evidence against the hypothesised 30-minute population mean at the 5% significance level, assuming the test conditions hold.
What Is a T-Test?
A t-test evaluates hypotheses about means using estimated standard deviations.
Common types include:
- One-sample t-test
- Independent two-sample t-test
- Paired t-test
The one-sample statistic is:
\[ t=\frac{\bar{x}-\mu_0}{s/\sqrt{n}} \]
Python Example
from scipy import statsweights = [498, 502, 501, 499, 503, 500, 497, 504, 501, 499]result = stats.ttest_1samp( weights, popmean=500)print(result.statistic)print(result.pvalue)The test evaluates whether the population mean weight differs from 500 grams.
Independent Two-Sample T-Test
An independent two-sample t-test compares population means across two independent groups.
Welch's t-test does not assume equal population variances.
group_a = [1200, 1400, 1500, 1600, 1700]group_b = [1500, 1700, 1800, 1900, 2100]result = stats.ttest_ind( group_a, group_b, equal_var=False)print(result.pvalue)Random assignment is necessary for interpreting group differences causally in a controlled experiment.
What Is a Chi-Square Test?
A Chi-Square Test of Independence examines associations between categorical variables.
For example:
| Device | Purchased | Not Purchased |
|---|---|---|
| Mobile | 120 | 180 |
| Desktop | 150 | 150 |
The statistic is:
\[ \chi^2=\sum\frac{(O-E)^2}{E} \]
Here, \(O\) represents observed counts and \(E\) represents expected counts under independence.
import numpy as npfrom scipy.stats import chi2_contingencytable = np.array([ [120, 180], [150, 150]])result = chi2_contingency(table)print(result.statistic)print(result.pvalue)The test evaluates whether device category and purchase outcome are associated. It does not establish causation.
What Is ANOVA?
Analysis of Variance (ANOVA) tests whether population means differ across multiple groups.
For three groups:
\[ H_0:\mu_1=\mu_2=\mu_3 \]
The F-statistic is:
\[ F=\frac{MS_{between}}{MS_{within}} \]
Python Example
from scipy.stats import f_onewaycentre_a = [30, 32, 31, 29, 33]centre_b = [35, 36, 34, 37, 35]centre_c = [28, 29, 30, 27, 29]result = f_oneway( centre_a, centre_b, centre_c)print(result.pvalue)A significant result indicates evidence that at least one population mean differs.
Post-hoc tests may be needed to identify which groups differ, with appropriate multiple-comparison controls.
What Is the Mann-Whitney U Test?
The Mann-Whitney U test compares two independent groups using ranks.
It is useful for certain ordinal or numerical data.
from scipy.stats import mannwhitneyugroup_a = [3, 4, 4, 5, 3]group_b = [2, 3, 3, 4, 2]result = mannwhitneyu( group_a, group_b, alternative="two-sided")print(result.pvalue)The test evaluates rank-based distributional differences. It is not automatically a test of medians unless additional assumptions hold.
How Do You Choose a Statistical Test?
| Question | Suitable Test |
|---|---|
| Does a mean differ from a reference value? | One-sample t-test |
| Do two independent means differ? | Welch's t-test |
| Do paired measurements differ? | Paired t-test |
| Do three or more group means differ? | ANOVA or Welch ANOVA |
| Are categorical variables associated? | Chi-square test |
| Do two independent ranked distributions differ? | Mann-Whitney U |
| Do two conversion proportions differ? | Two-proportion z-test or exact method |
Test selection should follow the study design and assumptions, not whichever method produces the smallest p-value.
What Is A/B Testing?
A/B testing compares two versions of a product or intervention through random assignment.
Group A: Existing version or control.
Group B: Modified version or treatment.
For example, an e-commerce company may test whether a simplified checkout page improves conversion rates.
Main A/B Testing Steps
- Define the business objective.
- Select the primary metric.
- Specify null and alternative hypotheses.
- Randomly assign eligible participants.
- Determine the required sample size.
- Run the experiment under predefined rules.
- Analyse statistical and business impact.
How Do You Measure Conversion Rate?
\[ Conversion\ Rate= \frac{Completed\ Purchases}{Eligible\ Users}\times100 \]
Consider this hypothetical experiment.
| Metric | Version A | Version B |
|---|---|---|
| Eligible Users | 10,000 | 10,000 |
| Purchases | 1,200 | 1,350 |
| Conversion Rate | 12% | 13.5% |
Absolute Improvement
\[ 13.5\%-12\%=1.5 \]
Version B improves the observed conversion rate by 1.5 percentage points.
Relative Improvement
\[ \frac{13.5-12}{12}\times100=12.5\% \]
Version B shows a 12.5% relative increase in observed conversion.
The experiment still requires statistical analysis to quantify uncertainty.
How Do You Analyse A/B Testing Results in Python?
A two-proportion z-test can compare conversion probabilities under suitable assumptions.
import numpy as npfrom statsmodels.stats.proportion import proportions_ztestconversions = np.array([1200, 1350])sample_sizes = np.array([10000, 10000])z_statistic, p_value = proportions_ztest( count=conversions, nobs=sample_sizes, alternative="two-sided")print("Z-statistic:", z_statistic)print("P-value:", p_value)The test evaluates the null hypothesis that the two population conversion probabilities are equal.
The sign of the Z-statistic depends on group order, but the two-sided p-value does not.
Statistical Significance vs Practical Significance
Statistical significance evaluates evidence against the null hypothesis.
A common threshold is:
\[ \alpha=0.05 \]
Practical significance evaluates whether the effect is large enough to justify a business decision.
For example, a statistically significant conversion improvement may be financially unimportant if implementation costs exceed the additional revenue.
Confidence intervals help estimate the plausible magnitude of the effect.
An illustrative 95% confidence interval for a 1.5-percentage-point conversion improvement might range from 0.6 to 2.4 percentage points.
Such an interval excludes zero, but the business still needs to evaluate costs and operational consequences.
What Are Common A/B Testing Mistakes?
- Stopping Early: Repeatedly checking significance and stopping without a suitable sequential method increases false-positive risk.
- Insufficient Sample Size: Small experiments may lack adequate statistical power.
- Changing Metrics: Selecting favourable metrics after observing results increases misleading conclusions.
- Poor Randomisation: Incorrect assignment can introduce systematic group differences.
- Ignoring Multiple Tests: Testing many metrics or variants increases false-discovery risk.
- Confusing Significance With Value: A detectable effect is not automatically profitable.
Where Are Statistical Tests and A/B Testing Used?
| Industry | Application |
|---|---|
| E-Commerce | Checkout and pricing experiments |
| Marketing | Landing-page conversion |
| Healthcare | Treatment comparisons |
| Manufacturing | Production process analysis |
| Product Development | Engagement and retention experiments |
| Logistics | Delivery process improvements |
Statistical testing supports evidence-based decisions when combined with an appropriate design, effect-size estimates, and careful interpretation.
Data Science Tutorial for Beginners
Vaishali Ardhana








