Menu

Statistical Tests and A/B Testing

Statistical tests help determine whether observed differences or relationships provide meaningful evidence under a specified probability model.

For example, an online store may compare checkout conversion rates, while a manufacturer may analyse differences in product weight across production lines.

Statistical tests evaluate hypotheses using sample observations. A/B testing uses controlled experiments to estimate the effect of one version or intervention compared with another.

What Are the Main Statistical Tests?

Test Purpose Example
Z-Test Tests means or proportions under suitable conditions Delivery time
T-Test Compares population means Customer spending
Chi-Square Tests association between categorical variables Device and purchase outcome
ANOVA Compares means across multiple groups Production centres
Mann-Whitney U Compares independent groups using ranks Satisfaction ratings
Wilcoxon Signed-Rank Tests paired differences using ranks Before-and-after measurements
Fisher's Exact Tests association in small contingency tables Rare categorical outcomes

The correct test depends on data types, group structure, independence, and statistical assumptions.

What Is a Z-Test?

A Z-test compares a standardised statistic with the standard normal distribution under suitable assumptions.

For a sample mean with known population standard deviation:

\[ Z=\frac{\bar{x}-\mu_0}{\sigma/\sqrt{n}} \]

Example

Suppose average delivery time is claimed to be 30 minutes.

A sample of 100 deliveries has:

\[ \bar{x}=32,\quad \sigma=8 \]

Then:

\[ Z=\frac{32-30}{8/\sqrt{100}}=2.5 \]

The two-sided p-value is approximately 0.0124.

This provides evidence against the hypothesised 30-minute population mean at the 5% significance level, assuming the test conditions hold.

What Is a T-Test?

A t-test evaluates hypotheses about means using estimated standard deviations.

Common types include:

  • One-sample t-test
  • Independent two-sample t-test
  • Paired t-test

The one-sample statistic is:

\[ t=\frac{\bar{x}-\mu_0}{s/\sqrt{n}} \]

Python Example

from scipy import statsweights = [498, 502, 501, 499, 503,           500, 497, 504, 501, 499]result = stats.ttest_1samp(    weights,    popmean=500)print(result.statistic)print(result.pvalue)

The test evaluates whether the population mean weight differs from 500 grams.

Independent Two-Sample T-Test

An independent two-sample t-test compares population means across two independent groups.

Welch's t-test does not assume equal population variances.

group_a = [1200, 1400, 1500, 1600, 1700]group_b = [1500, 1700, 1800, 1900, 2100]result = stats.ttest_ind(    group_a,    group_b,    equal_var=False)print(result.pvalue)

Random assignment is necessary for interpreting group differences causally in a controlled experiment.

What Is a Chi-Square Test?

A Chi-Square Test of Independence examines associations between categorical variables.

For example:

Device Purchased Not Purchased
Mobile 120 180
Desktop 150 150

The statistic is:

\[ \chi^2=\sum\frac{(O-E)^2}{E} \]

Here, \(O\) represents observed counts and \(E\) represents expected counts under independence.

import numpy as npfrom scipy.stats import chi2_contingencytable = np.array([    [120, 180],    [150, 150]])result = chi2_contingency(table)print(result.statistic)print(result.pvalue)

The test evaluates whether device category and purchase outcome are associated. It does not establish causation.

What Is ANOVA?

Analysis of Variance (ANOVA) tests whether population means differ across multiple groups.

For three groups:

\[ H_0:\mu_1=\mu_2=\mu_3 \]

The F-statistic is:

\[ F=\frac{MS_{between}}{MS_{within}} \]

Python Example

from scipy.stats import f_onewaycentre_a = [30, 32, 31, 29, 33]centre_b = [35, 36, 34, 37, 35]centre_c = [28, 29, 30, 27, 29]result = f_oneway(    centre_a,    centre_b,    centre_c)print(result.pvalue)

A significant result indicates evidence that at least one population mean differs.

Post-hoc tests may be needed to identify which groups differ, with appropriate multiple-comparison controls.

What Is the Mann-Whitney U Test?

The Mann-Whitney U test compares two independent groups using ranks.

It is useful for certain ordinal or numerical data.

from scipy.stats import mannwhitneyugroup_a = [3, 4, 4, 5, 3]group_b = [2, 3, 3, 4, 2]result = mannwhitneyu(    group_a,    group_b,    alternative="two-sided")print(result.pvalue)

The test evaluates rank-based distributional differences. It is not automatically a test of medians unless additional assumptions hold.

How Do You Choose a Statistical Test?

Question Suitable Test
Does a mean differ from a reference value? One-sample t-test
Do two independent means differ? Welch's t-test
Do paired measurements differ? Paired t-test
Do three or more group means differ? ANOVA or Welch ANOVA
Are categorical variables associated? Chi-square test
Do two independent ranked distributions differ? Mann-Whitney U
Do two conversion proportions differ? Two-proportion z-test or exact method

Test selection should follow the study design and assumptions, not whichever method produces the smallest p-value.

What Is A/B Testing?

A/B testing compares two versions of a product or intervention through random assignment.

Group A: Existing version or control.

Group B: Modified version or treatment.

For example, an e-commerce company may test whether a simplified checkout page improves conversion rates.

Main A/B Testing Steps

  1. Define the business objective.
  2. Select the primary metric.
  3. Specify null and alternative hypotheses.
  4. Randomly assign eligible participants.
  5. Determine the required sample size.
  6. Run the experiment under predefined rules.
  7. Analyse statistical and business impact.

How Do You Measure Conversion Rate?

\[ Conversion\ Rate= \frac{Completed\ Purchases}{Eligible\ Users}\times100 \]

Consider this hypothetical experiment.

Metric Version A Version B
Eligible Users 10,000 10,000
Purchases 1,200 1,350
Conversion Rate 12% 13.5%

Absolute Improvement

\[ 13.5\%-12\%=1.5 \]

Version B improves the observed conversion rate by 1.5 percentage points.

Relative Improvement

\[ \frac{13.5-12}{12}\times100=12.5\% \]

Version B shows a 12.5% relative increase in observed conversion.

The experiment still requires statistical analysis to quantify uncertainty.

How Do You Analyse A/B Testing Results in Python?

A two-proportion z-test can compare conversion probabilities under suitable assumptions.

import numpy as npfrom statsmodels.stats.proportion import proportions_ztestconversions = np.array([1200, 1350])sample_sizes = np.array([10000, 10000])z_statistic, p_value = proportions_ztest(    count=conversions,    nobs=sample_sizes,    alternative="two-sided")print("Z-statistic:", z_statistic)print("P-value:", p_value)

The test evaluates the null hypothesis that the two population conversion probabilities are equal.

The sign of the Z-statistic depends on group order, but the two-sided p-value does not.

Statistical Significance vs Practical Significance

Statistical significance evaluates evidence against the null hypothesis.

A common threshold is:

\[ \alpha=0.05 \]

Practical significance evaluates whether the effect is large enough to justify a business decision.

For example, a statistically significant conversion improvement may be financially unimportant if implementation costs exceed the additional revenue.

Confidence intervals help estimate the plausible magnitude of the effect.

An illustrative 95% confidence interval for a 1.5-percentage-point conversion improvement might range from 0.6 to 2.4 percentage points.

Such an interval excludes zero, but the business still needs to evaluate costs and operational consequences.

What Are Common A/B Testing Mistakes?

  • Stopping Early: Repeatedly checking significance and stopping without a suitable sequential method increases false-positive risk.
  • Insufficient Sample Size: Small experiments may lack adequate statistical power.
  • Changing Metrics: Selecting favourable metrics after observing results increases misleading conclusions.
  • Poor Randomisation: Incorrect assignment can introduce systematic group differences.
  • Ignoring Multiple Tests: Testing many metrics or variants increases false-discovery risk.
  • Confusing Significance With Value: A detectable effect is not automatically profitable.

Where Are Statistical Tests and A/B Testing Used?

Industry Application
E-Commerce Checkout and pricing experiments
Marketing Landing-page conversion
Healthcare Treatment comparisons
Manufacturing Production process analysis
Product Development Engagement and retention experiments
Logistics Delivery process improvements

Statistical testing supports evidence-based decisions when combined with an appropriate design, effect-size estimates, and careful interpretation.


Data Science Tutorial for Beginners

VA

Vaishali Ardhana



Learn via Video Course

undefined Logo

0 hrs



Get in Touch For Details