A/B Testing in Practice

An A/B test splits users randomly into two groups. Group A sees the current version, and group B sees the changed version. Because assignment is random, you can trace differences in outcomes to the change itself.

Comparing Two Conversion Rates

z = (p̂ᴮ − p̂ᴬ) / √( p̂ × (1 − p̂) × (1/nᴬ + 1/nᴮ) )

• p̂ᴬ and p̂ᴮ: the observed conversion rates in groups A and B.
• nᴬ and nᴮ: the number of users in each group.
• p̂ (pooled rate): total conversions across both groups divided by total users. H₀ says both groups share one true rate, so we combine the two samples to estimate it.
• Denominator: the standard error of the difference between the two rates.

Example: Version A converts 200 of 2,000 users (10%). Version B converts 250 of 2,000 users (12.5%). Is the difference significant at α = 0.05?

1. Pooled rate = (200 + 250) / 4,000 = 0.1125
2. SE = √(0.1125 × 0.8875 × (1/2000 + 1/2000)) = √0.0000998 ≈ 0.00999
3. z = (0.125 − 0.10) / 0.00999 ≈ 2.50
4. Two-tailed p ≈ 0.012, which is below 0.05, so reject H₀.

from statsmodels.stats.proportion import proportions_ztest

stat, p = proportions_ztest(count=[250, 200], nobs=[2000, 2000])
print(stat, p)

Code Explanation:

• count lists the conversions for B and then A.
• nobs lists the number of users in each group in the same order.
• The function pools the rates and runs the two-proportion z-test.
• It returns z of about 2.50 and a p-value of about 0.012, matching the manual result.

Practical Rules

• Fix the sample size first. Decide how many users each group needs before starting, based on the smallest improvement worth detecting.
• Do not stop early. Repeatedly checking results and stopping when p dips below 0.05 raises the false-positive rate well above 5%.
• Check practical size. With a very large sample, a tiny difference can be statistically significant yet too small to matter to the business.