Comparing two proportions
Two groups, two rates: a new checkout flow converting 64 of 120 visitors against the old one's 48 of 110. The two-proportion z-test asks whether the population rates differ. Because the null hypothesis says both groups share one rate, the test pools all the data into a single p for its standard error; the confidence interval for the difference, reported alongside, does not pool, because it is estimating a gap that may really exist.
p̂₁ = 0.5333 (64/120), p̂₂ = 0.4364 (48/110), pooled p = 0.4870. z = 1.4697, two-tailed p = 0.1416. A 9.7-point gap in the samples, but with these sample sizes, gaps that large arise by chance about one time in seven when nothing differs. The 95% interval for the difference (−0.032 to 0.226) contains zero and reaches from slightly negative to a hefty +23 points: the honest summary is "unresolved: get more data", not "no effect".
Assumptions, in plain sight
- Independent random samples from both groups; the groups must not share members.
- Each group needs enough successes and failures (roughly 10 of each, using the pooled p) for the normal approximation to hold.
- A/B tests qualify naturally: randomised assignment is the gold-standard version of the random-sample assumption.
Peeking at an A/B test daily and stopping the first time p dips below 0.05. Repeated peeking manufactures false positives at several times the advertised rate. Fix the sample size (or the analysis date) in advance. The margin-of-error calculator sizes it.
Frequently asked questions
Why pool for the test but not for the interval?
The test's standard error must describe the world where the null is true: one shared rate, hence pooled. The interval estimates the actual difference, so it respects each group's own rate.
How many visitors does an A/B test need?
It depends on the baseline rate and the smallest difference worth detecting. For proportions near 0.5 and a ±3-point margin you are near a thousand per group. Plan with the margin-of-error calculator before launching.