About this tool
A/B Test Significance Calculator is a free, in-browser tool that tells you whether the difference between two conversion rates is statistically significant or just noise. Enter each variant's visitors and conversions and it runs a two-proportion z-test locally — no data leaves your browser.
Conversion rates are pA = cA ÷ nA and pB = cB ÷ nB, and relative uplift is (pB − pA) ÷ pA × 100. The test pools the two groups, p = (cA + cB) ÷ (nA + nB), computes the standard error SE = √(p(1−p)(1/nA + 1/nB)) and the z-score z = (pB − pA) ÷ SE. The two-tailed p-value is 2 × (1 − Φ(|z|)), where Φ is the normal CDF built from an error-function approximation (Abramowitz-Stegun 7.1.26); confidence is (1 − p-value) × 100.
Use it after a split test to decide if variant B genuinely beat A. The verdict tile marks a result significant at 95% confidence or above. Remember that significance is not the same as a large or durable effect — always weigh the uplift and sample size too.
Frequently asked questions
What test does this use?
A two-tailed two-proportion z-test with a pooled standard error — the standard method for comparing two conversion rates. The p-value comes from the normal distribution via an error-function approximation (Abramowitz-Stegun formula 7.1.26), and confidence is 100% minus the p-value as a percentage.
What does 95% confidence mean here?
It means that if the two variants truly performed the same, you would see a difference this large or larger only about 5% of the time by chance. The verdict reads 'Significant' at 95% confidence or above. It does not tell you the size of the effect — check the uplift for that.
Why do I get an invalid result?
Conversions cannot exceed visitors, and both visitor counts must be above zero. The test also needs some variance: if nobody converted in either group, or everybody did, the standard error is zero and no z-score can be formed, so the tool reports the input as invalid.
Is significance enough to call a winner?
Not on its own. A tiny, unimportant difference can be significant with a huge sample, while a real effect can miss significance with a small one. Consider the relative uplift, the absolute numbers and whether the test ran long enough. All math here runs privately in your browser.
More tools