Business & E-commerce
A/B Test Significance Calculator
Check whether an A/B test result is statistically significant — conversion rates, relative uplift, p-value and confidence using a two-proportion z-test.
How to use
- 1Enter visitors and conversions for version A (control) and B (variant).
- 2Pick your confidence level (95% is standard).
How it's calculated
Two-proportion z-test with pooled standard error: z = (p_B − p_A) ÷ √(p(1 − p)(1/n_A + 1/n_B)). Two-sided p-value = 2 × (1 − Φ(|z|)).
Frequently asked questions
What does 95% significance mean?
If there were truly no difference, a result this extreme would happen less than 5% of the time. It doesn't mean there's a 95% chance B is better.
Can I stop the test as soon as it's significant?
No — 'peeking' inflates false positives. Decide the sample size before the test (use the sample size calculator) and run to completion.
What if it's not significant?
You can't conclude there's no difference — the test may be underpowered. The effect might be too small to detect with this traffic.
What the p-value actually answers
A p-value under 0.05 means: if A and B truly performed identically, a gap this large or larger would happen by chance less than 5% of the time — it is not the probability that B is really better, a common misreading. The two-proportion z-test used here pools both groups' conversion rates to estimate the standard error, which is the standard, more conservative approach for comparing two percentages.
Worked example: A/B Test Significance Calculator
Control A: 10,000 visitors, 300 conversions (3.00%). Variant B: 10,000 visitors, 360 conversions (3.60%) — a 20% relative uplift. The pooled conversion rate is 660 ÷ 20,000 = 3.3%, giving a standard error that produces a z-score of about 2.375 and a two-sided p-value of about 0.0175. Since 0.0175 is below the 0.05 threshold for 95% confidence, the result is reported as significant, with B as the winner. Run the identical test at one-tenth the traffic — 1,000 visitors and 30 vs. 36 conversions, the same 20% relative lift — and the result flips to not significant, because the same proportional gap is far less reliable at a tenth of the sample size.
Why stopping early inflates false positives
Checking results daily and stopping the moment the test crosses significance ('peeking') dramatically increases the chance of a false positive, because random noise will cross the significance line at some point during almost any test if you keep checking. Use the sample size calculator to decide the required traffic before launching, then let the test run to that number rather than to a calendar date or a lucky-looking p-value.
Why 'not significant' doesn't mean the variants are equal
A test that fails to reach significance could mean there's genuinely no difference between A and B, or it could simply mean the test didn't collect enough traffic to detect a real but smaller difference — the two explanations look identical in the output. This is why the sample size calculator's minimum-detectable-effect concept matters: a test designed to reliably detect only a 20% relative lift will routinely report 'not significant' for genuine but smaller effects, which is a limitation of the test's statistical power, not evidence that the variants perform the same.
Conversions cannot outnumber visitors, on either side
Both groups need a visitor count above zero, and conversions cannot exceed visitors on either side, or I am stopped before a verdict. The rates are conversions divided by visitors, and the uplift is B's rate relative to A's, so a zero rate on A makes that uplift zero rather than infinite. The p-value is two-sided. The verdict says significant, and names whichever side is ahead, only when that p-value beats the confidence I picked. The choices are 90, 95, and 99, and 95 is where it opens. A result that misses the bar says not significant, even if B's rate is visibly higher. The z-score is shown either way. Nothing here peeks at a running test or tells me to stop early.
Related tools
- Sample Size Calculator (Surveys & A/B Tests)
Find the sample size you need for a survey (margin of error) or an A/B test (minimum detectable effect) at your chosen confidence and power.
- Pricing Page A/B Test Idea Generator
Get proven pricing-page experiments for your business and goal — plan layout, anchoring, annual billing, trials, social proof and more — each with the reason it works and the metric to track.
- SaaS Pricing & Unit Economics Calculator
Model SaaS revenue and unit economics — MRR and ARR from customers and price, churn, customer lifetime value (LTV), customer acquisition cost (CAC), LTV:CAC ratio, and CAC payback months.
- Customer Lifetime Value (CLV) Calculator
Estimate customer lifetime value from average order value, purchase frequency, gross margin and retention — and compare it with acquisition cost.
- eBay Fee Calculator
Estimate eBay final value fees and the per-order fee on a sale, plus your profit after item cost and shipping.
- P-Value Calculator (z, t, chi-square)
Get the p-value for a z-score, t-statistic or chi-square statistic — one- or two-tailed — and check it against your significance level.