Statistical significance calculator
Ran an A/B test? Enter the visitors and conversions of both versions to see whether the difference is real or could still be luck.
Version A (control)
Version B (variant)
Result
Not significant yet
Conversion rate
A 2.00% · B 2.90%
Change, B vs A
+45.0%
p-value · confidence
0.0656 · 93.4%
The difference could still be chance. Don't stop early because one version is ahead: keep the test running until the sample size you planned. Plan the sample size.
How to read the result
The calculator compares the conversion rate of version A with version B and asks one question: if the two versions really converted the same, how often would you see a gap this big by chance? That chance is the p-value. Under 0.05 the result is called significant at 95% confidence.
Significance tells you the difference is probably not luck. It doesn't tell you the difference is large, or that it will hold next month. Look at the change between the two rates too: a significant +2% on a page that sells little may not be worth the work.
The two mistakes that create fake winners
Stopping early.If you check the result every day and stop the first time it crosses 95%, you'll declare winners that aren't. Decide the sample size before you start with the A/B test sample size calculator, then read the result once you reach it.
Testing too little traffic.With a few hundred visitors per version, only huge differences can be significant. If that's your traffic, compare equal weeks before and after a change instead; we explain how in conversion rate optimization with low traffic.
Before you test
A test picks the better of two versions. It's worth testing changes that could matter. To find what's costing your page sales in the first place, run it through the free sales page grader, and see your current rate with the conversion rate calculator.
Questions
What does statistically significant mean?
That a difference as large as the one you saw would be unlikely if the two versions really converted the same. By convention, "unlikely" means less than 5% of the time: a p-value under 0.05, or 95% confidence.
What is the p-value?
The probability of seeing a difference at least this large by chance alone, if the versions were actually equal. A p-value of 0.03 means a 3% chance. It is not the probability that B is better.
Which test does the calculator use?
A two-sided two-proportion z-test with a pooled rate, the standard test for comparing two conversion rates. It's the same test our A/B sample size calculator plans for, so the two agree.
My result is significant. Can I stop the test?
Only if you reached the sample size you decided before starting. Checking every day and stopping at the first significant result makes false winners far more likely. Plan the sample first, then read the result once.
My result isn't significant. Does that mean there's no difference?
No. It means this test can't tell yet. The difference may be real but smaller than your traffic can detect. Either keep the test running to the planned sample, or test a bigger change.
Do you store what I enter?
No. The calculator runs in your browser; nothing you type is sent anywhere.