A/B test calculator
Before you test two versions of a page, find out how many visitors a fair test needs, and how long that takes at your traffic.
Confidence and power
Visitors needed in each version
13,809
27,618 in total, to tell 2.00% from 2.50%. At 500 visitors a week that takes about 13 months.
That's longer than most tests can run cleanly: seasons, ads and offers change in the meantime. With this traffic, make bigger changes and compare equal weeks before and after instead.
Why sample size matters
An A/B test splits your visitors between two versions and compares how many convert. With too few visitors, the difference you see is mostly chance: run the same test twice and the "winner" can flip. The calculator tells you how many visitors you need before the difference means something.
The numbers grow fast at low conversion rates. To tell a page converting at 2% from one at 2.5%, you need about 13,800 visitors on each version. To tell 2% from 3%, about 3,800 each.
When your traffic isn't enough
Most small businesses don't have that traffic on one page. That doesn't mean you can't improve: it means you improve differently. Fix what's clearly wrong without testing it, make changes big enough to show up, and compare equal full weeks before and after. We cover the method in conversion rate optimization with low traffic.
Testing isn't the only lever
A test picks the better of two versions of the same page, shown to everyone. When your visitors need different things, a better single page has a ceiling. Sending each visitor to the offer that fits them is a different lever; we compare the two in funnel vs A/B testing. To see your current rate, use the conversion rate calculator; to find what's costing you sales, the sales page grader.
Questions
What is the minimum detectable effect (MDE)?
The smallest relative change you want the test to be able to detect. 25% on a 2% page means telling 2% apart from 2.5%. The smaller the lift, the more visitors you need: halving it roughly quadruples the sample.
What are confidence and power?
Confidence (usually 95%) is how sure you want to be that a difference isn't luck. Power (usually 80%) is the chance the test spots a real difference of the size you set. Raising either one raises the sample.
Which formula does it use?
The standard two-sided test for two proportions, with the pooled rate under no difference. Different tools use slightly different methods, so expect small differences between calculators.
Can I stop the test as soon as one version is ahead?
No. Checking every day and stopping at the first good-looking result makes false winners much more likely. Decide the sample first, then wait for it.
What if my traffic is too low?
Then a test won't decide in a reasonable time. Fix the obvious problems first, make bigger changes rather than small tweaks, and compare equal full weeks before and after each change.