Sample Size for Split Tests: How Much Traffic Is Enough?

Why clicks are the wrong unit

A test with 10,000 clicks per variant sounds large. At a 0.5% conversion rate it is 50 conversions per variant, and the difference between 45 and 55 conversions is mostly noise. What decides whether a result is real is the number of conversions and the size of the difference you want to detect.

A rule of thumb that works

For a simple A/B test with one goal, a usable approximation for the conversions you need per variant is:

conversions per variant ≈ 16 × (1 − p) / (d²)
p = baseline conversion rate (for example 0.02)
d = relative lift you want to detect (for example 0.20 for +20%)

This corresponds to the usual 95% confidence and 80% power. For low conversion rates, 1 − p is close to 1, so the formula simplifies to 16 divided by the lift squared:

The lesson is in the shape of the table: halving the lift you want to see quadruples the traffic you need. Test bold changes first; small tweaks need volumes most campaigns never reach.

A worked example

A landing page converts at 2% and you hope a new headline lifts it by 30%. You need about 180 conversions per variant, so roughly 9,000 clicks per variant at 2%, 18,000 in total. At 3,000 clicks a day the test takes six days. If the budget only allows 1,000 clicks a day, either run it for nearly three weeks or test a change big enough to show +50%.

When to stop a test

• Decide the size up front and stop when you reach it. Peeking every hour and stopping on the first good day inflates false winners.
• Run whole weeks. Traffic on Sunday night is not traffic on Tuesday noon. A test shorter than seven days measures the calendar.
• Stop early only for disasters: a variant that is broken, or one that loses so badly that the cost is not worth the certainty.

Traps that fake a winner

• Unequal traffic. If one variant got more mobile or more of one GEO, you compared audiences, not pages. Split at the tracking link so both variants share sources and hours.
• Bots in one variant. A burst of fake clicks lowers one variant's CR. Filter bad traffic before it enters the test, and compare the share of filtered clicks per variant.
• Counting the wrong goal. A variant that gets more leads of lower quality wins on CR and loses on payout. Compare EPC or revenue per click when the network pays by quality.
• Many variants at once. With five variants, one will look good by chance. Either size each variant for the comparison or test two at a time.