The CRO Math: How to Calculate Whether Your A/B Test Is Worth Running Before You Run It

Three numbers determine whether your test will produce a real answer.

That is the whole input list. Three numbers.

Standard A/B test sample size, for 95% confidence and 80% power, simplifies to:

Sample size per variant ≈ 16 / (baseline × MDE²)

Both inputs as decimals. Baseline 2.5% \= 0.025. MDE 10% relative \= 0.10.

Plug it in for a baseline of 2.5% and an MDE of 10%:

• 16 / (0.025 × 0.01) \= 16 / 0.00025 \= 64,000 visitors per variant

For both variants combined, that is 128,000 visitors. At 5,000 weekly visits per variant, the test runs for 12.8 weeks.

Three months. To detect a 10% relative lift on a 2.5% baseline. With no seasonal contamination. With no other tests running. Tests that long are unreliable for reasons we will hit in a minute.

Example A: Small DTC, $20K/month, 500 weekly PDP visits per variant

• Baseline CVR: 2.0% (0.02)

• MDE: 20% relative lift (anything smaller is below your noise floor)

• Sample size needed: 16 / (0.02 × 0.04) \= 20,000 per variant

• Runtime: 20,000 / 500 \= 40 weeks

Forty weeks. Almost a year. The test is dead before you press start. Your traffic budget is too small for meaningful tests on the homepage or PDP. You are testing on vibes whether you write up the result or not.

Honest answer for this tier: do not run A/B tests on site conversion. Run them on email subject lines or ad creative, where the audience is bigger and the math is on your side. For site CRO, use judgment and intentional design choices instead. Test a redesign with cohort tracking month over month, not split traffic.

Example B: Mid-DTC, $200K/month, 4,000 weekly PDP visits per variant

• Baseline CVR: 3.0% (0.03)

• MDE: 15% relative lift (a meaningful change, not a button-color twitch)

• Sample size needed: 16 / (0.03 × 0.0225) \= 23,704 per variant

• Runtime: 23,704 / 4,000 \= 6 weeks

Six weeks is on the edge. You will pick up some weekly noise. Seasonal drift starts to bite if you cross a promo window or a paid-spend change.

Real move for this tier: bump your MDE to 20% relative. That gets you to ~13,300 per variant, 3.3 weeks runtime, which is the sweet spot. You will test fewer things, but the tests you run will settle.

Example C: Bigger ecom, $750K+/month, 15,000 weekly PDP visits per variant

• Baseline CVR: 3.5% (0.035)

• MDE: 10% relative lift

• Sample size needed: 16 / (0.035 × 0.01) \= 45,714 per variant

• Runtime: 45,714 / 15,000 \= 3 weeks

This is the comfort zone. You have enough traffic to test meaningful changes in a clean window. Your math says go. The risk at this tier is the opposite: testing too many things at once, or testing things that should have been a design decision instead of a split.

If the runtime math says your test takes longer than 4 weeks, kill it.

Reasons your test corrupts past week 4:

• Seasonal drift. Black Friday energy in mid-November is not the same as the energy of the first week of December. Your variants pick up different buyer cohorts.

• Audience drift. Returning visitors weighted differently than first-time visitors. Email lists pushed during the test send a spike of one audience type to both variants, but on different days.

• Ad creative drift. If you change your Meta creative during the test, the audience composition changes. Your test is no longer testing what you think.

• Promotional contamination. A sitewide 10% off run for a week wipes the baseline.

A test over 4 weeks in DTC is a test of "did the world change while we measured" more than "does B beat A."

If your test does not fit under 4 weeks at your current traffic, you have three options.

The most common mistake we see in DTC operator-run tests: an MDE of 3% or 5% on a 2-3% baseline.

A 3% relative lift on a 3% baseline is a 0.09 percentage point absolute difference. To detect it at 95%/80% with a 3% baseline:

• 16 / (0.03 × 0.0009) \= 592,593 per variant

You would need over a million total visitors to that surface to see a real signal. At 5,000 weekly per variant, that is 118 weeks. Two and a quarter years.

Nobody runs a test that long. So what happens: the test runs 4 weeks, hits some random fluctuation, shows a "winner" with 87% statistical confidence (the calculator default), and the operator ships the change. The change is a coin flip wearing a confidence interval.

If you only care about 3% relative lifts, you should not be running split tests. You should be redesigning surfaces meaningfully and measuring period-over-period.

These are the rules we apply on every test before we approve runtime.

• Below 500 conversions per variant in your runtime window? Kill the test. Below this floor, the result is dominated by random visitor variance.

• Runtime over 4 weeks? Kill the test or raise the MDE.

• MDE below 5% relative? Raise it or kill the test. Tests this sensitive will lie to you.

• Baseline below 1%? You need much bigger sample sizes than this article covers. Test higher-funnel surfaces first.

• Multiple tests running on the same traffic? Run them sequentially or use a proper experimentation platform with mutual exclusion. Otherwise you are reading interference noise.

If your test passes those five gates, run it. If it does not, the math is telling you to stop trying to test and start trying to decide.

Most small and mid-DTC sites should A/B test less and design with more conviction.

A few moves that beat underpowered split-testing:

• Cohort redesign tracking. Ship a redesigned PDP. Track the next 30 days against the prior 30 days, controlling for traffic source and AOV. Imperfect, but cleaner than a 40-week split test.

• Qualitative research. Five customer interviews tell you more about your PDP than 5,000 underpowered visits to a button-color test.

• Reading the actual decision being made. The 5-Decision Funnel post linked below is the framework we use. Figure out which of the five shopper decisions is leaking. Fix the decision with intentional creative work. Skip the test.

The math is brutal for most DTC sites: your traffic does not support the tests you want to run, your MDE is too small, your runtime is too long. The honest answer is to test less and design with more conviction.

When you do test, run the math first.

• Baseline CVR, MDE, weekly traffic per variant.

• Sample size \= 16 / (baseline × MDE²).

• Runtime \= sample size / weekly traffic per variant.

• Kill if runtime \> 4 weeks, or sample per variant \< 500 conversions.

Three inputs. One formula. Thirty seconds.

We built the spreadsheet version of this math. Type in your baseline, your MDE, and your weekly traffic. It tells you whether to run the test, how long the test will take, and what MDE you would need to make the test feasible at your current traffic.

[Get the A/B test calculator spreadsheet](/contact). Free. Email-only signup.

For more on how we think about CRO before we ever get to a split-test, read [the 5-Decision Funnel](/blog/the-5-decision-funnel-dtc-conversion). That is the underlying framework. The math here is what we use to decide whether a hypothesis from that framework is worth A/B testing or whether to ship the change outright.