ISSUE 008
A/B Testing When You Don't Have the Traffic
Somebody told a business owner in Cape Girardeau to A/B test their website. It is good advice if you have a hundred thousand visitors a month and expensive nonsense if you have eight hundred. Nearly every article written about A/B testing silently assumes the first case, and almost none of them show the arithmetic that would let you work out which case you are in. So this entry is the arithmetic. It is not encouraging, and the second half is what to do instead.
An idea worth thinking about
Checking the test every day turns a five percent risk into twenty-six
Evan Miller wrote the clearest short piece I have found on the single most common way small A/B tests go wrong, which is looking at them. He simulates a site with a 50 percent baseline conversion rate and a new logo that does absolutely nothing. The experimenter watches the results and stops as soon as the difference reaches five percent significance, or gives up at 150 observations. That is how almost everyone runs a test.
The false positive rate in that simulation is 26.1 percent. More than five times the five percent the experimenter thinks they are risking. The logo did nothing. Roughly a quarter of the time, the dashboard told them it did something.
The reason is that a significance level assumes you decided your sample size in advance and looked once. Every additional look is another chance for noise to cross the line. Miller gives the correction table: if you intend to peek twice, you need to see 2.2 percent significance before you believe it. Five peeks, 1.4 percent. Ten peeks, 1.0 percent. His fix for experimenters is blunt — fix the sample size before you start and do not stop early — and his fix for software builders is that dashboards should stop reporting significance for tests that are still running.
He is not disinterested: he sells a desktop statistics package and links his own sample size calculator from the article. Neither affects the simulation, which anyone can rerun. And the problem is strictly worse at small scale. A business with a few hundred visitors a week is running a test for months, which means months of opportunities to check whether it is working yet. The temptation is not a character flaw. It is a design flaw in how the test was set up.
A tool I disappeared into
A clinical trial calculator that does your conversion arithmetic
ClinCalc publishes a sample size calculator built for medical research, and it is the cleanest free one I found for this job. You pick a study design, enter the anticipated value for group one and group two, set alpha and power, and it tells you how many subjects each arm needs. The page was last updated in June 2024 and cites Rosner's Fundamentals of Biostatistics as its reference. There is no sign-in and nothing to buy on the page.
To use it for a website or an offer, translate. Group one is your current conversion rate — say four percent of the people who ring you book a job. Group two is the rate you would need to see before you changed anything. Alpha stays at 0.05 and power at 80 percent, which the page notes are the conventional choices in medical literature and are equally conventional online. The number that comes back is per group. Double it for the total, then divide by your weekly traffic to get the honest answer, which is a number of weeks.
Do that once and the main lesson arrives on its own: it is not the calculator that is the obstacle, it is the gap between your current rate and the rate you entered. Put in a five percent relative improvement and the number is absurd. Put in a doubling and it becomes a few months. The calculator is a machine for discovering which questions you are allowed to ask.
The limits, fairly. It was built for clinical trials, so the vocabulary is medical — incidence, enrollment ratio, endpoints — and you have to do the translating yourself. It does not show its formulas, so you are trusting the reference. And it computes a single fixed sample size, which means it will not help you with sequential designs where you are permitted to stop early. For the question most small businesses actually have, which is roughly how long would this take, it is more than enough.
Learn something
At Bing, one experiment in five hundred was worth doing
Four experimenters from Microsoft and LinkedIn wrote down what a decade of running online experiments had taught them, and published it at a 2014 conference. It is readable, it is free, and two of its seven rules reset expectations permanently.
Rule two: changes rarely have a big positive impact. Successful changes, the ones that worked, typically improve key metrics by 0.1 to 1.0 percent once the effect is diluted to the whole business. Not ten percent. A tenth of one percent to one percent. Rule one says the same thing from the other side: at Bing, roughly one experiment in 500 met the bar of high return and replicable positive impact.
Those two facts combine into the sentence that should be printed on every A/B testing dashboard. The authors work it through: if a third of your ideas are genuinely good, a statistically significant result has an 89 percent chance of being true. If only one in 500 is good, a statistically significant result has a 3.1 percent chance of being true. Same test, same p-value, wildly different meaning, and the difference is how good your ideas were before you tested them.
Rule three is the one aimed at the reader of marketing blogs: your mileage will vary, published case studies vary enormously in quality, and many do not check significance at all. Which brings us to the disclosure. These authors ran experiments at Microsoft and LinkedIn, where traffic is effectively unlimited and a test costs a day. That is precisely the position almost all A/B testing advice is written from, and almost none of it says so. Read the rules, take the effect sizes, and then notice that if the honest wins are under one percent and you can only detect enormous changes, the only tests worth running at your size are tests of things that might change everything.
What we’re seeing
Under 122,000 visitors in each version, to detect a five percent lift
The standard reference on this is a 2009 survey and practical guide in Data Mining and Knowledge Discovery, and it does something the blog posts do not: it puts the numbers in. The minimum sample size formula is sixteen times the variance divided by the square of the change you want to detect. That assumes 95 percent confidence and 80 percent power, and n is the number of users in each variant, not in total. Want 90 percent power instead, replace the sixteen with twenty-one.
Then the worked example, which is the part to read twice. An e-commerce site where five percent of visitors buy, buyers spend about 75 dollars, so the average visitor is worth 3.75 dollars with a standard deviation of 30. To detect a five percent change in revenue, the paper says you will need over 409,000 users. Switch to measuring conversion instead of revenue, which has far lower variance, and you need less than 122,000 users — a factor of 3.3 saved by changing what you measure, nothing else. And if you only care about detecting a 20 percent change rather than a five percent one, the requirement drops by a factor of sixteen, to 7,600.
Now the arithmetic the source does not do, which is mine. Two hundred visitors a week, split evenly, is a hundred per version per week. At 122,000 per version you are running that test for 1,220 weeks. Twenty-three years. At the 20 percent threshold, 7,600 per version is 76 weeks, which is eighteen months — long, but for the first time a real number. The paper also recommends running for at least a week or two and in whole-week multiples to absorb day-of-week effects, advice that is quaint at this scale.
Two caveats on the source. It is from 2009 and written for large websites, by people who had them. And the example's five percent conversion rate and 75 dollar order are illustrative, not a benchmark — plug in your own and the numbers move. What has not moved in sixteen years is the shape: sample size scales with the square of how small an effect you want to see, which is why halving the effect you are chasing quadruples the traffic you need.
So here is what to do instead, in order. Only test changes large enough to show up — the price, the offer, the headline, whether somebody answers the phone on the second ring. A 20 percent swing is detectable at a small scale and a five percent one is not, which means small refinements are simply not knowable at your size, and the honest response is to decide them on judgement and say so, rather than run a test that cannot see. Measure the thing with the lower variance: did they convert, not how much they spent. Write down your sample size and your end date before you start, and then do not look. And if the test would take twenty-three years, you have not failed at analytics. You have been handed a very clear answer about which tool this is.
The alternative, which this pillar keeps arriving at, is the one small businesses are actually equipped for. You cannot detect a five percent lift. You can ring the eleven people who did not book and ask what happened. That is not statistics and it does not generalise, and it will tell you more this quarter than any split test you are able to run.
One question
How many people land on your site or walk through your door in a week?
A rough number is fine. It is the one input that decides whether any of this arithmetic is available to you.
More in this pillar as it's written.
— Shana
Sources link to the original publisher, never to a summary.