ISSUE 006
The Minimum Viable Product
Build something small, put it in front of people, see what happens. I have heard this advice so many times that I assumed there was a pile of research under it. So I went looking for the pile. What I found was a mapping study that counted thirty-three papers, a free manual that cites almost nothing, and one randomised trial whose most measurable result was that more people quit. None of that makes the advice wrong. It does mean nobody has checked it as carefully as the confidence around it suggests.
An idea worth thinking about
The most repeated advice in business has thirty-three papers behind it
Five researchers went through the published literature on how software minimum viable products are actually built and evaluated. They found 33 papers published between 2013 and 2020. Then they took the findings to two focus groups with twelve industry practitioners to see whether any of it matched working life.
What the 33 papers cover is lopsided. On coming up with what to put in the MVP, the research describes six approaches, mostly informal involvement of end users. On evaluating it afterwards, the emphasis is on usability tests, A/B testing and usage data. On the two questions a business owner actually asks first — can we technically build this, and what will it cost us in time — the authors report that research is limited. Their closing sentence is the one I keep thinking about: more effort needs to be invested into empirically evaluating the existing MVP-related practices.
Three honest limits before anyone over-reads this. It is a preprint posted to arXiv, so it has not been through the journal review process. It is a mapping study, which counts and categorises papers rather than testing whether the practices work. And it is entirely about software, which means that if you install heat pumps or run a dental practice, the thin evidence base described here is not even thin evidence about you — it is thin evidence about somebody else.
I am not telling you to ignore the advice. Launching something incomplete and watching what people do with it is a reasonable thing to try, and the rest of this entry is about how. I am telling you that when a piece of advice is repeated with total confidence by everyone, it is worth ten minutes to find out what is underneath it. Here, the researchers who looked hardest came back and said: somebody should check this properly.
A tool I disappeared into
A free manual for faking it before you build it
Alberto Savoia wrote a short book — about seventy pages, free to download, second edition dated October 2011 — about testing whether anyone wants a thing before you build the thing. It is the most practical catalogue of cheap tests I have found, and it is also almost entirely anecdote. Both of those are worth knowing before you open it.
The techniques have silly names and they work. The Mechanical Turk: a person does by hand what the machine is supposed to do later, so you can find out whether the output is worth anything. The Pinocchio: a dead object the right size and weight, carried around to learn whether you would actually use it. The Fake Door: an ad or a button for something that does not exist yet, to count the clicks. The Provincial: run the whole real thing properly, but only in one small place. Pretend-to-Own: rent or borrow the expensive machine before you buy it. Re-label: put your name on somebody else's product and see whether anyone wants it.
Now the limits, which are considerable. There are no studies in it. The claims are stories, several of them secondhand — the author says of one that he heard it from someone else and probably got a few details wrong. And the statistics it throws around are uncited: ninety percent of mobile apps make no money, four start-ups out of five lose money, eighty percent of new restaurants close within one year. That last one is a well-travelled figure with a thin provenance, and the book offers no source for any of the three. A manual that tells you to test everything should probably not assert numbers it has not checked.
Read it anyway, for the techniques and for one example near the end. A made-up founder called Tom asks about two dozen friends whether they would buy his product. Seventy percent say yes. Three of them buy. Savoia presents it as a hypothetical, so treat it as an illustration rather than data — but it is the exact gap this pillar has spent five entries documenting, and he put it in a free book fifteen years ago.
Learn something
The projects that ran a real-world test launched less often
Two researchers at imec in Belgium went back through 86 innovation projects their own user-research team had run between 2011 and 2018, and sorted them by one thing: whether the project had included a real-life field trial with actual users. Thirty-six had. Fifty had not. Then they looked at what happened to each project.
The projects with a field trial were aborted more often: 28 percent against 22 percent. They were rebooted — sent back to be substantially rethought — far more often: 14 percent against 4 percent. And they reached the market less often: 53 percent against 60 percent. Five of the seven reboots in the whole sample came from projects that had run a field trial.
The authors are careful and the copy should be too. They call the differences suggestive rather than conclusive, because the numbers in each cell are small. The projects were not randomly assigned, so it is entirely possible that harder, riskier ideas were the ones sent for field testing in the first place. And the sample contains no lean startup projects at all, so their comparison between the two approaches is drawn from the literature rather than from their data. The paper is free, published under a Creative Commons licence, and it is the authors reporting an unflattering result about their own method, which I will take over a case study any day.
My reading, and it is a reading rather than their finding: testing in the real world does not appear to raise your chances of launching. It appears to change what you decide. More of the ideas that met real users got killed or rebuilt, and fewer of them shipped. If you go into a test hoping for permission to proceed, that is the number to look at first.
What we’re seeing
Thirty-three of 250 start-ups produced any revenue at all
Three researchers at Bocconi ran a randomised controlled trial on 250 nascent entrepreneurs in an Italian pre-acceleration programme. Everyone got 24 hours of training across eight sessions. Everyone was taught the Business Model Canvas and the minimum viable product. The only difference was that the treatment group was also taught to use those tools scientifically: state the theory behind the choice, turn it into a hypothesis, test it, and then evaluate honestly. The control group was taught the ordinary trial-and-error version. Then the researchers phoned all 250 of them eighteen times over 66 weeks.
The number that stopped me is not the treatment effect. It is the baseline. In the paper's own words, 33 of the 250 start-ups produced some revenue in that period — sixteen in the treatment group and seventeen in the control. Separately, 105 of the 250 abandoned their ideas outright, which is roughly 42 percent by my arithmetic rather than theirs. Sixty-six weeks of a structured programme, with training, with instructors, with researchers ringing every fortnight, and seven out of eight of these businesses had not yet taken a single euro from anybody.
What the training changed was mostly the quitting. Treated entrepreneurs were more likely to exit, by about ten percentage points in the cross-section, and they exited earlier. They were more likely to pivot exactly once. Revenue moved in their favour but weakly: 1,573 euros more on average, which the paper reports as not statistically significant, and a weekly revenue flow effect that only clears the ten percent threshold. Be careful with the revenue result. The draft contradicts itself on which group the sixteen and the seventeen belong to, and the authors themselves describe their mechanism evidence as suggestive.
Two disclosures, because they govern the weight. This is a preliminary draft from May 2020, not a finished peer-reviewed paper, and it should be read as one. And the trial did not test whether minimum viable products work — both groups were taught the MVP. It tested whether being taught to treat your assumptions as hypotheses changes what you do. The answer was yes, and what it mostly changed was that people stopped sooner.
Which is the uncomfortable version of the advice in the first three slots. Launch something small and watch what happens, and the most likely thing that happens is you find out it does not work and you stop. That is a good outcome, cheaply bought. It is just not the outcome anyone is selling when they tell you to ship early.
So here is the version for a business with customers and a payroll. Pick the thing you have been meaning to build. Before building it, sell it: name the real price, name the date, and ask three people who are not your friends to commit. If all three say yes, build it. If all three find a reason not to decide today, you have just saved yourself a quarter, and you learned it the only way this pillar has found that holds up, which is that somebody either paid or they did not.
One question
What have you been meaning to build but have not sold yet?
One line is enough. If you have already taken money for it, say that instead — that is the more interesting answer.
More in this pillar as it's written.
— Shana
Sources link to the original publisher, never to a summary.