ISSUE 002
"Would you buy this?" is least predictive for new products
The last entry in this pillar found that simulated customers are unreliable at predicting what any particular person will do. The obvious response is to go ask real people instead. So I went looking for what happens when you do that, and the research is less comforting than I expected.
An idea worth thinking about
Intent surveys work best on products that already exist
Morwitz, Steckel and Gupta set out to find when stated purchase intentions actually track purchases, running a meta-analysis and then a second study across sixty product tests. They found the relationship holds up far better under some conditions than others, and the list of conditions is uncomfortable reading for anyone validating an idea.
Intentions correlate more strongly with purchases for existing products than for new ones. For durable goods than for things people buy repeatedly. Over short time horizons rather than long ones. When people are asked about a specific brand or model rather than a product category. When the measure is trial rate rather than total market sales. And when intentions are collected comparatively rather than in isolation.
Read the first item again. The method is least trustworthy precisely for new products — which is the only kind you have when you're deciding whether to build something. Asking a hundred people whether they'd buy your thing is a technique borrowed from forecasting demand for products already on shelves, and it degrades exactly where founders reach for it hardest.
The moderators also double as instructions. Ask about your specific product, not the category. Ask about the next month, not someday. Put it beside real alternatives at real prices. That won't make the answer reliable, but it stops it being meaningless.
Read the study
A tool I disappeared into
A free nine-section manual for building a survey that actually measures intent
If you're going to ask people about future behaviour, there is a proper way to do it, and somebody wrote the manual. This one was produced for health-services researchers and sits in City, University of London's open access repository: nine sections covering how the theory works, how to construct the questionnaire, how to measure intention, how to measure each predictor separately, how to manage the project, and how to analyse what comes back. Appendix A includes worked example materials for the preliminary elicitation study and for scoring.
The part that changed how I think about survey questions is the front end. Before you write a single item, you run an elicitation study — open interviews to find what this specific population actually believes about the behaviour: perceived advantages and disadvantages, whose approval matters, what would get in the way. Only then do you write closed questions, using their words.
Which means a well-built survey is downstream of conversations, not a substitute for them. Most founder surveys skip the elicitation step entirely and go straight to questions the founder invented, which is how you end up measuring your own assumptions with a decimal point attached.
Open the manual
Learn something
Icek Ajzen publishes his own instructions, for nothing, at a university URL
Ajzen is the source of the theory of planned behaviour, which is the framework most intent research runs on, and he hosts a short guide to building questionnaires from it on his university page. It's a PDF, not a course, and it will take an evening.
The one idea worth the download is TACT: any behaviour you want to predict has to be defined by its Target, Action, Context and Time, and every question you ask must match that definition. "Would you use an AI tool?" fails on all four. "Would you pay $40 a month for this scheduling tool for your own shop, starting in January?" is a question whose answer can be checked against reality. He also recommends five or six items per construct rather than a single question, and recontacting people afterward to see whether they did the thing — which is the step that turns a survey into evidence.
Fair warning that the writing is academic and the examples are about exercise and medical compliance. The method transfers cleanly; the prose doesn't.
Download the guide
What we’re seeing
The average correlation between what people say and what they do is 0.53. The confidence interval runs from 0.15 to 0.92.
Morwitz and colleagues cite a broad meta-analysis by Sheppard, Hartwick and Warshaw that put the frequency-weighted average correlation between intention and behaviour at 0.53. That sounds like a usable relationship. The number underneath it is the one that matters: the 95 percent confidence limits ran from 0.15 to 0.92.
A range that wide means the average is close to useless as guidance. Somewhere in the literature, stated intent predicts behaviour almost perfectly. Somewhere else, it is barely better than a coin weighted slightly your way. Without knowing which situation you're in, an intent survey returns a number with no error bar you can defend — and the moderators above suggest new products sit at the wrong end of that range.
Independent work points the same direction. A cross-cultural study of online purchase intention and actual purchase behaviour, published open access, reported a clear discontinuity between the two and concluded that intention is an inadequate path for predicting what a customer actually does.
I want to be careful about the conclusion, because "surveys are worthless" is not it, and neither is "only talk to people." What the spread actually argues for is ranking your evidence. Somebody saying they would buy it is the weakest signal. Somebody describing what they did last time they had the problem is better, because it's memory rather than forecast. Somebody putting down money, or a deposit, or their calendar time, is better still. The research doesn't tell you to stop asking. It tells you what asking is worth, which is roughly the floor of your evidence rather than the ceiling.
Provenance note, since this pillar keeps insisting on it: the Sheppard figures reached me through the Morwitz paper rather than the 1988 original, so treat them as well-sourced secondhand rather than verified at the root.
Purchase intention and purchase behavior online, open access
One question
What's the strongest evidence you have that someone wants it?
A survey, a conversation, a waiting list, a deposit, a paying customer. Reply with which one — and be honest about the top of your list.
More in this pillar as it's written.
— Shana
Sources link to the original publisher, never to a summary.