ISSUE 007
Customer Interviews That Aren't Leading
Entry 002 in this pillar established that what people say they will buy is a weak guide to what they buy. The obvious follow-up question is whether you can ask better. It turns out there is a whole research literature on this, built by people who run national surveys and who have been measuring how much a question changes its own answer since the 1940s. None of it is written for business owners, which is a shame, because the effects they measure are bigger than most of the things we try to learn from a customer conversation.
An idea worth thinking about
One added clause moved the answer twenty-five points
Pew Research Center keeps a public guide to how it writes questions, and the useful part is that it shows its working. In January 2003 Pew split a sample in two. Half were asked whether they favoured taking military action in Iraq to end Saddam Hussein's rule: 68 percent in favour, 25 percent opposed. The other half got the identical question with one clause added — even if it meant that US forces might suffer thousands of casualties. Support fell to 43 percent and opposition rose to 48. Same topic, same people in aggregate, same day. One clause.
The format matters as much as the wording. In a 2008 poll, 58 percent named the economy as the most important problem when it appeared on a list of options. When the question was left open and people had to volunteer an answer, only 35 percent said the economy. And 43 percent of the open-ended respondents gave an answer that was not on the closed list at all, against fewer than ten percent of the closed-ended group. The list does not just measure opinion. It supplies it.
Two biases in that guide have your name on them. Acquiescence: people tend to agree with a statement put to them, and Pew reports the tendency is stronger among less-informed respondents and stronger again when an interviewer is physically present. Social desirability: people understate the things they are not proud of and overstate the things they are, and that too is worse with an interviewer in the room. Now consider the structure of a customer interview. There is an interviewer present. The interviewer is the owner. The owner built the thing and is visibly hoping. You have assembled, by accident, close to the worst-case conditions for both biases at once.
Worth saying, because almost nothing else in this entry can: Pew is a nonpartisan research centre and is not selling you a method, a platform or a consulting engagement. The guide is undated on its face and the experiments it cites run from roughly 2003 to 2008, so the examples are old. The mechanisms are not.
A tool I disappeared into
A free library of questions that were tested on real people and failed
Q-Bank is a searchable library of question evaluation research. Before the federal statistical agencies put a question into a national survey, they test it — they sit people down, ask the question, and then ask what the person thought it meant. Q-Bank publishes the reports. You can search by question or by report, browse by survey, by topic and by contributing agency, and read the write-ups. Recent reports on the site are dated 2023 through 2026 and credited to the National Center for Health Statistics.
The reason to open it is not the questions. It is watching the failure mode. You read a question that looks completely clear, then you read the transcript of someone reading it and taking it to mean something else entirely, and you start to see how much work a respondent is doing to guess what you want. Nobody does this for business questions. This is the nearest free equivalent.
The limits are straightforward and they are real. The subject matter is federal health surveys — disability, education access, attention deficit, chronic conditions — so the content has nothing to do with whether anyone wants your new service line. What transfers is the mechanism, not the material. The site does not publish a total count of what it holds, so I cannot tell you how big it is. Browsing needs no account; submitting a report of your own goes to a login page.
If you have twenty minutes, open one cognitive interview report and read the section where respondents explain their own answers. Then go back to the question you have been asking your customers and read it the way a stranger would. That is the whole exercise, and it costs nothing.
Learn something
Two leading versions of one question, thirty-seven points apart
In February 2023 YouGov published two of its own split-sample experiments, which is a generous thing for a polling firm to do in public. In the first, respondents were randomly given one of three versions of a question about whether the UK should leave the European Convention on Human Rights.
The version leading towards leaving — the UK should leave, so that it can better control who comes in and out of the country — got 38 percent agreeing and 41 disagreeing. The version leading towards staying — the UK should remain, so that everyone can enjoy strict legal protection of their fundamental human rights — got 61 percent agreeing and 21 disagreeing. The neutral version simply asked whether the UK should withdraw or stay, and got 20 percent for withdrawing and 48 for staying. The two leading versions sit 37 points apart on the same question, and each of them lands 13 to 18 points away from the neutral one.
The second experiment is the subtler lesson. Asked do you agree that the government is not doing enough to help Ukraine, 22 percent agreed. Add the words or disagree to balance it and agreement rose slightly to 24 percent, a change YouGov says sits inside statistical uncertainty. But offer three options instead — too little, too much, or about right — and only 17 percent chose too little. The agree-or-disagree format ran five points above the version that made people choose.
The disclosure is obvious but say it anyway: YouGov is a market research company and this article is, among other things, a demonstration of why you should hire people who know how to write questions. The experiments are also published without sample sizes or fieldwork dates, which from a research firm is a genuine gap and makes the smaller differences hard to weigh. The large one does not need weighing. If you want to know what a customer thinks, do not hand them a statement to agree with. Hand them two statements and make them pick.
What we’re seeing
Five percentage points, or fifteen, depending on whose panel you ask
Here is the inconvenient half of everything above. Acquiescence bias — the tendency to agree with whatever you are handed — is real, well documented and apparently not a fixed quantity. Three researchers published a short open-access letter in Political Analysis in 2026 that reanalysed an earlier survey experiment and then ran it again twice, preregistered.
The design is clean. Respondents were randomly given either a positively worded or a negatively worded version of the same true-or-false claim. If people simply agree with what is put to them, the positively worded version will collect more agreement, and the gap is the bias. In the original study and in one 2025 replication on the Lucid and Cint panels, with 2,193 respondents, the authors report the effect running between 10 and 15 percentage points on average. In the second 2025 replication, on the higher-quality Bovitz-Forthright panel with 3,036 respondents, the same effect came out roughly two to three times smaller: about five percentage points.
The original study also found that the bias was larger among less-educated respondents. The high-quality panel showed no such association at all. And attention checks did not rescue it — among Lucid respondents who passed every attention check, the education pattern was still there. The authors' conclusion is a warning about survey quality rather than about acquiescence: how big this bias looks depends substantially on who you recruited and how carefully they are answering.
Three things follow for a business, and the first is mine rather than theirs. None of these studies interviewed a customer. They are about online survey panels answering political and factual claims, so the mechanism transfers and the magnitudes do not — I am not going to tell you that your customers agree with you five or fifteen percent more than they mean to. Second, the direction is reliable even when the size is not, and it always runs the same way: towards agreement. Third, and this is the part worth keeping, a bias whose size swings by a factor of three between two decent panels is not something you can correct for afterwards. You can only avoid creating it.
Which makes the practical advice unusually short. Stop asking whether they would use it. Ask what they did the last time they had the problem, what it cost them, and what they did instead. Give them two statements and make them choose rather than one statement to nod at. Ask the question before you describe the product, not after. And when you have asked everything, ask for the sale, because the only answer in this entire pillar that has not needed a correction factor is somebody handing over money.
One question
What is the last question you asked a customer that they could only really answer yes to?
Reply with the question, not the answer. Most of us have one and have never looked at it written down.
More in this pillar as it's written.
— Shana
Sources link to the original publisher, never to a summary.