ISSUE 010
What Twenty-Five Replies Can Tell You
Here is the situation I keep being asked about. You email 400 customers, 25 reply, and someone wants to know whether that is enough to act on. The honest answer has two halves and only one of them is about arithmetic. The arithmetic half is easy and bleak. The other half is that the twenty-five people who replied are not a small version of your customer list, they are the subset of it that replies to emails, and there is a serious body of research on exactly how much that matters. I went and read some of it.
An idea worth thinking about
The response rate does not tell you how wrong the survey is
The instinct is that a 6 percent response rate is bad and a 60 percent one is fine, and that the number is a kind of quality score. Michael Davern, writing in a 2013 journal article that is free to read in full, argues that this instinct fails on its own terms. His sentence: response rates lack validity in that there is not even a moderate correlation with nonresponse bias. He is citing earlier work by Robert Groves for that, and he is not claiming bias does not exist — he is claiming the rate does not measure it.
His second point is stranger and I had to read it twice. The response rate is not even a stable number. Citing an analysis by Skalland, he notes that for one survey — the 2008 Behavioral Risk Factor Surveillance System — you could calculate response rates anywhere from 26.9 percent to 65.6 percent depending on what you assume about the cases whose eligibility you never established. Same survey, same fieldwork, same year. The thing hinges on how you count the people you could not classify, which is a judgement call rather than a measurement.
Then he counts what the profession talks about. Across the AAPOR conference programmes from 2010 to 2012, the terms response rate or nonresponse rate appeared 767 times in paper titles and abstracts. The terms for bias appeared 218 times. His recommendation is to spend less effort chasing the rate and more on actually analysing whether respondents differ from non-respondents — and he points out that money spent dragging a response rate upward could instead buy a larger sample at a lower rate, which in one of his own studies produced no statistically different result across 33 estimates.
The paper is thirteen years old and it is an argument rather than an experiment. But the practical translation is the useful bit, and it is not what you would guess. Stop reporting your response rate as if it settles anything. The question that settles something is whether the people who replied differ from the people who did not, on something you can check.
A tool I disappeared into
A free calculator that will not let you fudge the denominator
The American Association for Public Opinion Research publishes a response rate calculator, free, no account. There is an online version that walks you through a series of questions, and a download link to the Excel workbook — version 5.1, dated 14 April 2023 — with a tab for every sample type including one called List Samples, which is the closest thing to a customer list.
What makes it worth an hour is not the arithmetic, which is division. It is that the workbook refuses to let you be vague about what you are dividing by. Every case has to be assigned a disposition: completed, partial, refused, non-contact, ineligible, unknown eligibility. That last bucket is where the trouble lives. Of your 400 emails, how many bounced, how many went to people who are no longer customers, how many landed in a shared inbox that nobody reads? Each answer moves the denominator. The workbook also makes you estimate a quantity it calls e, the proportion of the unknown-eligibility cases that were probably eligible, and it tells you its default method is conservative and may underestimate your rates.
The limits are stated by AAPOR itself and they are not small. The online calculator says it is meant for probability household- and individual-level cross-sectional surveys, and explicitly not for longitudinal surveys or panels, not for establishment surveys, and not for non-probability samples. Your customer list is a non-probability frame, so the formulas do not strictly apply to it. The online version is also labelled as being in beta testing.
I would use it anyway, for one reason. Most small businesses do not have a wrong response rate, they have no denominator at all — the email went to whoever was in the list, and nobody knows how many of those addresses still belong to a customer. Working through the dispositions takes an afternoon and usually teaches you more about the state of your contact list than about the survey.
Learn something
A free 408-page survey manual, and the one sentence in it I would tape to a wall
Statistics Canada publishes a complete survey manual, catalogue number 12-587-X, 408 pages, free to download. It was written in October 2003 for a statistical training programme and reissued electronically in 2010, so it is old and it is Canadian, and neither of those matters much because survey error has not changed nationality or decade.
Section 3.4.2.3 is three pages on nonresponse and it contains the sentence I would tape to a wall: while increasing the sample size reduces the sampling variance and therefore corrects for nonresponse that is randomly distributed, it does not reduce any bias introduced by systematic nonresponse. Read that against the thing every small business does next, which is send the survey to more people. Emailing 4,000 instead of 400 narrows the random error and does precisely nothing to the systematic part. If the quiet, satisfied, busy customers are the ones not replying, they will not reply at ten times the volume either.
The chapter is blunt about why: nonrespondents often have different characteristics from respondents, and if the nonresponse rate is high, bias can be so severe as to invalidate the survey results. Its example is a literacy survey in which most non-respondents are illiterate. The small-business version writes itself. A satisfaction survey whose non-respondents are the people who have quietly stopped calling you.
There is also a table in chapter 4 that is worth ten seconds. Comparing collection methods, it rates personal interviews High on response rate, telephone Medium-high, and self-enumeration — the posted or emailed questionnaire somebody fills in alone — Low. Self-enumeration is also the cheapest, which is why everyone uses it. You are not doing surveys badly. You are doing the cheap kind, which is the one with the worst response, and the manual says so without any fuss.
What we’re seeing
Sixty-one point three percent replied, and the number was still 2.8 percent out
There is a survey in this country that can check its own homework against tax records, and in September 2026 the Census Bureau published the check. The Current Population Survey's annual social and economic supplement is the source of the official income and poverty figures. Two Census researchers, Adam Bee and Jonathan Rothbaum, took the people who did not respond to it and matched the addresses to administrative records — American Community Survey data, Social Security's Numident file, the 2010 Census, and IRS returns including W-2s and 1040s. Then they re-weighted the survey to account for what the non-respondents looked like, and compared the two answers.
Start with the response rate. The weighted response rate for the 2026 supplement was 61.3 percent, against 62.8 percent in 2020. That is a government survey with interviewers, follow-ups, a legal basis and a budget, and four people in ten still do not answer it.
Now the bias. For 2017, 2018 and 2019 the survey-only estimates and the adjusted estimates were not statistically different, which is to say the ordinary weighting was doing its job. Since 2020 they have diverged: survey-only income estimates have run about 2 to 4 percent higher than the adjusted ones, and this year the bias at the median was 2.8 percent. The poverty rate came out 0.6 percentage points lower than the adjusted figure, a statistically significant gap, in line with the 0.3 to 0.5 points seen since 2020. Put plainly, the people who stopped answering were not a random slice, and the survey has been quietly reading a little richer than the country it describes.
Two honest caveats, both the authors' own. The adjusted numbers are a benchmark for evaluating bias, not a definitive measure of the truth, because they can only correct for the differences that show up in the linked records. And administrative data for 2025 was not available, so the 2026 adjustment used 2024 records. They are careful about this in a way that the people quoting survey results at you usually are not.
Here is why this sits in a pillar about what customers actually do. The best-resourced survey in the United States, at a 61.3 percent response rate, with tax records to check itself against, found a measurable bias it could name to one decimal place. You emailed 400 customers and 25 replied. That is a 6 percent response rate, and you have no administrative records, no linked file, and no way to find out what the other 375 think. By my arithmetic, 25 responses alone put a plus or minus 19 point band around any yes-or-no question in that survey, before non-response bias is considered at all. Twenty-five replies is a conversation, not a measurement.
So treat it as one. Read the replies, pick up the phone to the people who wrote something interesting, and then do the one thing that is available to you and not to the Census Bureau: check your responders against your own books. Did they spend more than the average customer last year? Have they been with you longer? Are they concentrated in one service line? You already hold that data. If the 25 look different from the 375 on things you can see, assume they differ on the things you cannot.
One question
When you last surveyed customers, how many of them answered?
Out of how many asked. Two numbers, one line — and if you do not know the second one, that is the more interesting answer.
More in this pillar as it's written.
— Shana
Sources link to the original publisher, never to a summary.