We know it’s wrong but market researchers are still producing long, boring surveys. How can we change?

In their book Super Freakonomics, Stephen Levitt and Stephen Dubner discuss a common problem facing hospitals, doctors not washing their hands as often as they should. In 1999 the Institute of Medicine estimated that between 44,000 and 98,000 Americans a year die from avoidable hospital errors. The leading cause of these errors was wound infections, easily prevented by doctors washing their hands. The doctors knew that not washing their hands put patients at risk. Awareness wasn’t a problem but, despite knowing the potential consequences, doctors still weren’t washing their hands as often as they should. Various schemes were tried including posters, email messages, wall-mounted disinfectant dispensers and even handing out $10 Starbucks cards to doctors seen washing their hands properly. None of them shifted behaviour significantly. However, an effective solution was found in one hospital, the introduction of screensavers on all computers showing pictures of grisly bacteria-laden handprints. Hand-hygiene compliance shot up.

Like the doctors of the past the vast majority of survey designers are guilty of burying their heads in the sand and ignoring the consequences, the real threat they pose to the long-term viability of survey research. There’s probably not a month goes by when I don’t read a blog by a fellow market research professional that calls for the industry to write shorter, more engaging surveys. Evidence has been produced that shows that response rates fall and drop-out rates rise when questionnaire length goes up. Despite this weight of opinion we don’t seem to be doing anything differently. The highest profile, most reputable market research companies are still sending out the same old gruelling questionnaires. We know it’s wrong but we still do it.

For the hard pressed survey designer there is little incentive to do things differently. Producing shorter, more focussed, more engaging studies requires additional care, time and skill, costs that few are seemingly willing to bear. There are rarely any immediate, personal consequences for bad survey design. Data quality might be compromised, but who notices and who cares? Respondents may be less inclined to respond to future surveys but that’s not a concern for now. In my experience, the gatekeepers to respondents, panel companies and fieldwork agencies have low levels of quality control. Few insist on any particular standards before subjecting respondents to the latest elongated gridfest.

For the perpetrator, bad survey design is a victimless crime. For online surveys the designer doesn’t see or feel the respondents’ pain. For face-to-face and telephone surveys, interviewers bear the brunt of respondents’ ire and boredom not the execs who design the surveys.

Shouldn’t our efforts now be concentrated on how to change behaviour rather than repeatedly highlighting the issue? Awareness alone won't change behaviour. What incentives can change the way survey designers act? What is our equivalent of the dirty hand screensaver? Let’s start the debate.,,


Steven D. Levitt and Stephen J. Dubner: SuperFreakonomics: Global Cooling, Patriotic Prostitutes, and Why Suicide Bombers Should Buy Life Insurance (Penguin, 2010)


2 comments on “We know it’s wrong but market researchers are still producing long, boring surveys. How can we change?

  1. Great article and a stunning blog in terms of content. Well done!

    Hard to disagree with your points of course, but I can’t help thinking old fashioned economics has a lot to do with it. If a client asks me for a 45 or 60 minute online survey I’d first of all suggest splitting into two if at all possible. From a quality perspective we all know that’s best. However, I find that the fieldwork / panel companies encourage me to run it as a one-off and publicise the standard “start from where you left off if you need to take a break” functionality with online surveys. They want me to do that because it’s a lot cheaper than quoting for x2 shorter surveys (and they want to win the job). And of course ultimately I want to make sure that my proposal to the client is at least in with a chance. And let’s face it, price is the greater part of that (in the world of online quantitative research anyway).

    • austinresearch Post author

      Thanks for the kind words John. Much appreciated.

Submit comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Recent Posts