When I started working in market research at what was then called BMRB, online surveys didn’t exist. Interviewing was mainly carried out face-to-face or by telephone.
Online interviewing has, of course had quite an impact on survey research, making it cheaper and usually faster. However, one downside to the growth of online surveys has been a decline in piloting questionnaires. By piloting I mean actually testing questionnaires with respondents before starting the survey proper.
At BMRB, when carrying out face-to-face and telephone fieldwork we almost always used to pilot questionnaires. Execs would accompany interviewers in home or on the street or listen in to telephone interviews to check everything was working as intended. The interviewers had a big part to play as their experience helped us identify which areas of the questionnaire needed tweaking. We gained an understanding of how the questionnaire worked and how respondents interacted with it. This understanding was invaluable when it came to looking at the results and interpreting them.
Pilot surveys are also used to test survey length. Both clients and researchers can have a tendency to try to squeeze as many questions as possible into a survey thinking that this will result in greater value for money. However, longer questionnaires can be counter-productive in terms of quality of response. Pilot surveys often provide the focus needed to streamline lengthy surveys.
In terms of testing how questionnaires are working, it is now rightly common practice to check online surveys by looking at the data after a few interviews have been completed. But this a test of logistics rather than the effectiveness of the questionnaire. We know if respondents are being routed to the correct questions but we don’t know how they are interpreting the wording of questions and if they have any difficulties in understanding. Therefore these data checks are a complement rather than a replacement for piloting.
Piloting questionnaires needn't be something confined to market research history books. Before carrying out any survey, be it on or offline you can test the wording face-to-face or over the phone with colleagues, friends or family. Piloting questionnaires on clients can be a great way to get them to understand the strain placed on respondents by some surveys as well as involving them in the design process.
The process of piloting gives you the confidence that you are measuring what you intended to measure and that respondents are getting the best survey experience possible. As mentioned earlier it will also help when interpreting results as you will have a better understanding of the interview process from the respondents’ point of view. The benefits are clear. Piloting may have fallen out of favour but it’s as important now as it ever was.