Avoiding response bias due to answer order - randomise and rotate

How does answer order result in bias?

As discussed in our previous blog on the importance of balancing scales, response bias can be caused by the order in which answer choices are presented. As an example here’s a simplistic question asking respondents why they use their first choice supermarket:

And why is this supermarket your first choice? Please select all answers that apply to you:

Convenient location
Best prices
Good range of products
Good service
Easy car parking
Other
Not sure

Respondents tend to want to get through surveys as quickly as possible. Surprising as it may seem some would rather be spending their time doing something else. Respondents, therefore, may have a tendency just to look for the first answer that makes sense to them. In this case if they do use their first choice supermarket because of its convenient location they may pick this answer and go to the next question without paying much attention to the rest of the answer list. If answers are presented in the same order for each respondent we might then get a bias towards those answers at the top. Their importance may be over-stated when we look at the final data.

The instruction "Please select all answers that apply to you:" is also important here as it is it makes it clear to respondents that we are looking for as many answers that are relevant to them and not just the first one that comes into their mind.

Answers at the bottom of an answer list might also be chosen disproportionately. This is especially true of telephone surveys where interviewers read out the complete answer list. Respondents then tend to better remember the answers they heard most recently (those at the bottom of the answer list) potentially biasing respondents to choose those answers.

Randomisation

The way to avoid these biases is to randomise the answer list, programming your survey so that each respondent is presented with answers in a random order.

In the example above the “other” and “not sure” answers should not be randomised along with the rest. Respondents should only read these answers after they have read the other answer alternatives. It doesn’t make sense for respondents to be asked for an “other” or a “not sure” answer is they haven’t been given the chance to consider all the alternatives.

Rotation

In some instances rotation may be more suitable than randomisation in avoiding order bias. For instance, take the question below:

How likely are you to go to a supermarket tomorrow?

Very likely
Quite likely
Not very likely
Not at all likely

In this case randomising the answers is probably going to cause them to appear muddled and nonsensical to respondents. However by rotating answers the scale still makes sense:

How likely are you to go to a supermarket tomorrow?

Not at all likely
Not very likely
Quite likely
Very likely

Typically rotating answer lists in a survey means the first respondent completing the survey is presented with the first answer list, the second respondent with the alternative, the third with the first answer list again and so on. In this way half of respondents see each version of the answer list.

Unless there is a logical order to the answer list I use randomisation rather than rotation. With rotation the same answers are still at the extremes of the answer lists and, therefore, possibly still subject to order bias.

When to avoid answer order bias

Not every question needs to be prepared to avoid answer order bias. Factual questions such as gender, age and income don’t need to be randomised or rotated as the respondent’s answer will not be biased by the answer list; only one response will ever apply to them. Resist the urge to over-engineer questionnaires and only randomise or rotate those answer lists that might be subject to order bias.

Click here for our guide on how to randomise and rotate answer lists in SurveyMonkey.

Submit comment

Recent Posts

Twitter