Rank bad questions: how not to ask ranking questions

I've seen a couple of questions recently which have asked respondents to rank long lists of attributes. An example is shown below:



Ignoring the fact that I’ve never looked for any "benefit" in a loyalty rewards programme for an online travel agency, and I’ve no idea which criteria I should be using for ranking these benefits, there are several reasons why I don’t think questions like this should be asked:

  • They are very time consuming for respondents and very boring. Understanding and evaluating a long list of different benefits of something you’re probably not interested in is hard work.
  • It is likely that the individual respondent thinks that some of the benefits are irrelevant, unappealing or pointless. The respondent is therefore ranking things they care little about. Results may show one benefit ranked fifth and one ranked ninth but in reality the respondent cares little for either. The question is unlikely to yield meaningful results, especially for the lower ranked benefits.
  • The data set you get from a question like this is quite complex and difficult to interpret. Even if the question works, do you really need this depth of information?

Ranking questions can work if you lighten the respondent’s load. For example in this case you could ask which benefit the respondent felt was most appealing and then perhaps ask which would be second and third most appealing. I prefer not to go beyond asking respondents for their top 2 answers, perhaps their top 3 at a push. You will still be able to provide a rank order for all benefits when it comes to analysis, one that will doubtlessly be more meaningful than asking respondents to order all the attributes.

Submit comment

Recent Posts