Mitigating the survey blues…
By David Bomphrey
Online surveys have had a bit bad press for a while now and some of the criticisms are valid such as: the data is biased or unrepresentative; the collection methodology is flawed; or analysis limited. Despite these I feel that so long as you know and understand the bias in a data set, cleanse the data and analyse appropriately, all of which a seasoned research agency should be doing for you anyway, these are not big problems. Besides, I challenge researchers and site owners to consider how they are going to get this vital data cost effectively, by other means?
I consider online surveys to be good at four particular things and research should be designed (irrespective of the technology used) to understand one or more of these and not much else.
1) Profiling site visitors – Who are you? Demographics, attitude, type of visitor etc.
2) Understanding intent – What are you here to do on this visit?
3) Task success – How successful were you in accomplishing your task?
4) Net Promoter Score – (satisfaction) – How likely is it that you would recommend this site to a friend or colleague?
To eliminate some of the problems associated with “bad” surveys I would urge people to think about the following three things which are all linked:
1) The running order. I’m going to state the obvious now but if respondents get bored or frustrated you will only get the answers from the questions they have seen to the point they give up. It is vital therefore to keep things as short as possible but also it is sometimes advisable to prioritise the most important questions first. For example if the primary objective is to learn what they came to do and how successful they were then you could simply ask the “intent” question first and follow it with the rest starting with a success question. Lots of surveys start with the demographic questions and that is all the respondents answer. Sometime this is because that’s what the researcher wants to know and sometimes because they got carried away with the questions and the respondents got bored. Sometimes however you have to start with these because that’s what defines your “Quotas”…
2) Quotas – if you launch a study and you suddenly get a flood of 70 year old men with one leg and you are nearing your total prescribed total number of responses; is that likely to be representative of the whole population? This has to be a judgement call. If so then you have your audience. If not you should consider closing the survey to them (or to “screen” them out in survey parlance.) If you want to do this before they answer all the questions then you need to do this in an early question. The danger is that this really is the demographic of the site audience so by balancing responses you are also biasing the data… Beware!
3) As a result of the previous two questions you will have decisions to make about which questions you make mandatory? Often, clients will insist that all questions are vital when in actual fact this is not the case. Again, focus on specifically what you are trying to achieve with THIS study. Better to focus several small pieces of research than try and do a one size fits all piece in an attempt to meet all your needs. Which questions are then mandatory will become clear from the objectives and should be driven by your choices to the questions raised above. If there are any mandatory questions you should state this clearly to the respondent.
Other good advice can be found on Avinash Kaushik’s site and a case study about poor survey responses can be found on this blog post and this case study.