Analytics Basics: Interpreting your survey data wisely
This article, written by Neil Mason, was originally published on Clickz.com on 01/07/10 and is republished here with permission.
Last time I looked at some of the characteristics of data collected from surveys, particularly data collected from surveys run on websites where you have no control on who is answering the survey. Generally this lack of control can cause some bias in the data which can cause some issues if you are looking at the aggregated reports. For example the data on the profile of visitors (i.e. gender, age etc) that you collected from survey data may not actually reflect the true profile of visitors to your site because of the different propensities of different groups to respond to surveys. So, does that mean that survey data is useless? Not really but it does means that it needs to be handled with a bit of caution.
One way to reduce the impact potential biases in the data is to trend the results over time. I’m always think that survey data is most useful when you have it running continuously anyway as it means that you have a constant monitor of the health of the site and you can refer to it to assess the effect of all sorts of marketing and product development activity. Having a continuous dataset also helps to reduce some of the bias. Say for example, that your survey shows that the age profile of visitors to your website is 40% under 35 and 60% over 35. We know that generally younger people are less responsive to surveys than older people and so we might suspect that there is a bias in the data towards older people. If however, 6 months later you look at the data and it shows that the profile has changed and that it is now 60% under 35 and 40% over 35 then, all other things being equal, whilst we still can’t be sure that the profile is absolutely correct, we can be reasonably confident that there has been a change in the profile over time and that the profile has got older. If we wanted to we could also check whether the change had been statistically significant or not.
Another way of reducing bias in your data is to segment your data. In fact I would say that you absolutely have to segment your data to make it useful and to understand it properly. So whilst I might not be confident that the profile data is properly representative of the reality, I can still use the profile data to look for differences in some of my key metrics such as customer satisfaction or the Net Promoter Score (NPS). I can compare satisfactions scores amongst the younger age groups and the older age groups to see if there are any significant differences and because there often are, I should always be looking at these key metrics amongst key segments of the site’s visitors. This is because changes in the visitor profile of the site can have a significant impact on the changes in these key metrics. Let me give you an example.
As I mentioned last time you can see differences in metrics like satisfaction score or NPS amongst different segments depending on their familiarity with the site or the brand. Often people who are visiting your website for the first time will have lower scores for satisfaction and NPS than those who have visited before. Let’s assume that you have been running some campaigns either online or offline and have driven a significant amount of new traffic to the site. The survey you’re running on the site will probably reflect the increase in new visitors and as a result it’s possible that the overall satisfaction score will go down. This not because people are overall less satisfied with the site experience but because you have a greater proportion of people answering the survey (i.e. first time visitors) who generally tend to give lowers scores. Nothing may have actually changed in the site experience itself, the only change has been in the mix of visitors to the site. In fact, the satisfaction amongst first time visitors can have stayed the same and the satisfaction amongst repeat visitors also can also have stayed the same but apparently overall satisfaction can appear to have gone down.
So, on the face of it online survey based data looks to have some serious issues with it. However, by understanding the source of these issues and interpreting the data wisely can ensure that you can get some real value from this rich source of customer insight. And remember…segment, segment, segment!