Analytics Basics: Understanding survey data

This article, written by Neil Mason, was originally published on Clickz.com on 18/06/10 and is republished here with permission.

Having looked at some of the fundamentals around web analytics metrics over the past few weeks, this time I turn my attention to survey based data and metrics. Over the past couple of years there has been a growth in the number of organisations that are running on-site survey based voice of the customer programmes. These might range from simple “do it yourself” approaches using free or low cost survey software, to a packaged tool like 4Q or more sophisticated programmes like ForSee Results or iPercpetions. As with all things, you pay your money and you make your choices.

This growth in the use of survey data and other customer insight tools is great to see. I always say that web analytics can tell you what happened and when, but rarely tells you who or why. That’s where tools like surveys come in. They give you a different perspective to what’s happening or not happening. But like with your web analytics data, it’s important to understand the fundamentals of where the data comes from and what that means in terms of how to use it and interpret it.

Most online customer insight is captured on the site; either through a site intercept survey or using a page level feedback mechanism. Site intercept surveys usually offer an invitation to a sample of the site’s visitors to take part in the survey and then the survey is completed at the end of the visit. So, you generally have no control over who or which type of people are offered the invitation to take part in the survey and you have no control of which people actually decide to complete the survey. This means that the survey data will generally have a bias in it but you also don’t necessarily know what that bias is unless you have another source of data to compare it with. If you find from your survey that 40% of the respondents are male and 60% of the respondents are female, that doesn’t mean that 60% of your site’s visitors are female. What is literally means is that 60% of the people who answered your survey claimed to be female. It’s possible that in this instance, the true proportion women visiting your site is actually nearer 50% but that women had a higher propensity to answer the survey than men did, making it look like there were more women than men. Unless you have another source of data, like from an audience measurement panel, then it’s going to be difficult to know whether the profile of people in your survey is representative of your website’s visitors or not.

Typically there are some general biases that you might see in site-intercept surveys. One that we often see is that people who generally know you are more likely to respond to your survey than people who don’t. This can manifest itself in many ways. They are more likely to have transacted or interacted with you, they are more likely to be customers and they are more likely to be the more frequent users of your website. Quite often you might ask a question about how often the respondent has visited the website in the past. If you look at the survey data and compare to your web analytics data (and there are issues over both sets of data) then typically you will find that you have a greater proportion of people who are repeat visitors to the website in your survey data than is recorded in your web analytics data. Typically there are also usually demographic biases as well. Men are generally less likely to answer surveys (though there are expectations depending on the subject matter) and generally the younger generations are harder to get feedback from than the older generations. So, your respondent sample is often underrepresented by young male. If this is a core audience for you to understand, then this is something you need to be aware of.

So, with all these potential issues, does that mean survey data is rubbish as people often claim, particularly when they don’t like the results? Well, no it doesn’t, but it does mean that you need to treat it carefully, to be aware of some of the potential biases that may exist and what the impact of those biases is on the metrics that you are reporting. In general to take aggregated survey results at a point in time at face value can be a bit dangerous. You need to find strategies to try and overcome these biases and you also need to understand the impact of these biases on some of the key metrics that we use survey data to report on such as customer satisfaction and Net Promoter Scores. This is what I’ll be looking at next time. Till then…