ClickZ Articles

 This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

I’m increasingly convinced that the issues that most businesses face around the successful deployment of analytics in their business are not to do with their technologies but to do with their businesses processes. That view was reinforced this week when I was running a workshop with a group of students studying on a Masters Programme in Internet Retailing. Read more…

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

It’s often interesting how businesses can be lulled into a false sense of security by looking at superficial “topline” numbers without investigating what is really going on at a more granular level. I think this is often the case in online channels where there is some element of registration or subscription involved. A story from a business I worked at makes my point. Read more…

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

Despite the tough trading conditions a trend that I have observed here in the UK and I expect, is that organisations are continuing to build out their web analytics capabilities and grow their teams. A number of clients I work with are looking to bring on new people and the job market for web analysts remains reasonable healthy. In some cases this means that companies are looking to appoint their first person into a new role, but increasingly some organisations are looking to to expand the team to 2, 3 or more people. Read more…

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

I have just been reading a booked called “Super Crunchers” by Ian Ayres. It’s an interesting book on how the use of data mining and predictive analytics is becoming more widespread across all aspects of our societies, and is increasingly shaping our lives. He cites a number of different examples where these empirical approaches are able to outperform human experts in their ability to accurately predict the likely outcomes.

I particularly liked his story of an econometrician who was able to predict the expected quality of Bordeaux wine based on a simple regression analysis of weather data. He was able to predict the expected quality of a particular vintage based on just three variables; the amount of rainfall in the winter, the amount of rainfall during the harvest and the average temperature during the growing season. What was interesting for me was not the fact that he was able to make these predictions, but the accounts of the resistance and even hostility that he got from the “wine establishment” for his predictions. The wine experts of the time were very threatened and affronted by the fact that their “art” and “expertise” could be reduced to a simple equation.

Ayers goes on to give a number of other examples in various industries where the growth of data and technology has allowed data mining and predictive analytical techniques to change the rules of the game, from baseball scouting to social policy development and medicine. Quite often in each of these fields there has been resistance to the ascendency to the use of these techniques from the established experts in that field, such as baseball scouts, policy makers, doctors and so on. They would not, or could not accept that such empirical methods could be better than the expertise they had developed over years of training and experience. However numerous studies cited by Ayers have shown that predictive analytics outperforms “experts” in the ability to predict an outcome correctly. That doesn’t mean that predictive techniques always get it right just that they get it right more often than the experts.

In the digital marketing field Ayers uses the example of A/B and Multi-Variate Testing (MVT). The point he makes is that the volume of data and the technology, now allows people to run repeated tests and trials to predict which versions of which element on a page is most likely to be successful in driving the desired outcome. Those of you familiar with the MVT technologies will know that the marketing stance behind them is often that they eliminate the need for subjectivity in the design process. You just come up with some alternative versions and see which one works best. It’s the ultimate tool for overcoming bias and subjectivity of the various stakeholders involved in site development. Who needs usability testing, right?

Ayers’ background is not as a statistician or an analyst but as a lawyer. You don’t immediately think of lawyers as being masters of the empirical universe and why would a lawyer be an expert in number crunching? The interesting point being a lawyer could be similar to being an analyst. Often you are trying to prove or disprove a hypothesis and looking for the appropriate evidence to support your theory or disproves somebody else’s and, for me, this gives rise to one of the fallacies about econometrics and predictive analytics that it is purely a scientific discipline.

Predictive analytics is often as much about art as it is about science. To build a good model you need to have a good understanding of the way that the “system” you are trying to model works. More often than not, at the beginning of the model building process, there is some subjective opinion about what are going to be the likely factors influencing the thing that you are trying to predict. So where do these opinions come from? They usually come from the people who are knowledgeable or experts in that particular field. We sometimes called this the “domain expertise”. If we take the example of the econometrician predicting the quality of wine, the econometrician was also a wine buff so he had some previous knowledge about what the likely factors were that could potentially affect the quality of a particular vintage. His skill was in quantifying it.

In the same way, some domain expertise is needed in the development of good tests. If we look at MVT then the technology can help you determine which the best page design to use is. If you test 4 different versions of an element (say a call to action), then you will get a winner. That “winner” may be the one that you started out with, but it’s still the winner. It doesn’t mean though that it’s the best one, it’s just the one that was best out of the various options that you looked at. There may be a much better option out there which you haven’t tested. Usability experts can potentially provide better insights into what versions are the best ones to test in the first place, and also help to understand why the results have come out the way that they have.

So we need the experts to help us build better models. That expertise may come from years of experience or knowledge gained from understanding the effectiveness of previous models. In either case, there’s room for both the science and the art.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

Last time I looked at the importance of getting a good understanding of the online user experience through tools such as surveys and other feedback mechanisms. The development of technology has meant that these types of tools can be obtained either for free or relatively cheaply, lowering one of the barriers to their adoption. However, just because the tools are cheap it doesn’t mean that the survey needs to be cheap as well, as asking for user feedback is all part of the user experience. So if you’re looking to run surveys yourself rather than use and agency to do it for you, then here are some things to consider along the way.

Be clear about the purpose of the purpose of the survey
Getting some user feedback is better than getting none at all but don’t do a survey just for the point of doing a survey. Be clear about what it is that you want to know. Write down the objectives of the survey and keep it hard and focussed. You may have more than one objective but don’t try to answer everything in one go.

KISS – Keep it Short and Simple
It’s difficult to say exactly how long or short a survey should be, but try and be as efficient as possible. Stick to the objectives and avoid the temptation to cram too much in. If necessary do more than one survey on a different sample of users or only ask certain questions to certain people. Also be careful of the language that you use and the style of the questions that you ask. It’s easy to slip into either using your own terminology that you would use in the business or “market research speak”. Your users probably won’t understand either. Above all make it interesting. Make sure that the questions are relevant and the survey is engaging.

Work out how much data you need
Whilst some data is better than no data, be wary of basing decisions on a small number of responses. A rule of thumb I use is that 400 responses will give you a reasonable level of accuracy in the answers, getting loads more won’t necessarily give you a lot more accuracy but it will mean that you can filter the data to look at sub-groups (ie the differences in responses between young people and older people for example). Once you’ve worked out how many respondents you are likely to need then you need to work out how many people you need to ask to get them. The response rate on surveys can vary enormously so if you haven’t run surveys before on your site you may need to do some test first, which brings me onto my next point.

Test before going live
Test the survey on a small number of users before going fully live. This is especially true if you have a survey where the questions that people are asked depends on the answers that they have given before. You need to check that the survey works technically well and that also all it works well from the user perspective. It’s best to spot mistakes early!

Keep it ethical
Use your surveys for research purposes only. Don’t use them to try and sell people things.

See your survey as an extension of your brand
If you are using online surveys on your site then they are part of the user experience. Ensure that the tone, design and look and feel of the survey are complementary with the brand. You don’t want your users having too different experiences. I had a good reminder of this when I was asked to take part in an online survey for a well known technology brand. The brand image is all about ease of use and good design. The survey was badly designed, looked horrible and was very difficult to complete. I was so annoyed by the survey that I wrote and told them what I thought. Which brings me to my next point…

Be prepared for feedback
You will probably get more feedback that you anticipate. People will either reply to an email invite or will write comments in open text boxes. You will need to have processes in place to deal with any customer service issues that may bubble up. Many of these may be nothing to do with the website but they need to be dealt with.

Hopefully these few tips will help you along the road to getting some useful and interesting feedback from users about their online experience. With the range of tools available there’s no reason not to get started but make it a good experience!

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

One of the trends over the past couple of years has been the growth in the number of organisations who are using some form of visitor survey tool on their websites. It wasn’t so long ago that when running workshops I would be asked how many people were running surveys on their websites, and maybe 20% of people would put their hands up, these days its probably around 50%. That crude survey itself is indicative of the wider adoption of visitor feedback mechanisms as part of the digital analytics toolkit.

This adoption has been caused by two main factors I think. First of all there has been the growth of availability of online survey capabilities across all levels of sophistication and these surveys have become more “productised” which makes them easier for organisations to buy and deploy. Examples within the customer satisfaction measurement space include 4Q at one end of the spectrum, which is a limited but free survey tool, through to more enterprise level products such as Forsee Results and iPercpetions.

The second driver of growth I think has been the realisation by organisations that they can’t measure the effectiveness of their digital marketing strategies by purely looking at clickstream data. Web analytics tools can tell you what happened in terms of visitors’ behaviour and when it happened, but they are not necessarily the best tools for telling you who did what and why they did it. That’s where survey data comes in by providing this different perspective. By asking people questions about themselves, why they do what they do and what they think, it’s possible to fill in some of the blanks left by the volumes of clickstream data at our disposal.

As with all measurement and analysis tools and systems, the amount of thought and preparation put into configuration and deployment pays dividends later on in terms of the quality and robustness of the data. Survey tools are no different. There are various approaches that an organisation might take to developing and launching a survey.

First of all they may choose to outsource the whole thing to an agency to manage on their behalf. Here the agency would be responsible for designing the questionnaire, scripting the questionnaire in whichever survey tool they use, deploying the survey, collecting the data and analysing the results. This is the approach that most organisations take when doing offline market research and there’s nothing wrong with using the same approach online. The main concern of the organisation commissioning the research is to ensure that the research objectives are clear and aligned to their business objectives, to agree the questionnaire and also to ensure that the survey is fit for purpose and holds up to the brand values. This last point is particularly important as there is evidence that suggests that poorly executed online surveys do potentially damage the brand whether they are launched on the site or when people are invited to take part via email. I have certainly been on the receiving end of some surveys where I’ve thought that the style of the survey was completely at odds with the brand.

The second option for an organisation is to choose to design and manage the survey themselves. Certainly these days there are no end of free or cheap survey tools that allow you to run surveys of varying complexity. Quite often a provider will provide a basic version for free or at a low cost which has limited functionality and data capture limits, they also offer a higher end tool which allows more complex questionnaires to be designed and more responses to be captured. However just because the tool itself is free, it doesn’t mean that the survey doesn’t require the same sort of diligence in its preparation and deployment than you were using a more complex, enterprise level product. One of the dangers is that surveys deployed using cheap tools with little effort put into them look cheap themselves and may have a negative impact on the user experience and their perception of the brand.

In my next article I will outline some tips for maximising the effectiveness of your survey efforts.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

As I wrote in my last column, 2009 is going to be a challenge for digital analytics and insights teams as they will need to get leaner, smarter and faster. I also think there’s a big opportunity to be grasped in terms of increasing the impact that these teams have on the corporate entity. As planning horizons and measurement cycles get shorter, there will be more of an emphasis on understanding and optimising the direct response model. This is where digital and direct marketing analysts traditionally ply their trade.

However, I don’t think it’s merely a question of doing more of the same and doing it more often, I think that now is the time when there is an opportunity for digital analysts to broaden their remit and expand their influence on the organisation. As Tom Davenport (author of “Competing on Analytics) puts it: “The planets are aligned for analytics”. What he means by that is that the necessary fundamentals that will allow analytical approaches to thrive in business, are in place. We have the data and we have the technology to manage and manipulate that data. With the economic situation we find ourselves in, we also have the motive. Organisations will be looking for solutions and digital analysts are well placed to provide them. To do that though, I think that digital analysts need to increasingly position themselves as data integrators and cross-channel specialists.

Too much data sits in silos and this is particularly true in the digital channel. We have web analytics data, campaign data, survey data, customer data and so on. Quite often each of these data sources will have a different business owner either within the organisation or in an agency. The value of each data source is diminished when housed and analysed in isolation, the full value comes from layering in data sources to look at different aspects of a particular problem. To do this analysts need to become masters of the data universe and be able to understand and leverage these data sources. They need to understand the nuances of each data source and be able to explain these nuances to business users who may not understand the differences between the different types of information at their disposal.

So on one level digital analysts have the opportunity to deliver increased value by taking a more holistic view when addressing issues in the digital channel. On another level I believe that digital analysts can take the lead in the development of cross-channel analytical approaches. At the heart of cross-channel analytics is customer analytics. However, quite often in organisations today customer analytics tends to ignore the digital channel and certainly the interaction between the online and the offline channel is often not well understood.

For many industries the internet has become the primary tool for researching brands, products and services even if the final transaction takes place offline. One only has to look at travel and financial services as two examples where it is probably safe to say that the vast bulk of product research takes place online. Since the internet operates “upstream” in many customer acquisition processes, it’s in the interests of the digital community to understand and demonstrate the influence the channel has in the “downstream” conversion process irrespective of whether it happens online or offline. I see therefore the development of cross-channel measurement techniques as being the domain of the digital analysts so that the full and true return on investment in the channel can be better understood. Once better cross-channel measurement processes are in place it becomes possible to develop better cross-channel marketing processes.

So if the planets are aligned for analytics generally, then it’s certainly time for digital analytics to come in from the cold and exert its influence beyond the realms of a single data source in a single channel. Data integration and cross-channel analytics is becoming the name of the game in 2009.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

As you’re reading this, probably, like me, you’re already wondering what 2009 might bring. At this time of year I often wish I had the attributes of the Roman god Janus (with his ability to look backwards and forwards at the same time.) It’s easy to look back and review a year, it’s harder to look forward and see what might be. Having said that, this time last year I wrote in this column “If I’m to make one more prediction for 2008, it would be that I don’t think it’s going to be a dull year!”. Sometimes you can wish that you were less prescient! It’s been anything but dull and for a lot of us this is the first time we are experiencing anything other than relative economic stability and growth.

If you’re in the forecasting business then what you are mostly doing is identifying patterns and extrapolating trends. What makes forecasting difficult is when you suffer “shocks” to the system that radically change the way that the “system” works. On our news at the moment we hear comments like “nobody anticipated this”, “these are unprecedented times” and so on. This makes it hard to extrapolate any trends because the trends have been broken or disrupted. We can only look back to the last time such events occurred (decades ago) and try and get some signals from all the noise. The problem is that looking back to the last occasions when our economy went into recession is also not likely to help much. Circumstances were very different back in the late 80s and early 90s. For a start, technology did not impact our lives in the same way and the internet was in its infancy. Fax and telex were still the primary forms of “immediate” communication. With such a difference in our technological capabilities what can we learn as marketers from previous recessionary environments? Probably not a lot.

The other difference is the scale of the problem. Historically one global economy might be in recession when others might be stable or in growth. Multi-national companies could play a “portfolio” game, managing losses in some markets whilst seeing compensating growth in others. This time round there are no safe harbours.

If there is one prediction I would make about 2009, it would be that 2009 is going to be unpredictable. The degree of turbulence and volatility means that it will be difficult to separate the signals from the noise. One result I think will be that planning horizons are going to get shorter, from years to months and from months to weeks. As a result of that, measurement cycles are going to get faster. In turbulent times people are going to want more frequent information and better insight delivered more quickly. Analytics and insight teams (if they still exist) are going to be under pressure to produce more goods of a higher quality with a faster turnaround. For organisations this has implications around the use of the human and technological resources they have at their disposal.

Whenever I write or talk about this subject, I always get the same mental image. It’s of a boat or yacht sailing in a storm. On a calm day you can see for miles ahead and you probably don’t need to check your maps and equipment that often. You have the time to plan and plot your course ahead and the navigator has time for a coffee break. In a storm you have to be much more reactive to the circumstances you find yourself in. You can’t see that far ahead and so you need to rely on your equipment to tell you where you are and what to do next. You need to quickly communicate your decisions to those around you and the navigator is chained to the desk.

Without hopefully running the risk of torturing the metaphor, I see it the same way for organisations heading into 2009. We know the storm is coming but we may not know exactly when or where or how bad it’s going to be. We don’t know exactly what the impact will be and so we can’t predict how we may need to react. But we do know that the storm is coming and so we need to get ourselves prepared. For analytics and insight teams now is the time to make sure that the technology is working properly, the data is sound and that the business processes are in place. If you’re in the right kind of organisation you’re going to be having a very busy, challenging but hopefully rewarding year.

May I take this opportunity to wish you all the best for 2009.