ClickZ Articles

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

I have just been reading a booked called “Super Crunchers” by Ian Ayres. It’s an interesting book on how the use of data mining and predictive analytics is becoming more widespread across all aspects of our societies, and is increasingly shaping our lives. He cites a number of different examples where these empirical approaches are able to outperform human experts in their ability to accurately predict the likely outcomes.

I particularly liked his story of an econometrician who was able to predict the expected quality of Bordeaux wine based on a simple regression analysis of weather data. He was able to predict the expected quality of a particular vintage based on just three variables; the amount of rainfall in the winter, the amount of rainfall during the harvest and the average temperature during the growing season. What was interesting for me was not the fact that he was able to make these predictions, but the accounts of the resistance and even hostility that he got from the “wine establishment” for his predictions. The wine experts of the time were very threatened and affronted by the fact that their “art” and “expertise” could be reduced to a simple equation.

Ayers goes on to give a number of other examples in various industries where the growth of data and technology has allowed data mining and predictive analytical techniques to change the rules of the game, from baseball scouting to social policy development and medicine. Quite often in each of these fields there has been resistance to the ascendency to the use of these techniques from the established experts in that field, such as baseball scouts, policy makers, doctors and so on. They would not, or could not accept that such empirical methods could be better than the expertise they had developed over years of training and experience. However numerous studies cited by Ayers have shown that predictive analytics outperforms “experts” in the ability to predict an outcome correctly. That doesn’t mean that predictive techniques always get it right just that they get it right more often than the experts.

In the digital marketing field Ayers uses the example of A/B and Multi-Variate Testing (MVT). The point he makes is that the volume of data and the technology, now allows people to run repeated tests and trials to predict which versions of which element on a page is most likely to be successful in driving the desired outcome. Those of you familiar with the MVT technologies will know that the marketing stance behind them is often that they eliminate the need for subjectivity in the design process. You just come up with some alternative versions and see which one works best. It’s the ultimate tool for overcoming bias and subjectivity of the various stakeholders involved in site development. Who needs usability testing, right?

Ayers’ background is not as a statistician or an analyst but as a lawyer. You don’t immediately think of lawyers as being masters of the empirical universe and why would a lawyer be an expert in number crunching? The interesting point being a lawyer could be similar to being an analyst. Often you are trying to prove or disprove a hypothesis and looking for the appropriate evidence to support your theory or disproves somebody else’s and, for me, this gives rise to one of the fallacies about econometrics and predictive analytics that it is purely a scientific discipline.

Predictive analytics is often as much about art as it is about science. To build a good model you need to have a good understanding of the way that the “system” you are trying to model works. More often than not, at the beginning of the model building process, there is some subjective opinion about what are going to be the likely factors influencing the thing that you are trying to predict. So where do these opinions come from? They usually come from the people who are knowledgeable or experts in that particular field. We sometimes called this the “domain expertise”. If we take the example of the econometrician predicting the quality of wine, the econometrician was also a wine buff so he had some previous knowledge about what the likely factors were that could potentially affect the quality of a particular vintage. His skill was in quantifying it.

In the same way, some domain expertise is needed in the development of good tests. If we look at MVT then the technology can help you determine which the best page design to use is. If you test 4 different versions of an element (say a call to action), then you will get a winner. That “winner” may be the one that you started out with, but it’s still the winner. It doesn’t mean though that it’s the best one, it’s just the one that was best out of the various options that you looked at. There may be a much better option out there which you haven’t tested. Usability experts can potentially provide better insights into what versions are the best ones to test in the first place, and also help to understand why the results have come out the way that they have.

So we need the experts to help us build better models. That expertise may come from years of experience or knowledge gained from understanding the effectiveness of previous models. In either case, there’s room for both the science and the art.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

Last time I looked at the importance of getting a good understanding of the online user experience through tools such as surveys and other feedback mechanisms. The development of technology has meant that these types of tools can be obtained either for free or relatively cheaply, lowering one of the barriers to their adoption. However, just because the tools are cheap it doesn’t mean that the survey needs to be cheap as well, as asking for user feedback is all part of the user experience. So if you’re looking to run surveys yourself rather than use and agency to do it for you, then here are some things to consider along the way.

Be clear about the purpose of the purpose of the survey
Getting some user feedback is better than getting none at all but don’t do a survey just for the point of doing a survey. Be clear about what it is that you want to know. Write down the objectives of the survey and keep it hard and focussed. You may have more than one objective but don’t try to answer everything in one go.

KISS - Keep it Short and Simple
It’s difficult to say exactly how long or short a survey should be, but try and be as efficient as possible. Stick to the objectives and avoid the temptation to cram too much in. If necessary do more than one survey on a different sample of users or only ask certain questions to certain people. Also be careful of the language that you use and the style of the questions that you ask. It’s easy to slip into either using your own terminology that you would use in the business or “market research speak”. Your users probably won’t understand either. Above all make it interesting. Make sure that the questions are relevant and the survey is engaging.

Work out how much data you need
Whilst some data is better than no data, be wary of basing decisions on a small number of responses. A rule of thumb I use is that 400 responses will give you a reasonable level of accuracy in the answers, getting loads more won’t necessarily give you a lot more accuracy but it will mean that you can filter the data to look at sub-groups (ie the differences in responses between young people and older people for example). Once you’ve worked out how many respondents you are likely to need then you need to work out how many people you need to ask to get them. The response rate on surveys can vary enormously so if you haven’t run surveys before on your site you may need to do some test first, which brings me onto my next point.

Test before going live
Test the survey on a small number of users before going fully live. This is especially true if you have a survey where the questions that people are asked depends on the answers that they have given before. You need to check that the survey works technically well and that also all it works well from the user perspective. It’s best to spot mistakes early!

Keep it ethical
Use your surveys for research purposes only. Don’t use them to try and sell people things.

See your survey as an extension of your brand
If you are using online surveys on your site then they are part of the user experience. Ensure that the tone, design and look and feel of the survey are complementary with the brand. You don’t want your users having too different experiences. I had a good reminder of this when I was asked to take part in an online survey for a well known technology brand. The brand image is all about ease of use and good design. The survey was badly designed, looked horrible and was very difficult to complete. I was so annoyed by the survey that I wrote and told them what I thought. Which brings me to my next point…

Be prepared for feedback
You will probably get more feedback that you anticipate. People will either reply to an email invite or will write comments in open text boxes. You will need to have processes in place to deal with any customer service issues that may bubble up. Many of these may be nothing to do with the website but they need to be dealt with.

Hopefully these few tips will help you along the road to getting some useful and interesting feedback from users about their online experience. With the range of tools available there’s no reason not to get started but make it a good experience!

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

One of the trends over the past couple of years has been the growth in the number of organisations who are using some form of visitor survey tool on their websites. It wasn’t so long ago that when running workshops I would be asked how many people were running surveys on their websites, and maybe 20% of people would put their hands up, these days its probably around 50%. That crude survey itself is indicative of the wider adoption of visitor feedback mechanisms as part of the digital analytics toolkit.

This adoption has been caused by two main factors I think. First of all there has been the growth of availability of online survey capabilities across all levels of sophistication and these surveys have become more “productised” which makes them easier for organisations to buy and deploy. Examples within the customer satisfaction measurement space include 4Q at one end of the spectrum, which is a limited but free survey tool, through to more enterprise level products such as Forsee Results and iPercpetions.

The second driver of growth I think has been the realisation by organisations that they can’t measure the effectiveness of their digital marketing strategies by purely looking at clickstream data. Web analytics tools can tell you what happened in terms of visitors’ behaviour and when it happened, but they are not necessarily the best tools for telling you who did what and why they did it. That’s where survey data comes in by providing this different perspective. By asking people questions about themselves, why they do what they do and what they think, it’s possible to fill in some of the blanks left by the volumes of clickstream data at our disposal.

As with all measurement and analysis tools and systems, the amount of thought and preparation put into configuration and deployment pays dividends later on in terms of the quality and robustness of the data. Survey tools are no different. There are various approaches that an organisation might take to developing and launching a survey.

First of all they may choose to outsource the whole thing to an agency to manage on their behalf. Here the agency would be responsible for designing the questionnaire, scripting the questionnaire in whichever survey tool they use, deploying the survey, collecting the data and analysing the results. This is the approach that most organisations take when doing offline market research and there’s nothing wrong with using the same approach online. The main concern of the organisation commissioning the research is to ensure that the research objectives are clear and aligned to their business objectives, to agree the questionnaire and also to ensure that the survey is fit for purpose and holds up to the brand values. This last point is particularly important as there is evidence that suggests that poorly executed online surveys do potentially damage the brand whether they are launched on the site or when people are invited to take part via email. I have certainly been on the receiving end of some surveys where I’ve thought that the style of the survey was completely at odds with the brand.

The second option for an organisation is to choose to design and manage the survey themselves. Certainly these days there are no end of free or cheap survey tools that allow you to run surveys of varying complexity. Quite often a provider will provide a basic version for free or at a low cost which has limited functionality and data capture limits, they also offer a higher end tool which allows more complex questionnaires to be designed and more responses to be captured. However just because the tool itself is free, it doesn’t mean that the survey doesn’t require the same sort of diligence in its preparation and deployment than you were using a more complex, enterprise level product. One of the dangers is that surveys deployed using cheap tools with little effort put into them look cheap themselves and may have a negative impact on the user experience and their perception of the brand.

In my next article I will outline some tips for maximising the effectiveness of your survey efforts.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

As I wrote in my last column, 2009 is going to be a challenge for digital analytics and insights teams as they will need to get leaner, smarter and faster. I also think there’s a big opportunity to be grasped in terms of increasing the impact that these teams have on the corporate entity. As planning horizons and measurement cycles get shorter, there will be more of an emphasis on understanding and optimising the direct response model. This is where digital and direct marketing analysts traditionally ply their trade.

However, I don’t think it’s merely a question of doing more of the same and doing it more often, I think that now is the time when there is an opportunity for digital analysts to broaden their remit and expand their influence on the organisation. As Tom Davenport (author of “Competing on Analytics) puts it: “The planets are aligned for analytics”. What he means by that is that the necessary fundamentals that will allow analytical approaches to thrive in business, are in place. We have the data and we have the technology to manage and manipulate that data. With the economic situation we find ourselves in, we also have the motive. Organisations will be looking for solutions and digital analysts are well placed to provide them. To do that though, I think that digital analysts need to increasingly position themselves as data integrators and cross-channel specialists.

Too much data sits in silos and this is particularly true in the digital channel. We have web analytics data, campaign data, survey data, customer data and so on. Quite often each of these data sources will have a different business owner either within the organisation or in an agency. The value of each data source is diminished when housed and analysed in isolation, the full value comes from layering in data sources to look at different aspects of a particular problem. To do this analysts need to become masters of the data universe and be able to understand and leverage these data sources. They need to understand the nuances of each data source and be able to explain these nuances to business users who may not understand the differences between the different types of information at their disposal.

So on one level digital analysts have the opportunity to deliver increased value by taking a more holistic view when addressing issues in the digital channel. On another level I believe that digital analysts can take the lead in the development of cross-channel analytical approaches. At the heart of cross-channel analytics is customer analytics. However, quite often in organisations today customer analytics tends to ignore the digital channel and certainly the interaction between the online and the offline channel is often not well understood.

For many industries the internet has become the primary tool for researching brands, products and services even if the final transaction takes place offline. One only has to look at travel and financial services as two examples where it is probably safe to say that the vast bulk of product research takes place online. Since the internet operates “upstream” in many customer acquisition processes, it’s in the interests of the digital community to understand and demonstrate the influence the channel has in the “downstream” conversion process irrespective of whether it happens online or offline. I see therefore the development of cross-channel measurement techniques as being the domain of the digital analysts so that the full and true return on investment in the channel can be better understood. Once better cross-channel measurement processes are in place it becomes possible to develop better cross-channel marketing processes.

So if the planets are aligned for analytics generally, then it’s certainly time for digital analytics to come in from the cold and exert its influence beyond the realms of a single data source in a single channel. Data integration and cross-channel analytics is becoming the name of the game in 2009.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

As you’re reading this, probably, like me, you’re already wondering what 2009 might bring. At this time of year I often wish I had the attributes of the Roman god Janus (with his ability to look backwards and forwards at the same time.) It’s easy to look back and review a year, it’s harder to look forward and see what might be. Having said that, this time last year I wrote in this column “If I’m to make one more prediction for 2008, it would be that I don’t think it’s going to be a dull year!”. Sometimes you can wish that you were less prescient! It’s been anything but dull and for a lot of us this is the first time we are experiencing anything other than relative economic stability and growth.

If you’re in the forecasting business then what you are mostly doing is identifying patterns and extrapolating trends. What makes forecasting difficult is when you suffer “shocks” to the system that radically change the way that the “system” works. On our news at the moment we hear comments like “nobody anticipated this”, “these are unprecedented times” and so on. This makes it hard to extrapolate any trends because the trends have been broken or disrupted. We can only look back to the last time such events occurred (decades ago) and try and get some signals from all the noise. The problem is that looking back to the last occasions when our economy went into recession is also not likely to help much. Circumstances were very different back in the late 80s and early 90s. For a start, technology did not impact our lives in the same way and the internet was in its infancy. Fax and telex were still the primary forms of “immediate” communication. With such a difference in our technological capabilities what can we learn as marketers from previous recessionary environments? Probably not a lot.

The other difference is the scale of the problem. Historically one global economy might be in recession when others might be stable or in growth. Multi-national companies could play a “portfolio” game, managing losses in some markets whilst seeing compensating growth in others. This time round there are no safe harbours.

If there is one prediction I would make about 2009, it would be that 2009 is going to be unpredictable. The degree of turbulence and volatility means that it will be difficult to separate the signals from the noise. One result I think will be that planning horizons are going to get shorter, from years to months and from months to weeks. As a result of that, measurement cycles are going to get faster. In turbulent times people are going to want more frequent information and better insight delivered more quickly. Analytics and insight teams (if they still exist) are going to be under pressure to produce more goods of a higher quality with a faster turnaround. For organisations this has implications around the use of the human and technological resources they have at their disposal.

Whenever I write or talk about this subject, I always get the same mental image. It’s of a boat or yacht sailing in a storm. On a calm day you can see for miles ahead and you probably don’t need to check your maps and equipment that often. You have the time to plan and plot your course ahead and the navigator has time for a coffee break. In a storm you have to be much more reactive to the circumstances you find yourself in. You can’t see that far ahead and so you need to rely on your equipment to tell you where you are and what to do next. You need to quickly communicate your decisions to those around you and the navigator is chained to the desk.

Without hopefully running the risk of torturing the metaphor, I see it the same way for organisations heading into 2009. We know the storm is coming but we may not know exactly when or where or how bad it’s going to be. We don’t know exactly what the impact will be and so we can’t predict how we may need to react. But we do know that the storm is coming and so we need to get ourselves prepared. For analytics and insight teams now is the time to make sure that the technology is working properly, the data is sound and that the business processes are in place. If you’re in the right kind of organisation you’re going to be having a very busy, challenging but hopefully rewarding year.

May I take this opportunity to wish you all the best for 2009.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

Over the past few weeks I have had the opportunity to try out the new Advanced Segmentation features that are being rolled out in Google Analytics. Although it’s a beta version of the new capability it allows Google Analytics users to take a deeper dive into their data and to discover underlying patterns in visitor behaviour.

In the past couple of years since I first started writing about segmentation in this column (www.clickz.com/3615916) it has become easier for digital marketers to look at different groups of website users. Developments in the technologies mean that there is almost no excuse for not beginning to segment your website visitors and to begin to understand how different types of visitors behave on your site. Having said that there are still many organizations that only look at the top line numbers and treat all visitors with a “one size fits all” approach to marketing.

So what is segmentation and how can it be used? In marketing terms segmentation is the process of identifying groups of individuals that have something in common. Those individuals then belong to the same segment. Importantly, what those individuals in that segment have in common is different to what other individuals in other segments have in common. So a simple example would be to segment users by the number of times they have visited the website. You could classify them into “buckets” such as new users (the first time they have visted), light users (i.e. have visited 2 to 3 times) and heavy users (i.e. have visited 4 times or more). The number of buckets in this instance would be determined by looking at the distribution of visits per visitor on the site and making an appropriate decision.

The point of making segmentation such as the one described above would be to understand whether there are any differences in the behaviour of these different groups which could then lead to some kind of differentiated marketing message or user experience being developed. If there is, then there is the opportunity to improve the expected return on investment. Let’s imagine that you have just run some kind of email marketing campaign and experienced a 10% uplift in sales. If the campaign was a generic campaign that you sent to all your users on your database, then what probably happened was that some segments of users really responded to your campaign and others didn’t. The reality is that you might have experienced a 20% sales uplift in some segments and 0% uplift in others. The idea behind segmentation is that if you can design a series of different campaigns that are relevant to each of the different segments then you should achieve say a 20% uplift in sales across all segments.

With most web analytics tools these days it’s possible to carry out basic segmentation of the data. It’s up to the analyst or the user to determine and decide what are the useful segments to look at. In GA (Google Analytics) there are some segments that are already set up that cover some of the obvious behaviours that might be of interest, like new vs returning visitors, paid for vs non-paid for traffic but the analysts will want to explore and create their own segments based on their own understanding of the website, and also the kinds of issues that are being faced in the business. For example it might be that there is an issue in the business about how many people look at a quote for a hotel room but then don’t book. In that instance the analyst may set up three different visitors segments as follows:

• Browsers (people who visited the website but didn’t reach the quote process)
• Quoters (people who got a quote but didn’t book a room)
• Bookers (people who booked online)

Once the segments have been created, the analyst can look for differences between these segments in terms of how they get to the website and what they do when they get there. Are there differences in the type of channel they come in on or the keywords they use? Do they look at different types of content? Do they use different tools and applications on the site such as the on-site search? If so what are the kinds of keywords do they use on the on-site search engine?

As you can see, once the ability to create segments becomes a reality, the analysis possibilities become endless. Different tools offer different capabilities but the principles remain the same for the curious analyst. The challenge though with segmentation is not so much in doing the analysis but taking action as a result. The benefits from taking insight into action, are building and deploying more targeted marketing programmes as a result.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission. 

It’s been a busy week since I started a new job at Foviance. The reaction from people I know in the industry about Foviance and Applied Insights coming together has been very positive. People tell me that it makes sense, which is good news because it certainly made sense to me.

The reason why it made sense to me is that I passionately believe that analysis and insight lies at the heart of improving the online user experience. Working with one of the leading user experience consultancies in the UK as the person responsible for driving their analytical services capabilities forward is going to present some great opportunities to grow and to learn. It’s never too late to stop learning!

For me marketing has always been a blend of art and science and in the digital marketing space there is more science available to those who want to take advantage of it. For many organisations though it has taken some time for them to adopt the ability to improve the online customer experience through measurement and analysis. At times there is often a tension between the “creatives” and the “analysts”, whereas the reality is that both is needed and both need to be blended.

I think that the position where many organisations have got to know is that they have “adopted” the measurement and analysis side of things. They have plumbed in a web analytics system. They may be regularly measuring customer satisfaction. They may be routinely doing testing. They now have access to the science. However, what they haven’t managed to do is to integrate the science into the way that they do business.

Often decisions are made on the basis of judgement even when the data is available to them. But the trouble is, as someone once said, “Good judgement comes from experience. Experience comes from bad judgement”. Or as an old boss told me after I had made some cock-up: “Neil, all experiences are learning experiences. It’s just some are more pleasant than others”. One of the key roles of data, analytics and insights is to help us avoid having too many “unpleasant learning experiences”.

So the opportunity going forward is to blend the art and the science in a seamless approach to improve the user experience. Creative designers working alongside analysts to understand the impact of their design changes in a collaborative fashion.
Quantitative analysts such as web analysts working alongside qualitative researchers such as usability consultants to understand the user experience form all the angles. Not just looking at what users did but also understanding why they did it and what they felt about the outcome. This kind of integrated approach will need integrated thinking based on integrated data. Integrated thinking will come from the recognition from all the players that they only have a part of the solution and their instinct should be to go and seek out the other parts.

The difference between adoption and integration will come down to organisational culture and processes. This is a theme that I keep returning to in my consulting activities as well as in this column as I think this is one of the biggest challenges in the industry at the moment. People still worry too much about the technologies rather than worry about what they are going to do with the technologies. Organisations and their agencies will need to start thinking about how to build the science into the creative process in a systematic way and how to view the creative process as an iterative, cyclical process rather than just a linear process. The physical manifestation of this vision might be a roomful of designers, information architects, analysts, usability experts and brand marketers coming together to throw ideas around about what the user experience should look like. Each contributing their perspective and each contributing to the final outcome. Not on an ad-hoc basis for big projects, but on a regular basis constantly iterating the solution week after week. It might take a while to get there but I’m looking forward to the journey.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

As I write this, the Emetrics Marketing Optimisation Summit in Washington DC is drawing to a close and I am trying to process all the inputs and turn them into some outputs of the core themes and takeaways. Tough job, there’s been a lot of stuff to process.

One of the key messages that I picked up from a number of the sessions I attended was that people had moved on talking about the “what” to the “how”. So the talks focussed less on “What is web analytics?”, “What is optimisation?”, “What is multi-variate testing?” and more on “How do I embed web analytics into the organisation?”, “How do I create a testing culture?, “How do we move towards a performance driven organisation”. It’s like we know what the tools in the toolbox are now; we just need to figure out how to use them better and to get other people in the organisation on board. There were still some presentations that covered the “what” type questions but they seemed to mostly revolve around the newer technologies and the emerging measurement practices such as social media and mobile analytics.

A parallel theme that came through was the sense that some organisations or people within these organisations were hitting a glass ceiling. They had deployed the tools, generating the data, created the reports but were struggling to take it to the next level. They could see the opportunity but were not able to make the break through. Bill Gasman from Gartner outlined a number of requirements to move an organisation’s analytics capability forward and the first one of these was to have senior “C” level sponsorship. I’ve just finished reading Tom Davenport’s book “Competing on Analytics” and time and time again he also makes the point that companies that successfully deploy an enterprise wide approach to analytics usually have someone at the top making it happen. The question then becomes how you go about getting that support? I described one approach that worked for me in my last article. (www.clickz.com/showPage.html?page=3631145) and that seemed to be a view endorsed by Bill Gassman when asked the same question by someone in the audience. His answer was to start small and build momentum. It was interesting to observe that some of the issues we are encountering in Europe are not all that different to some of the issues being raised here in the US.

One of the highlights for me was watching Avinash Kaushik unveil the latest enhancements to Google Analytics. You sensed it was what the crowd had been waiting for. It’s not often you see a vendor being applauded for announcing feature releases. Of the various developments announced there were two that caught my attention. The first one is the new advanced segmentation feature. I’m a big fan of the ability to filter and segment your data and so any developments in this area are welcome. Providing a segmentation capability in a tool like Google Analytics will encourage property owners to look beyond the topline numbers and to start to think about their site in terms of different groups of visitors behaving in different ways. So hopefully people will start to look beyond teh bland averages of topline reports and start to drill down into their data.

The other feature that caught my eye was the announcement of a Google Analytics API to allow access to the underlying data. Details are still a bit sketchy at the moment but for me one of teh features of a more enterprise level tool is the data integration capabilities. There are many hacks out there for getting data out of Google Analytics and hopefully the API will make this easier in the future. This seems to be recognition by Google that web analytics data can’t operate in a silo if it is to survive.

Finally the presentation of the conference for me was by Jason Carmel from ZAAZ. Jason’s presentation went by the title of “Effectively using kittens for optimisation and usability” (Go figure!) and in it he looked at how site optimisation using tools such as Optimost and Google Website Optimiser is complementary to user-centric design processes and usability based optimisation. He outlined the process by which the two can work together in site optimisation projects with the site optimisation tools basically telling you what’s working and the usability analysis showing you why it’s working and how to use usability experts to improve the quality of the site optimisation tests. It reinforced to me that you always will need more than one tool, in the toolbox to get the job done properly.