Neil Mason

We’ve all been there. Sitting round a conference room table discussing with our colleagues about the design of the website, the flow of a particular user path or the layout of a particular page. Opinions differ on what would work best, whether the call to action button should be red or blue, square or round, flat or bevelled. We all know best, because we’re experts. Aren’t we? In some cases it may not matter how expert we are, because the loudest voice will win or the most important person’s opinion will be the one that counts.

This is what happens in the absence of good data to guide decision making. It becomes subjective and vulnerable to interests, politics and personal preference. Good optimisation strategies are built on good data, coupled with an ability to drive through change. In the ‘red versus blue’ debate, what better way to find out than to try both of them and see which one works best? That’s the basis of a testing and experimentation programme. A/B testing and its more sophisticated cousin, multivariate testing or MVT, have been around for years as techniques in direct marketing and other analytical disciplines. These techniques allow you to test different versions of a page or different combinations of different elements on a page to see which one works best. Although the analytical techniques have been around for years, it is only relatively recently that they have begun to get adopted and gain traction in online marketing.

A/B testing is conceptually very simple but can be difficult to execute. To see whether the blue button would be better than the existing red button, two different versions of the page would be created, one with each different button. As visitors arrive on the website one set of visitors would be shown version A, and the remainder would be shown version B. The effectiveness of each page is measured according to whatever success criteria are the most relevant. Multivariate testing uses a more sophisticated set of algorithms to allow you to test multiple versions of different elements of a page at the same time.

Whichever testing regime is the right one, the challenges historically have been in implementation. It can be technically challenging to manage the process of setting up different versions of pages, splitting the traffic between them, setting up the measurement and then analysing the results. This heavy lifting has now largely been addressed by the availability of specialist testing and experimentation platforms such as Omniture’s Test & Target, Autonomy’s Optimost or Google’s Website Optimiser. Whilst the implementation of these services is not trivial, they manage the whole testing and experimentation process from deployment through to results.

The challenge for organisations then, still remains “What to test” and “How to test”. Analytics, survey data and usability evaluations can all help to identify the priorities for a testing programme. Start at the greatest pain points but also sort out the operational processes around the initial deployment on pages that are not too complex or mission critical. Marketing landing pages are often a good place to start. The question then becomes how should the page be tested? The challenge here is to ensure that you come up with a good test design. In our ‘red versus blue’ debate, are these the best options to test? What about green or orange? You could test those as well but in reality the number of variants that can be tested is limited. It’s important to ensure that you’re testing the right things in the right way. For me, this is where user experience expertise adds value to an analytical testing programme. Customer experience experts can identify which variants are likely to be the most effective and ensure that the test is as efficient and as effective as possible and the analytics experts can run the experiments and do the analysis.

Testing is a powerful tool in the site optimisation toolkit but it’s important to test the right things in the right way.

When talking to businesses about measuring their digital marketing performance, the approach I often use is a simple three-stage maturity model as follows:

• Firstly, put in place an effective performance tracking capability. It’s important that the business is clear about what it needs to measure and what tools it needs to put in place to conduct that measurement
• Next, once the organisation is happy that it is measuring the right things, it must capitalise on that knowledge by using it to optimise online business performance, maximise the effectiveness of campaigns, develop lead generation and convert sales or other opportunities. At this stage, the organisation is still primarily website centric
• Finally comes the transition to customer centricity, which sees the organisation spend less time thinking about how to optimise its site for business, and more discerning how to enrich the whole customer engagement experience and lifetime value of individual customers

A customer centric organisation is primarily concerned with how its website can fit in and around the broader relationship with its customers. Most organisations actually prefer online transactions, as they’re cheap to process. But that’s not to say the same companies aren’t interested in getting as much value from their customers as possible, by catering for their needs through multiple channels. Many reasonably forward-thinking organisations operate multiple channels without actually being cross-channel. To become truly cross-channel they need to fully integrate these separate knowledge silos and create a joined-up view of all ‘touch points’ with customers. Customers should be the primary entity the business is focused on, not the product, the service, the order or whatever else.

Has the organisation developed a view of the customer relationship that spans more than one channel? Are, for example, its call centre customer records integrated with stores and across online services? The challenge of customer centricity is to be able to measure and understand the behaviour of customers across those multiple channels. This requires a fundamental shift in philosophy, business processes, and the way in which organisations measure their success.

Meeting such challenges requires organisations to join up their data in order to be able to better track customers. Other obstacles to overcome include ascertaining exactly whose job this is, who owns the customer within an organisation, and convincing the wider business to recognise the longer term return on investment of uniting disparate customer information silos. It is sometimes difficult, for example, for a shop floor manager to see the role the web plays in driving sales to his aisles. But if a customer does business through more than one channel, it’s generally thought that they will prove more loyal and, ultimately, more valuable.

Of course, none of this change in thinking is easy, but there are methodological approaches that can be taken, specific to different business sectors and individual organisations, for any business that has realised the need to tackle the issue and become more customer centric. It’s about putting effective frameworks in place, helping businesses think through their future strategy for measurement, spotting issues early, offering practical guidance for overcoming the challenges of data integration, and looking beyond the present to plan for a future of customer centricity.

This is an abridged version of a conversation between web analytics guru Jim Sterne and Foviance director of analytical consulting Neil Mason.
Part 1 of the uncut video is now available for viewing as Foviance’s latest Customer Experience Podcast. This is the most recent in a series of regular interviews with senior figures from the world’s most respected businesses, focusing on the ways organisations manage the customer experience.

Jim, you’ve been working in consulting and internet marketing for 15 years and specifically web analytics for 10 years - what major shifts have you seen over that time?
The big shift happened early, which was going from nothing, to reporting. Reports told us how many people visited our pages, which was a valuable way of proving that the web was a viable way of reaching people and that people were interested enough to reach in and grab the information they wanted. To make their websites better, people then went from just reporting to benchmarking, and from that to optimisation, and then to market optimisation. We’re now starting to see companies using web data to run their businesses, using web data to determine what territories they should be shipping which products into, whether they should open up new stores, add new products to their lines, or add new features to their products. These are business decisions, and people are realising that the web is a tool they can use to inform those decisions.

So organisations are now using online intelligence to make offline business decisions?
Exactly, yes. I think it’s a natural progression; an evolution. In the beginning the marketing person says to the web guy, “What’s happening on the website?” And IT says, “Well what do you want to know?” Marketing says, “Well, what do you have?” IT says, “Well, what do you need?” Finally there is enough understanding between the two, that marketing is also saying, “Gee, this data could also be used for this, this, this and this…”

And is this something you’re seeing both sides of the Atlantic?
Companies in the US are very happy to jump in and are not so risk averse, while companies in Europe will sit back and say, “Go right ahead, let us know how it goes, if it works out great, we’ll follow.” That’s part of it; the cultural issue. The next part is scale. I like to quote Bruce Sterling who said: “The future is already here, it’s just not evenly distributed.” There are companies in the US, UK and Europe who are all doing amazing things - there are just more of them in the US.

What characteristics differentiate those companies who are more innovative and more adoptive?
All the elements related to change management. Support of senior management, a grass roots understanding of the value at the coal face, and the ability to move an organisation forward. We’re moving from the idea of measuring people and pages to an understanding of customer experience. What is the customer perception of the company? This is a philosophical change. What matters is what customers want and whether we are prepared to build it. So we have to have all kinds of ways of listening to those customers. What do they want, how do they feel, and what are they responding to? That is a philosophical switch, a corporate cultural issue, and the companies that ‘get’ customer centricity are the ones that are most able to take advantage of these tools.

And are you seeing a shift towards more organisations building success around a customer centric strategy?
I am, and the ones that are doing it first are the ones that have always been data intelligent. Some organisations know everything I buy, when I visit and how much I spend. That’s a huge amount of data which they can monetise. These companies understand that if they can use that data on behalf of their customers, instead of about their customers, in other words to help me buy more things that I want to buy rather than sell me what they want to sell me, then they will be more successful. They understand that web data is just another data stream. If they look at what we are all doing on their websites as well as what we are putting in our trolleys, they can make projections and take some actions from a marketing perspective, and absolutely measure the results. The companies that are adding the web data to what they already know about their customers end up being the winners.

Web data is notoriously messy and horrible, and there’s lots of it. What are the challenges in taming that particular beast?
Web data, relatively speaking is new. There are many different technologies to measure it, there are advancements all the time, and there are innovative ways of communicating online. I mean, who thought a year ago that tracking your Twitter data would be important? Well now it is. It’s also important to go out and listen to the blogosphere and find out what people are saying about you on their Facebook page. We’ve got to join that up with how they’re responding to surveys, how they behave on websites, and what their facial expression is as they use that web page. It’s messy, it’s a bit frightening, but it’s hugely valuable and an amazing competitive advantage when it can all be brought together.

What other obstacles do you think companies will need to overcome to become more customer-centric?
Philosophy is number one. They have to comprehend from the top that customer centricity is important. Next they must get a handle on data, which is a silo nightmare for so many reasons and in so many ways. Then there’s this funny middle management problem of change management. Senior management and the people actually doing the job understand, but the people in the middle are disinclined to use this data to analyse their success. They must have the confidence to turn decisions over to their customers. Customers own the brand. It is their perception that matters, and you can’t tell them what to think. They’re telling each other what they think. You’ve got to be able to listen to that. You must accept the customer as a member of the team. Data must be collated and connected so that it is meaningful, so that when a call goes into the call centre or an activity happens on the website or something happens at the till in the store, it triggers other marketing. That’s where this is all heading - automation of marketing for the benefit of the customer.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

I have just been reading a booked called “Super Crunchers” by Ian Ayres. It’s an interesting book on how the use of data mining and predictive analytics is becoming more widespread across all aspects of our societies, and is increasingly shaping our lives. He cites a number of different examples where these empirical approaches are able to outperform human experts in their ability to accurately predict the likely outcomes.

I particularly liked his story of an econometrician who was able to predict the expected quality of Bordeaux wine based on a simple regression analysis of weather data. He was able to predict the expected quality of a particular vintage based on just three variables; the amount of rainfall in the winter, the amount of rainfall during the harvest and the average temperature during the growing season. What was interesting for me was not the fact that he was able to make these predictions, but the accounts of the resistance and even hostility that he got from the “wine establishment” for his predictions. The wine experts of the time were very threatened and affronted by the fact that their “art” and “expertise” could be reduced to a simple equation.

Ayers goes on to give a number of other examples in various industries where the growth of data and technology has allowed data mining and predictive analytical techniques to change the rules of the game, from baseball scouting to social policy development and medicine. Quite often in each of these fields there has been resistance to the ascendency to the use of these techniques from the established experts in that field, such as baseball scouts, policy makers, doctors and so on. They would not, or could not accept that such empirical methods could be better than the expertise they had developed over years of training and experience. However numerous studies cited by Ayers have shown that predictive analytics outperforms “experts” in the ability to predict an outcome correctly. That doesn’t mean that predictive techniques always get it right just that they get it right more often than the experts.

In the digital marketing field Ayers uses the example of A/B and Multi-Variate Testing (MVT). The point he makes is that the volume of data and the technology, now allows people to run repeated tests and trials to predict which versions of which element on a page is most likely to be successful in driving the desired outcome. Those of you familiar with the MVT technologies will know that the marketing stance behind them is often that they eliminate the need for subjectivity in the design process. You just come up with some alternative versions and see which one works best. It’s the ultimate tool for overcoming bias and subjectivity of the various stakeholders involved in site development. Who needs usability testing, right?

Ayers’ background is not as a statistician or an analyst but as a lawyer. You don’t immediately think of lawyers as being masters of the empirical universe and why would a lawyer be an expert in number crunching? The interesting point being a lawyer could be similar to being an analyst. Often you are trying to prove or disprove a hypothesis and looking for the appropriate evidence to support your theory or disproves somebody else’s and, for me, this gives rise to one of the fallacies about econometrics and predictive analytics that it is purely a scientific discipline.

Predictive analytics is often as much about art as it is about science. To build a good model you need to have a good understanding of the way that the “system” you are trying to model works. More often than not, at the beginning of the model building process, there is some subjective opinion about what are going to be the likely factors influencing the thing that you are trying to predict. So where do these opinions come from? They usually come from the people who are knowledgeable or experts in that particular field. We sometimes called this the “domain expertise”. If we take the example of the econometrician predicting the quality of wine, the econometrician was also a wine buff so he had some previous knowledge about what the likely factors were that could potentially affect the quality of a particular vintage. His skill was in quantifying it.

In the same way, some domain expertise is needed in the development of good tests. If we look at MVT then the technology can help you determine which the best page design to use is. If you test 4 different versions of an element (say a call to action), then you will get a winner. That “winner” may be the one that you started out with, but it’s still the winner. It doesn’t mean though that it’s the best one, it’s just the one that was best out of the various options that you looked at. There may be a much better option out there which you haven’t tested. Usability experts can potentially provide better insights into what versions are the best ones to test in the first place, and also help to understand why the results have come out the way that they have.

So we need the experts to help us build better models. That expertise may come from years of experience or knowledge gained from understanding the effectiveness of previous models. In either case, there’s room for both the science and the art.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

Last time I looked at the importance of getting a good understanding of the online user experience through tools such as surveys and other feedback mechanisms. The development of technology has meant that these types of tools can be obtained either for free or relatively cheaply, lowering one of the barriers to their adoption. However, just because the tools are cheap it doesn’t mean that the survey needs to be cheap as well, as asking for user feedback is all part of the user experience. So if you’re looking to run surveys yourself rather than use and agency to do it for you, then here are some things to consider along the way.

Be clear about the purpose of the purpose of the survey
Getting some user feedback is better than getting none at all but don’t do a survey just for the point of doing a survey. Be clear about what it is that you want to know. Write down the objectives of the survey and keep it hard and focussed. You may have more than one objective but don’t try to answer everything in one go.

KISS - Keep it Short and Simple
It’s difficult to say exactly how long or short a survey should be, but try and be as efficient as possible. Stick to the objectives and avoid the temptation to cram too much in. If necessary do more than one survey on a different sample of users or only ask certain questions to certain people. Also be careful of the language that you use and the style of the questions that you ask. It’s easy to slip into either using your own terminology that you would use in the business or “market research speak”. Your users probably won’t understand either. Above all make it interesting. Make sure that the questions are relevant and the survey is engaging.

Work out how much data you need
Whilst some data is better than no data, be wary of basing decisions on a small number of responses. A rule of thumb I use is that 400 responses will give you a reasonable level of accuracy in the answers, getting loads more won’t necessarily give you a lot more accuracy but it will mean that you can filter the data to look at sub-groups (ie the differences in responses between young people and older people for example). Once you’ve worked out how many respondents you are likely to need then you need to work out how many people you need to ask to get them. The response rate on surveys can vary enormously so if you haven’t run surveys before on your site you may need to do some test first, which brings me onto my next point.

Test before going live
Test the survey on a small number of users before going fully live. This is especially true if you have a survey where the questions that people are asked depends on the answers that they have given before. You need to check that the survey works technically well and that also all it works well from the user perspective. It’s best to spot mistakes early!

Keep it ethical
Use your surveys for research purposes only. Don’t use them to try and sell people things.

See your survey as an extension of your brand
If you are using online surveys on your site then they are part of the user experience. Ensure that the tone, design and look and feel of the survey are complementary with the brand. You don’t want your users having too different experiences. I had a good reminder of this when I was asked to take part in an online survey for a well known technology brand. The brand image is all about ease of use and good design. The survey was badly designed, looked horrible and was very difficult to complete. I was so annoyed by the survey that I wrote and told them what I thought. Which brings me to my next point…

Be prepared for feedback
You will probably get more feedback that you anticipate. People will either reply to an email invite or will write comments in open text boxes. You will need to have processes in place to deal with any customer service issues that may bubble up. Many of these may be nothing to do with the website but they need to be dealt with.

Hopefully these few tips will help you along the road to getting some useful and interesting feedback from users about their online experience. With the range of tools available there’s no reason not to get started but make it a good experience!

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

One of the trends over the past couple of years has been the growth in the number of organisations who are using some form of visitor survey tool on their websites. It wasn’t so long ago that when running workshops I would be asked how many people were running surveys on their websites, and maybe 20% of people would put their hands up, these days its probably around 50%. That crude survey itself is indicative of the wider adoption of visitor feedback mechanisms as part of the digital analytics toolkit.

This adoption has been caused by two main factors I think. First of all there has been the growth of availability of online survey capabilities across all levels of sophistication and these surveys have become more “productised” which makes them easier for organisations to buy and deploy. Examples within the customer satisfaction measurement space include 4Q at one end of the spectrum, which is a limited but free survey tool, through to more enterprise level products such as Forsee Results and iPercpetions.

The second driver of growth I think has been the realisation by organisations that they can’t measure the effectiveness of their digital marketing strategies by purely looking at clickstream data. Web analytics tools can tell you what happened in terms of visitors’ behaviour and when it happened, but they are not necessarily the best tools for telling you who did what and why they did it. That’s where survey data comes in by providing this different perspective. By asking people questions about themselves, why they do what they do and what they think, it’s possible to fill in some of the blanks left by the volumes of clickstream data at our disposal.

As with all measurement and analysis tools and systems, the amount of thought and preparation put into configuration and deployment pays dividends later on in terms of the quality and robustness of the data. Survey tools are no different. There are various approaches that an organisation might take to developing and launching a survey.

First of all they may choose to outsource the whole thing to an agency to manage on their behalf. Here the agency would be responsible for designing the questionnaire, scripting the questionnaire in whichever survey tool they use, deploying the survey, collecting the data and analysing the results. This is the approach that most organisations take when doing offline market research and there’s nothing wrong with using the same approach online. The main concern of the organisation commissioning the research is to ensure that the research objectives are clear and aligned to their business objectives, to agree the questionnaire and also to ensure that the survey is fit for purpose and holds up to the brand values. This last point is particularly important as there is evidence that suggests that poorly executed online surveys do potentially damage the brand whether they are launched on the site or when people are invited to take part via email. I have certainly been on the receiving end of some surveys where I’ve thought that the style of the survey was completely at odds with the brand.

The second option for an organisation is to choose to design and manage the survey themselves. Certainly these days there are no end of free or cheap survey tools that allow you to run surveys of varying complexity. Quite often a provider will provide a basic version for free or at a low cost which has limited functionality and data capture limits, they also offer a higher end tool which allows more complex questionnaires to be designed and more responses to be captured. However just because the tool itself is free, it doesn’t mean that the survey doesn’t require the same sort of diligence in its preparation and deployment than you were using a more complex, enterprise level product. One of the dangers is that surveys deployed using cheap tools with little effort put into them look cheap themselves and may have a negative impact on the user experience and their perception of the brand.

In my next article I will outline some tips for maximising the effectiveness of your survey efforts.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

As I wrote in my last column, 2009 is going to be a challenge for digital analytics and insights teams as they will need to get leaner, smarter and faster. I also think there’s a big opportunity to be grasped in terms of increasing the impact that these teams have on the corporate entity. As planning horizons and measurement cycles get shorter, there will be more of an emphasis on understanding and optimising the direct response model. This is where digital and direct marketing analysts traditionally ply their trade.

However, I don’t think it’s merely a question of doing more of the same and doing it more often, I think that now is the time when there is an opportunity for digital analysts to broaden their remit and expand their influence on the organisation. As Tom Davenport (author of “Competing on Analytics) puts it: “The planets are aligned for analytics”. What he means by that is that the necessary fundamentals that will allow analytical approaches to thrive in business, are in place. We have the data and we have the technology to manage and manipulate that data. With the economic situation we find ourselves in, we also have the motive. Organisations will be looking for solutions and digital analysts are well placed to provide them. To do that though, I think that digital analysts need to increasingly position themselves as data integrators and cross-channel specialists.

Too much data sits in silos and this is particularly true in the digital channel. We have web analytics data, campaign data, survey data, customer data and so on. Quite often each of these data sources will have a different business owner either within the organisation or in an agency. The value of each data source is diminished when housed and analysed in isolation, the full value comes from layering in data sources to look at different aspects of a particular problem. To do this analysts need to become masters of the data universe and be able to understand and leverage these data sources. They need to understand the nuances of each data source and be able to explain these nuances to business users who may not understand the differences between the different types of information at their disposal.

So on one level digital analysts have the opportunity to deliver increased value by taking a more holistic view when addressing issues in the digital channel. On another level I believe that digital analysts can take the lead in the development of cross-channel analytical approaches. At the heart of cross-channel analytics is customer analytics. However, quite often in organisations today customer analytics tends to ignore the digital channel and certainly the interaction between the online and the offline channel is often not well understood.

For many industries the internet has become the primary tool for researching brands, products and services even if the final transaction takes place offline. One only has to look at travel and financial services as two examples where it is probably safe to say that the vast bulk of product research takes place online. Since the internet operates “upstream” in many customer acquisition processes, it’s in the interests of the digital community to understand and demonstrate the influence the channel has in the “downstream” conversion process irrespective of whether it happens online or offline. I see therefore the development of cross-channel measurement techniques as being the domain of the digital analysts so that the full and true return on investment in the channel can be better understood. Once better cross-channel measurement processes are in place it becomes possible to develop better cross-channel marketing processes.

So if the planets are aligned for analytics generally, then it’s certainly time for digital analytics to come in from the cold and exert its influence beyond the realms of a single data source in a single channel. Data integration and cross-channel analytics is becoming the name of the game in 2009.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

As you’re reading this, probably, like me, you’re already wondering what 2009 might bring. At this time of year I often wish I had the attributes of the Roman god Janus (with his ability to look backwards and forwards at the same time.) It’s easy to look back and review a year, it’s harder to look forward and see what might be. Having said that, this time last year I wrote in this column “If I’m to make one more prediction for 2008, it would be that I don’t think it’s going to be a dull year!”. Sometimes you can wish that you were less prescient! It’s been anything but dull and for a lot of us this is the first time we are experiencing anything other than relative economic stability and growth.

If you’re in the forecasting business then what you are mostly doing is identifying patterns and extrapolating trends. What makes forecasting difficult is when you suffer “shocks” to the system that radically change the way that the “system” works. On our news at the moment we hear comments like “nobody anticipated this”, “these are unprecedented times” and so on. This makes it hard to extrapolate any trends because the trends have been broken or disrupted. We can only look back to the last time such events occurred (decades ago) and try and get some signals from all the noise. The problem is that looking back to the last occasions when our economy went into recession is also not likely to help much. Circumstances were very different back in the late 80s and early 90s. For a start, technology did not impact our lives in the same way and the internet was in its infancy. Fax and telex were still the primary forms of “immediate” communication. With such a difference in our technological capabilities what can we learn as marketers from previous recessionary environments? Probably not a lot.

The other difference is the scale of the problem. Historically one global economy might be in recession when others might be stable or in growth. Multi-national companies could play a “portfolio” game, managing losses in some markets whilst seeing compensating growth in others. This time round there are no safe harbours.

If there is one prediction I would make about 2009, it would be that 2009 is going to be unpredictable. The degree of turbulence and volatility means that it will be difficult to separate the signals from the noise. One result I think will be that planning horizons are going to get shorter, from years to months and from months to weeks. As a result of that, measurement cycles are going to get faster. In turbulent times people are going to want more frequent information and better insight delivered more quickly. Analytics and insight teams (if they still exist) are going to be under pressure to produce more goods of a higher quality with a faster turnaround. For organisations this has implications around the use of the human and technological resources they have at their disposal.

Whenever I write or talk about this subject, I always get the same mental image. It’s of a boat or yacht sailing in a storm. On a calm day you can see for miles ahead and you probably don’t need to check your maps and equipment that often. You have the time to plan and plot your course ahead and the navigator has time for a coffee break. In a storm you have to be much more reactive to the circumstances you find yourself in. You can’t see that far ahead and so you need to rely on your equipment to tell you where you are and what to do next. You need to quickly communicate your decisions to those around you and the navigator is chained to the desk.

Without hopefully running the risk of torturing the metaphor, I see it the same way for organisations heading into 2009. We know the storm is coming but we may not know exactly when or where or how bad it’s going to be. We don’t know exactly what the impact will be and so we can’t predict how we may need to react. But we do know that the storm is coming and so we need to get ourselves prepared. For analytics and insight teams now is the time to make sure that the technology is working properly, the data is sound and that the business processes are in place. If you’re in the right kind of organisation you’re going to be having a very busy, challenging but hopefully rewarding year.

May I take this opportunity to wish you all the best for 2009.