Neil Mason

One of the toughest aspects about analytics is not getting the right data or even getting the right data right. It’s not about making sense of the numbers or extracting meaningful insights. All of these things can be tough. But the toughest challenge? Getting the data or insight actually used in organisations and getting at the heart of the decision-making processes. There are two issues here: accountability and relevance. As soon as you want to start measuring things, you start to make people accountable, and inevitably there can be resistance to that. However, another challenge is having data and metrics that are relevant to what the business or a person is trying to do. If metrics are not relevant, then they won’t get used.

The main challenge to getting relevant metrics is aligning them as closely as possible to the objectives. If metrics aren’t aligned to what people and organisations are actually trying to do, then people aren’t going to take much notice of them. All too often metrics are “retro-fitted” into an organisation because they’re available rather than because they’re useful. However, aligning metrics with objectives is often easier said than done; frequently the problem lies with the nature of the objectives themselves.

Objectives are often like marshmallows: they look good on the outside but they are soft and squidgy on the inside. Marshmallow objectives don’t really describe what the end result looks like and do not give an idea of whether the result is being achieved. An example of a marshmallow objective? For the development of a new checkout process, such an objective would be “to improve the user experience and build loyalty.” However, objectives should be as SMART as possible – i.e., specific, measurable, achievable, realistic, and time bound. The key to getting objectives as measurable as possible and having relevant metrics is to make them as specific as possible. You’ve got to ask yourself “what does good look like?”

Asking yourself what good looks like is a useful way of creating harder and more specific objectives. It’s a particularly useful technique when there may be no immediate or clear financial outcome. It’s easier in my checkout example, above, to make the objective “smarter” by setting specific goals in terms of the increase in orders as a result of the improved experience. But if you’re developing a new section of content or a new site, how do you get smart objectives?

The approach is to keep asking the question “what does good look like?” and trying to describe what will be happening on the site, off the site, or wherever that objective is being achieved. For example, you may be looking to refresh the content in the help and support section but what will good look like afterwards? Hopefully more people will be able to find the help and support they need more easily, meaning that they don’t need to call the call center or send in an email. It should also mean that people are more satisfied with the experience than they were before and would be more likely to use the help and support section.

By drilling into what good looks like, we can begin to define metrics that can then measure whether these desired outcomes are being achieved or not, either on the site or in the contact center. Hopefully the number of emails and calls for certain types of queries will be going down and customer satisfaction levels will be going up. Going through this exercise may show that not all the data that is needed is currently available. Actions may need to be taken to improve the configuration of the web analytics systems, to introduce a voice-of-the-customer program, or to extend the scope of an existing one. In some cases it may mean getting access to data that sits in another part of the organisation such as the contact center.

This simple approach to creating objectives that are as specific as possible can be used at all levels and in many ways, from determining the success of the whole digital channel through to understanding the effectiveness of a particular piece of product development or a specific campaign. So each time you come to measure something, ask yourselves: Why are we doing this? What will good look like? How can we measure success?

This article was originally published on ClickZ

The statistics are compelling. Here’s an example: “Poor online user experience, coupled with a lack of insight about why customers are abandoning websites, is costing businesses billions of dollars / pounds. Companies able to quantify site abandonment estimate they are losing the equivalent of 24% of their annual online revenue due to a bad online experience. This equates to more than $50 billion lost in the US and around £14 billion lost in the UK in the last year.” (Source: Econsultancy/Tealeaf report 2011). There are many others out there like that that show the impact that poor customer experiences have on customer loyalty and ultimately business performance. But as the old saying goes, “If you can’t measure it, you can’t manage it,” so organisations need to build up their ability to track and diagnose the customer experience.

Tracking the customer experience, though, is a multi-faceted challenge. Experience is broadly an attitudinal outcome that results in a set of behaviors. The digital marketing industry has historically focused on tracking the customer experience through behavioural observation using tools like web analytic systems. This has had limited success, as it’s possible to see what is going on (like a shopping cart abandonment), but it’s not that easy to see why it’s happening (poor usability, price issues, etc.). If measuring and understanding the customer experience is a multi-faceted challenge, then a multi-faceted approach to the problem is required.

The diagram below maps out the customer experience data ecosystem. The ecosystem is two-dimensional. One dimension is from behavioural data to attitudinal data, and the other dimension is from tracking to diagnostic. There are five main classes of data/tools that sit in the ecosystem.

Web Analytics Systems

Web analytics is a mainstay of the customer experience data ecosystem but it’s not the only game in town. Web analytics data is great for tracking what’s actually happening on a site but it can be limited in its ability to help the business understand why it’s happening and so has limited diagnostic utility. This is mainly due to the fact that most data in web analytics systems is reported at the aggregate level and it can be difficult to isolate and understand individual user behaviour.

Customer Experience Measurement Systems

This class of tools encompasses systems such as Tealeaf, ClickTale, and others. These types of tools are still mainly focused on tracking user behaviour but have the ability to be highly diagnostic, as they can be used to isolate out and analyse either a small number of users behaving in a particular way or even reviewing individual sessions. These tools give the ability to do deep forensic analysis of a particular problem but are best utilised alongside other tracking systems to identify potential issues in the first place.

Voice of the Customer Programmes

As can be seen from the diagram, a voice of the customer (VoC) programme can straddle all four quadrants of the ecosystem and is therefore a vital component of any customer experience measurement approach. A VoC programme can track both behaviours (or at least claimed behaviours) as well as attitudes over time. Tracking customer satisfaction or Net Promoter scores is a common output from any VoC program. A lot of the value from the program, though, is from the diagnostic capabilities either from an analysis of the quantitative data or from the rich insight often available from users’ actual responses from open-ended questions such as “How could we have improved your experience today?” Smart companies are integrating VoC programs with customer experience measurement systems described above to get at the root issues behind customer dissatisfaction and then working out what needs to be fixed.

User Testing

User testing, either in the laboratory or by using remote testing technologies, is a rich diagnostic approach and works well alongside other core tracking systems such as web analytics or VoC programs. Lab-based testing is often done with small samples with respondents being asked to complete a task while the consultant observes their behaviour, but also can listen to what they are thinking by means of what is called the “think aloud protocol.” Remote testing can use larger samples, which are robust enough to generate some metrics that could be used as part of a tracking system.

Experience Research

The experience research piece contains a variety of techniques that are often highly qualitative and attitudinal in their approach. Classic techniques include the use of focus groups, which can be offline or online, whereas other techniques such as ethnography are increasingly being used to get real insight into how users experience brands and services.

Social Listening

Social listening is a relatively new addition to the customer experience data ecosystem. Feeding off social media “buzz,” it tends to be attitudinal in nature but can present challenges in “extracting the signals from the noise” when looking to understand the customer or user experience. It is often used for sentiment analysis and tracking but is probably more useful from a customer experience perspective for classification of comment into different groupings in the same way that you might classify survey comments.

Measuring and understanding the customer experience requires a multitude of different tools and systems generating different styles and types of data. Each on their own provides a limited perspective, but when aggregated together can give an organisation the ability to measure and manage the customer experience.

 This article was originally published on ClickZ

It’s great these days, isn’t it? When it comes to measuring and analysing the performance of your digital channels, there’s an abundance of free or cheap tools that you can use across the spectrum of different measurement technologies. At the last eMetrics conference I went to, there was a session devoted entirely to the best free tools there are out there, whether they be web analytics tracking tools, surveys tools, competitive analysis tools, social media tools, or site performance tools. You don’t have to be a large, sophisticated online marketing organization to be able to collect lots of data cheaply. Unlike other marketing channels, the marginal cost of data is very low.

So all this free or cheap data must be a good thing, right? And the return on investment (ROI) must be really good since you don’t have to spend a massive amount on getting the data. Right? Well, I think it all depends on what you do spend your money on and what investments are actually made. Here’s a scenario we see a lot with the daddy of free tools, Google Analytics.

An organisation has a website (or possibly multiple websites). It decides that it needs to do some tracking to see what’s happening but doesn’t want to spend too much money at this stage, and so decides to use Google Analytics. This is fine; Google Analytics is a strong, robust web analytics tool with a user-friendly reporting interface. Next, a developer puts the tag code on the site, data begins to flow, and after a while pretty charts begin to appear. At this point, things look good, data is coming in, and not a lot of effort or cost has been expended.

Next, the organisation starts to try and use the data. Some of it seems OK but other bits may not look right. People start asking questions, but it proves difficult to get the answers from the data that’s in the reports. After a while, people begin to question the value or even the integrity of the data. So while not much has been expended, not much has been gained either, so the ROI is not very good. The organisation may struggle on for a bit trying to get better data or reports, but ultimately remains frustrated. At this stage, the organisation may look for help. One option would be to invest in developing or hiring a skilled resource within the organisation and to dedicate someone (or part of someone) to looking at the data and creating some insight from it. Other options include using a consulting organisation, and often this is where we get involved.

A typical approach would be to understand what the business is trying to do and therefore what it needs to know. By understanding what the organisation needs to know, it’s possible to work out what data needs to be collected, and once that’s done, it’s possible to figure out how the code on the site needs to be configured to ensure the right data is collected in the right way. The reporting interface can then be configured so that people get access to the right kind of reports they need rather than a “one size fits all” approach to reporting. Often we’ve seen that relatively modest levels of investment in either people or professional services start to dramatically increase the integrity, utility, and value of the data that the organisation is collecting and reporting. The ROI starts to improve as decisions are made on the data that improves business performance or reduces risk.

I’ve used Google Analytics as an example here as it’s a widely used tool in the digital space, but the arguments apply to the many cheap or free tools out there. Free or cheap data isn’t necessarily good data until someone starts to add some value to it, and organizations are unlikely to see any return until some investment is made in ensuring that the data is relevant to the needs of the business and fit for its purpose. Don’t get me wrong, I’m all for cheap data, but don’t fall into the trap of thinking that just because it’s free that there isn’t going to be any cost of ownership.

This article was originally published on ClickZ

A key trait of an analytically empowered organisation is its investment in “humanware,” the right kind of people to extract value from the data and create insight. But what does “good look like” when it comes to analysts?

A good analyst has the capacity to analyse data and generate insight. Let’s have a look at those words in a bit more detail. An analysis represents a “detailed examination of the elements or structure of something” and insight is defined as “the capacity to gain an accurate and deep understanding of someone or something.” On that basis, the analyst needs to be able to look at data in detail and understand it. But it depends also on what you mean by “understand.” In addition to understanding the data, an analyst must also understand and communicate what it means for the organisation. So here are five things to look for in an analyst:

Data Dexterity

A good analyst must be able to handle data – any data. While analysts may have a functional speciality, such as a web analyst or a customer insight, they have to be comfortable in handling any data sources they encounter. In the data world, two and two really does equal five. That’s because the value of data from one source increases when it is integrated with other data sources. Web analytics data is great, but it becomes even more valuable when it’s analysed alongside voice-of-the-customer data, customer experience data, and other customer data.

Pattern Recognition

For me, analysis is a blend of the creative and the deterministic. A good analyst can look at a set of data and begin to see what the data is telling her. Data tells stories and the analyst is there to interpret the data and make sense out of what the data is saying. It’s all about extracting signals from the noise.

Attention to Detail

On the other hand though, an analyst must have strong attention to detail. Data can be very nuanced at times; it might look like it’s saying one thing when in fact it’s saying something else. Or the data may be wrong. A good analyst must be able to spot problems with data that just doesn’t look right.

Commercial Awareness

An analyst must be aware of the ramifications of the what the data is saying. She needs to add value to the analysis by not only explaining, “This is what’s happening and why” but by also elaborating on “This is what I think the impact is” and “This is what needs to be done about it.” The latter analysis tends to come with experience, but any analyst at any level needs to interpret outcomes and not just outputs of their work.

Positive Presence

All the above counts for nothing if the analyst cannot get her message across within the business. She must articulate and communicate what the business needs to know. So good analysts need to have good verbal and written communication skills, possess the ability to construct an argument based on evidence, and tell stories.

A lot of these characteristics can be learned or developed over time. If you are looking to hire talent, be sure she is comfortable with data and has the ability to understand how the data relates to the business. When we look to hire an analyst, we ask candidates to create a presentation from a set of data. We want to know not only that they analyse the data but whether they can make sense of it and then articulate and present that message in a clear and convincing way.

This article was originally published on ClickZ

Fifteen or so years after organisations first started to measure what was happening on their websites by parsing log file data, what does good look like? There are five characteristics that define an analytically empowered organisation.

• Clearly defined key performance indicators (KPIs)
• Holistic approach to measurement
• Integrated data strategy
• Investment in “humanware”
• Ability to execute

Clearly Defined KPIs

At analytically empowered organisations, considerable effort goes into defining digital key performance indicators. To do this, the digital strategy must be clear and coherent. If the strategy isn’t clear, how can you possibly measure its success? In my experience, defining good, robust KPIs is not an easy task. As a result, KPIs are often not very good. Going through the process forces an organisation to think hard about its strategy, define what success looks like, and make a commitment to measurement. If you can’t measure it, then you can’t manage it.

Holistic Approach to Measurement

The old saying goes, “If the only tool you have is a hammer, then every problem looks like a nail.” Ever since the log file was developed, the digital marketing industry has been banging away with its web analytics hammer. The analytical empowered organisation understands that it needs a whole toolbox. Web analytics provides some but not all of the answers about digital performance measurement. It’s great for telling you what is going on but even a well-configured web analytics tool (itself a rarity) isn’t very diagnostic. Organisations need to invest in additional quantitative and qualitative data sources to truly understand what is going on and why. Additional investment requirements include voice-of-the-customer feedback on a number of levels (based on visit, page level, and post-experience), ongoing user experience measurement and analysis, site performance tracking, and contextual information about trends in the marketplace.

Integrated Data Strategy

A holistic approach to measurement also requires a unified approach to data integration. An organisation also needs to understand how all the pieces of the jigsaw fit together. This requires some effort around data definition (what the metrics actually mean) and where different types of data will be housed. In an ideal world, data is integrated around known users but this may not always be appropriate or possible. Different data types have different characteristics, so planning is needed to understand how the different components fit together. For example, some internal data may be on a customer level, but digital data is often based on cookie level data and one customer may use a number of different devices to interact with the organisation resulting is a number of different cookie records.

One powerful outcome of data integration is the ability to match behavioral data with data around attitudes and opinions. By integrating web analytics data with voice-of-the-customer data, it’s possible to look at the relationship between what people do in a digital channel and the experience they have. This type of integration gives the organisation the ability to measure outputs (things that happen in the channel) and to understand outcomes, which are often the most important things to know.

Investment in Humanware

All the hardware and software in the world will get you nowhere without “humanware” to extract insight and value from data. Too often in the past, investments have been made in technology without appropriate investments in people. The result is often disappointment, if not failure.

Today, analytics teams are taking a more multi-disciplinary approach. As data becomes more integrated, an integrated approach to analysis and insight is needed as well. Web analysts must start working alongside customer insight specialists and user experience experts, sharing their knowledge and expertise.

Ability to Execute

Organisations gain a competitive advantage from the application of insight, not by the generation of insight. Insight has no value unless something happens as a result. So the analytically empowered organisation has the ability to execute and make decisions. This has implications beyond the immediate concerns of analytical technologies; it also concerns a business’s entire technology landscape. Often, a product or site development process and technology constrain an organisation’s ability to affect change. So the analytically empowered organisation must develop strategies in technology and processes that enable it to act on its insights.

This article was originally published on ClickZ

While I have no doubt Albert Einstein was talking about digital marketing when he said, “Not everything that can be counted counts, and not everything that counts can be counted,” I have also come to the view that Mark Twain was an analytics guru. While possibly most famous for his “There are lies, damned lies and statistics” quote, I have discovered a number of other quotes that demonstrate his understanding of the world of analytics. Here is a selection of some of them:

“All generalisations are false, including this one.”

Here he was talking about segmentation. Generalised metrics such as overall conversion ratios, average time on site, and so on are largely useless. They only make sense when the data is segmented and the underlying patterns and differences can be understood. The same is for survey data as well. Metrics such as NPS can vary wildly among different customer segments and so these segments need to be identified and looked at separately.

“Few things are harder to put up with than the annoyance of a good example.”

A great insight into the power of evidence-based decision making. Solid facts, presented well, are hard to ignore. Twain was probably also thinking about the need to have an approach to testing and experimentation when he said this. There’s no better way to silence the HIPPO (highest paid person’s opinion) than to have concrete evidence from testing about why one approach is better than the other.

“It’s no wonder that truth is stranger than fiction. Fiction has to make sense.”

Sometimes the data doesn’t make sense and the skilled analyst needs to understand why. Is the data telling her something new? Is this real insight or is there a problem with the data? Often, if it doesn’t seem right, then it probably isn’t right, but occasionally there may be something that the business has missed, the competition has missed, and there is real opportunity to be exploited. Just make sure you’ve checked the data thoroughly before you tell the CEO!

Twain also had some great advice for budding analysts:

“The more you explain it, the more I don’t understand it.”

This is a great piece of advice. Basically, keep it simple, but don’t be simplistic. A good analyst will be able to present quite complex notions in ways that are easy for people to understand. This is the art of storytelling. Developing compelling narratives that engage the audience is quite a skill. There’s an important role for data visualisation here as well. It’s harder to create simple charts and graphics than to scatter lines and bars all over a PowerPoint slide. That’s why an analyst also has to be an artist.

“I was gratified to be able to answer promptly, and I did. I said I didn’t know.”

I think it’s OK to say that you don’t know; it’s better than bluffing…

“It is better to keep your mouth closed and let people think you are a fool than to open it and remove all doubt.”

…and being found out later, as long as you say that you’ll find out the answer and come back to them. And then make sure you do.

And finally:

“The most interesting information comes from children, for they tell all they know and then stop.”

Don’t be afraid of silence, particularly if you’re presenting to your audience. Deliver your insight and wait for a response. Maybe ask them a question, and then wait for a response. “Is that something you recognise?” and “Does that fit in with what you know?” are all ways of seeing whether what you’re saying is hitting the mark or missing the mark. It’s better to find out in the course of a conversation that what you’re saying is not falling on fertile ground rather than just to be ignored later. Maybe you need to get more evidence or maybe you need to explain the point in a different way.

So that’s why Mark Twain was an analytics guru!

This article was originally published on Clickz

The left side of our brain tends to be associated with functions that are analytical, rational, logical, and objective. The right side of our brain tends to orientate to creativity, intuition, and flexibility. In analysts, the left-hand side of the brain will tend to dominate, whereas with creatives and designers, the right-hand side will be stronger.

Optimisation using testing and experimentation technologies (such as Adobe Test and Target, Webtrends Optimize, or Google Website Optimizer) is mainstream for a lot of organisations. Companies such as Dell have built teams and processes to drive testing and experimentation through the business. Those companies have learned – and others are painfully discovering – that successful testing and experimentation is not only about implementing one of the many available technologies. It’s also about the need to harness people, resources, and processes around technology.

It’s similar to web analytics a few years ago. Back then, organisations implemented a system thinking that it would solve their measurement problems but then realised they needed analysts. Likewise, organisations must build structures and processes around testing and experimentation technologies, otherwise the business will not extract the potential from the system.

Testing and experimentation involves a lot of moving parts. Tests must be designed, assets created, technologies configured, and results analysed. Successful testing and experimentation programs require strong project and program management capabilities. Larger organisations typically have dedicated resources for program management, whereas in smaller businesses it might be part of someone’s role. In either case, a central function must identify which tests are planned and then manage them through the system. Workflows must be created to ensure that the assets to be tested are created and deployed onto the system at the right time. Tests must be monitored to ensure that any variants that are adversely impacting the experience can be dealt with.

Two Places Where Right-Brained Creatives Can Assist With Testing

All of this is predominately “left-brain” activity, i.e., managing, coordinating, analysing, testing, and experimenting also needs “right-brain” input, a more qualitative approach incorporating a user experience perspective. This right-brain input is important into two areas:

• Test program development (what to test)
• Test design (how to test)

Test programs are often built on the basis of web analytics reports showing which parts of the site might have problems. An additional input into the test program can come from understanding what’s working and what’s not working from the user experience perspective. The main sources of insight are from voice-of-the-customer survey programs and user experience testing. Many organisations have ongoing survey programs and many elicit user feedback through open-ended questions such as “How else can we improve the site?” User feedback can be a rich source of insight, but it must be mined, contextualised, and interpreted. These are right-brain attributes. This qualitative input helps to define what are the important areas of the site to improve and where to direct testing.

Second, right-brain input is needed for test design. Once a test area has been decided, the next issue is to decide what different elements will be tested. In a test, there will always be a winner even if it’s the existing version. With testing and experimentation technologies, you can cycle through many different combinations until there’s a significant improvement. But the challenge is how do you know that the variants that you’ve decided to test are the best ones? How do you know that the winner is not the best of a mediocre bunch? Optimisation specialists may know that certain things tend to work from the body of tests they’ve seen, but other inputs such as user experience expertise help to improve the quality of testing.

Good testing and experimentation is a combination of art and science, rational approaches and intuitive perspectives, and left-brain and right-brain inputs. It’s time to take a whole-brain approach to testing and experimentation.

This article was originally published by ClickZ

This week, I’ve assembled five favourite quotes that I use when discussing analytics.

“Not everything that can be counted counts, and not everything that counts can be counted.”
- Albert Einstein

There’s some debate whether Albert Einstein said this or whether it was only on a sign hanging on his office wall at Princeton. No doubt, he was referring to digital marketing! In our world, there’s no shortage of numbers. Instead, our problem is often having the right numbers. For me, this quote epitomises the challenges of creating good key performance indicators (KPIs). We can measure a lot these days. But just because we can measure something, it doesn’t mean that it’s important. Quite often the important things to measure in business, our KPIs, are hard to measure, and it’s often because we are thinking about measurement in the wrong way (see next quote). The challenge is to count the things that count (good KPIs) and to get the right numbers right (good data integrity).

“If your only tool is a hammer, every problem looks like a nail.”
- Abraham Maslow

For a long time, digital marketing analysts would try to solve every problem with data from a web analytics system. It’s getting a lot better, but there’s still a lot of it going on out there. We don’t need a single tool; we need a whole toolkit. Having good web analytics is absolutely necessary, but rarely sufficient. In addition to web analytics, we need good voice-of-the-customer intelligence, an understanding of the competitive situation, and an appreciation of the actual technical performance of the channel. This has implications for the skills base and training for analytics teams within organisations and also levels of investment required.

“The price of light is less than the cost of darkness.”
- Arthur C. Nielsen

Art Nielsen was a pioneer of modern marketing research. This quote characterises the challenges in getting organisations to spend on measurement and analytics as they often view it as a cost rather than an investment. These days investment in data collection technologies may not need to be that high. There are lots of free tools for web analytics, voice-of-the-customer research, site performance measurement, and so on. However, there is no point collecting data if you don’t have resources to analyse, interpret, and act on it. It’s not a question of whether you can afford to invest in these resources, as Art said, it’s a question of whether you can afford not to. This quote should be written in the footer of every business case for investment in analytical resources!

“Most people use statistics the way a drunkard uses a lamp post, more for support than illumination.”
- Mark Twain

This one is one of my favorites. The weekly report is the scourge of most organisations. It goes out every week, because it’s the weekly report. Most likely, this report was started by someone who is no longer in the organisation. Most likely, it doesn’t tell anybody anything that they need to know. Next week, don’t send out the weekly report and see what happens!

“After all is said and done, more needs to have been done than said.”
- Neil Mason

OK, I cheated here, but this is along the lines of “actions speak louder than words.” The whole point of measurement and analysis is that it leads to different outcomes. Analytics is about making better decisions and reducing risk in the business. There’s no point having good data and sophisticated analysis if the results are not acted upon, either because of the organisation’s culture or business processes. Successful use of analytics requires organisational agility to bring change and measure results. That’s what we mean by optimisation.

So those are my favorite quotes about analytics. Has anyone got any others?

This article was originally published at ClickZ