• Clearly defined key performance indicators (KPIs)
• Holistic approach to measurement
• Integrated data strategy
• Investment in “humanware”
• Ability to execute

Clearly Defined KPIs

At analytically empowered organisations, considerable effort goes into defining digital key performance indicators. To do this, the digital strategy must be clear and coherent. If the strategy isn’t clear, how can you possibly measure its success? In my experience, defining good, robust KPIs is not an easy task. As a result, KPIs are often not very good. Going through the process forces an organisation to think hard about its strategy, define what success looks like, and make a commitment to measurement. If you can’t measure it, then you can’t manage it.

Holistic Approach to Measurement

The old saying goes, “If the only tool you have is a hammer, then every problem looks like a nail.” Ever since the log file was developed, the digital marketing industry has been banging away with its web analytics hammer. The analytical empowered organisation understands that it needs a whole toolbox. Web analytics provides some but not all of the answers about digital performance measurement. It’s great for telling you what is going on but even a well-configured web analytics tool (itself a rarity) isn’t very diagnostic. Organisations need to invest in additional quantitative and qualitative data sources to truly understand what is going on and why. Additional investment requirements include voice-of-the-customer feedback on a number of levels (based on visit, page level, and post-experience), ongoing user experience measurement and analysis, site performance tracking, and contextual information about trends in the marketplace.

Integrated Data Strategy

A holistic approach to measurement also requires a unified approach to data integration. An organisation also needs to understand how all the pieces of the jigsaw fit together. This requires some effort around data definition (what the metrics actually mean) and where different types of data will be housed. In an ideal world, data is integrated around known users but this may not always be appropriate or possible. Different data types have different characteristics, so planning is needed to understand how the different components fit together. For example, some internal data may be on a customer level, but digital data is often based on cookie level data and one customer may use a number of different devices to interact with the organisation resulting is a number of different cookie records.

One powerful outcome of data integration is the ability to match behavioral data with data around attitudes and opinions. By integrating web analytics data with voice-of-the-customer data, it’s possible to look at the relationship between what people do in a digital channel and the experience they have. This type of integration gives the organisation the ability to measure outputs (things that happen in the channel) and to understand outcomes, which are often the most important things to know.

Investment in Humanware

All the hardware and software in the world will get you nowhere without “humanware” to extract insight and value from data. Too often in the past, investments have been made in technology without appropriate investments in people. The result is often disappointment, if not failure.

Today, analytics teams are taking a more multi-disciplinary approach. As data becomes more integrated, an integrated approach to analysis and insight is needed as well. Web analysts must start working alongside customer insight specialists and user experience experts, sharing their knowledge and expertise.

Ability to Execute

Organisations gain a competitive advantage from the application of insight, not by the generation of insight. Insight has no value unless something happens as a result. So the analytically empowered organisation has the ability to execute and make decisions. This has implications beyond the immediate concerns of analytical technologies; it also concerns a business’s entire technology landscape. Often, a product or site development process and technology constrain an organisation’s ability to affect change. So the analytically empowered organisation must develop strategies in technology and processes that enable it to act on its insights.

This article was originally published on ClickZ

We’ve been working to develop a measurement framework for understanding the effectiveness of self-service and support strategies. Often the challenge with measuring the online channel is that the impact or benefits are often felt in another channel and organisations don’t often have a holistic view of customer attitudes, opinions and behaviour across multiple channels. Our approach is to look at this from the customer’s perspective and to look at their journey around the adoption and usage of online servicing and support capabilities. The framework is built on four key measurement areas: awareness, consideration, usage and performance.

Awareness

You might have the greatest online capabilities in the world but do people know about them? Most people these days would expect most organisations to provide some sort of online capability but the actual offer will vary from company to company. Are your customers aware about how they can interact with you online? Because if they don’t, then they aren’t going to be using these wonderful new applications that you’ve been developing! So the start point is to measure the degree that your customers are aware of the services and support that you can provide them online. What we are measuring here is the extent to which the promotion of your online services is effective and would typically be measured using survey methodologies.

Consideration
 
People might be aware of the range of online services available but would they actually consider using them? So the next step in the measurement framework is to understand what extent customers would consider using the online channel for different types of service or support. Once again, this is probably best understood using survey methodologies and it will be important to also understand here whether there are significant differences between different customer segments. The aspect that is being measured here is whether the proposition is sufficiently compelling enough to get people to shift from their existing channel behaviours into new ones. For example, to go online to renew an insurance policy rather than to call the contact centre.

Usage

The next step in the measurement framework is then to measure actual usage. From a measurement perspective, this is where life gets a bit easier and is probably the point of focus for many companies at the moment. Usage of online services can be easily tracked using well configured web analytics systems and also back-end transactional systems. There will be some effort involved in determining what usage metrics are the most useful ones to track and to keep the framework focussed around the most meaningful metrics.

Performance

The performance element of the framework is about understanding whether the online channel is achieving its objectives. As discussed in my last article,  it’s useful to differentiate between performance outputs and performance outcomes.

Performance outputs are those activities that are happening in the online channel that are considered to be “good”. Effectively they are the conversion points on the site. Performance output metrics might include things like the number of bill payments made online in the case of a banking environment or the number of policy renewals made online for an insurance company. For a support site, for example, output metrics may include the number of articles viewed, or downloads made.

Performance outcome metrics are the strategic measures of success. These metrics are the real KPIs of an organisations self-service initiative. From an organisation’s perspective these metrics will be looking at whether channel shift objectives have been achieved. For example, fewer bill payments being made in branches, less calls to the call centre and so on. Measuring something that is not happening is typically harder than something that is happening and so careful consideration needs to be given to how these metrics are put together. Outcome metrics also need to be considered from the customer’s perspective, such as whether they were satisfied with the online experience, whether they would use it again and also potentially whether they would recommend other people to use the online channels.

At a time when many organisations are looking to digital channels to make customer service more cost effective, a measurement framework is needed that takes a holistic approach to understanding the effectiveness of those strategies.

This article, written by Neil Mason, was originally published on Clickz.com on 01/03/11 and is republished here with permission.

From the start, social media applications were ‘C2C’, as it were – interactive, collaborative and user-centric. Consumer generated content creation and networking were fundamental to early players such as Facebook, which when it was first unleashed all of six years ago held no promise for the enterprise. It was primarily used for sharing holiday snaps and anecdotes with remote family members while avoiding the complexity of bulky e-mail attachments. Social media platforms weren’t about business, they were about people – enabling users to connect across communities sharing similar interests while pointedly leaving big brands out of it. The buzz was about corporate blogs, RSS and VoIP. ‘Viral’ still referred primarily to e-mail, not YouTube.

In recent months, however, social media applications have morphed – or perhaps been high-jacked – into fresh corporate communications and sales channels. Whereas Twitter used to be handy for telling your friends which watering hole to meet at without having to send a text message to a dozen people, today it is being ‘leveraged’ by forward-thinking brands like Carphone Warehouse to solve low-level customer service issues – and successfully so. Businesses have realised that merely dabbling in social media can prove as useless – and in some cases more damaging  than staying out of the game entirely. Major brands are now taking social media seriously, deploying enterprise-class applications that hook them into the leading social platforms in a more intelligent and rewarding manner.

Today Facebook has become just another platform (albeit one with 500 million users) through which brands such as Starbucks attempt to ‘engage’ with their customers (and flog a few more coffees). And why not? Analysts believe the opportunity for online retailers is huge. In its 2010 Social Media Report, research firm ForeSee claims that 60% of online shoppers use social media sites regularly, while 56% of that group ‘friend’ or ‘follow’ retailers that choose to engage on their networks of choice. From a business perspective, it is becoming obvious that social media is conjuring up exciting new touch-points between the retail world and consumers that talk to customers on a level they enjoy.

In his recent blog post ‘The Rise of Social Commerce’, acclaimed social media and business author Brian Solis said: “What’s clear is that the 3F’s (friends, fans, and followers) are not created equal. Those brands who examine the composition of their existing community will find that many are simply seeking access to exclusive specials and content.” It is clearly increasingly important for companies to develop social media research programmes, in order to understand what their customer base is up to in this new medium.

In fact, ‘social media’ has evolved into just another channel, adding to the lengthening list of channels all companies must keep track of, market to, measure and integrate data from – not to forget, of course, create great user experiences for.

This article was written as part of the Foviance September Newsletter

Despite all the known issues and problems with the way we measure the effectiveness of acquisition marketing activity such as the use last-click attribution models, I often wonder whether we’re measuring the right thing at all. By that I mean the point at which we define “acquisition” and therefore the point at which we determine the return on investment (ROI) of our marketing budget.

In the old days we generally considered acquisition to mean getting traffic to the site. We measured the ROI of different channels and different activity using metrics like the Cost per Click (CPC) and we managed the budgets to try and optimise the CPC. To be honest we often didn’t care what happened when traffic reached the site, we were just interested in getting them there.

Gradually that began to change as the value of a business was determined by how much money it was making rather than how many visitors it had or how many page impressions it was generating. The focus shifted to defining acquisition in terms of some conversion event, such as a transaction, a registration or the like. The point of measurement changed to further down the value chain and new metrics were used to measure ROI such as the Cost per Action (CPA). This changed the way that we viewed the effectiveness of different channels or campaigns and we discovered that some activity might be great at driving traffic but it was generally of low quality and didn’t actually generate much value. So by changing the concept of what “acquisition” meant we added in the notions of value and quality into our measurement framework. But have we gone far enough?

Obviously the meaning of acquisition is dependent on the nature of the business and the organisation involved but as a generalisation most marketing activity tends to be measured in terms of the transaction rather than in terms of the longer term customer value. Let me give you an example from a type of business where this is more relevant. Often in subscription type businesses acquisition is defined as the point of registration or sign up. CPA targets may be set based upon an overall expected life time value. But often with any subscription type model there is churn and people stop subscribing or using the service. So the notion of value is not that they necessarily signed up but the extent to which they continued to use the service and for how long.

In some work we did recently where we tracked the customer lifecycle from initial acquisition through to the point at which people stopped using the service, we found that generally is people had used the service four times, then they were very likely to continue to use it. The point of true customer acquisition in this case was when someone had used the service four times, not when they had initially signed up for it. Analysis of the effectiveness of different marketing activity in different markets showed different patterns when you compared the ROI based on initial subscription compared to actual usage. Certain types of activity were great at driving registrations but the level of initial churn tended to be quite high and so not that many would eventually turn into longer term valuable customers, whereas other activity wasn’t so good at driving great volumes of registrations but the ones that did register were of better quality and were more likely to pass the magic “four times” mark. By shifting the point of measurement and focus, radical decisions about the mix of marketing activity could be made.

Maybe it’s time to move our understanding of the effectiveness of acquisition marketing to a CPC model again but this time evaluating the Cost per Customer rather than the cost per click. Acquisition is about customers rather than actions and certainly for some businesses it’s about taking a longer term view rather than focussing in on a specific point in time.

I’m not pretending that this type of analysis is necessarily easy to do. It’s not something that really comes “out of the box” from your typical web analytics tool or from your campaign management systems. It will usually involve some level of data integration between these tools and internal data sources such as the customer database. However as we continue to try and improve the efficiency and effectiveness of our marketing spend we need to move the measurement capabilities forward and to get that “cradle to grave” understanding of the customer lifecycle.

I don’t know whether the number is right or not but a while back I heard that sophisticated marketing companies such as the world’s leading brands spent about 10% of their marketing budget on research and measurement. Irrespective whether the number is accurate or not, it’s a good benchmark I think – 10% feels about right. These leading companies have been marketing brands for over a 100 years and so over that time they have come to understand the importance and relevance to their business in investing in understanding the effectiveness of their activities.

I wonder what the equivalent proportion is for leading online businesses? How much do online businesses invest in measuring and tracking the effectiveness of their online activities? I don’t know the answer to that question but my suspicion is that the answer is generally “too little, too late”. We all know of companies that have a reputation for using analytics as part of their strategic armoury and have invested heavily in analytical technologies and also have built up formidable analytic teams. These are the companies that people travel to see and hear from at events like Emetrics and Exchange. However these companies are the exception rather than the norm.

What I write in these articles is often triggered by recent events in my consulting activities with clients. Customers and potential customers are a rich source of content! I went to visit one company last week where they had recently appointed a user experience manager and given him responsibility for web analytics and site optimisation. The client had been busy over the past six months implementing a solid and robust web measurement tracking programme. This had involved completely re-implementing their web analytics tool, hiring in a web analyst, revisiting all the business requirements and producing new reports and dashboards. He had also hired someone to specifically focus on site optimisation and to run their testing and experimentation programme. They had been busy laying the foundations and investments had been made in people, processes and technology. I could see how quickly they would begin to reap the rewards.

At the other end of scale I’ve also been working with a client who is developing a brand new site. New sites don’t come cheaply but all the way through the senior stakeholders in the business have been reluctant to invest in the appropriate measurement and analytics. The new site is close to launch and we’re now trying to shoe horn in the analytics requirements into the tail end of the development process. They use one of the free tools which can cope with most of their needs but not all of them and despite the significant investment in the new site itself, getting a relatively modest budget released to develop the data collection specification and the reporting configuration has been difficult. For me these two experiences highlight the difference between companies that “get it” and those that don’t.

For those companies that struggle to recognise the value of investing in decent measurement and analytics, I’m reminded of one of my favourite quote from A C Nielsen. Arthur Nielsen used to say that “The price of light is less than the cost of darkness”. The point is elegantly made – it’s not a question of whether you can afford to invest in measurement, it’s a question of whether you can afford not to. For me the point of measurement and analytics is to increase the effectiveness and efficiency of decision making and to reduce the risk of failure. It also leads to better accountability, which is possibly why sometimes it’s not welcomed with open arms!

Determining or justifying the return on investment in analytics can be hard. In some cases, like multi-variate testing, the ROI can be very explicit and indeed that’s how the technologies are often sold. However, working out the ROI on an analytics team and general analytics technologies can be harder, particularly in non-transactional environments. But there is always “a cost of darkness” and the trick is to try and work out what that cost might be. For transactional environments it might be not knowing how to improve the conversion ratio, for media environments it might be around not understanding how to monetise the traffic more effectively and for service environments it might be about not understanding which content is helping to deflect calls from the contact centre.

Although hard to prove I believe that even small investments in measurement and analytics can return a significant ROI particularly in the early days of adoption. Perhaps “The price of light is less than the cost of darkness” should be in the footer for every business case for investment funds for measurement and analytics.

I remember the first time I attended an eMetrics conference in 2005 and being overawed by the calibre of the people speaking. It was also the first time I came away from a conference with a souvenir bag full of goodies including what looked like a sextant.

The 2005 line-up included: Avinash Kaushik (Who was he then? Reminds me of a time when Bryan Adams was a supporting act at a Police concert and no one listened. I tell you what, everyone listened to Avinash that day and from then on), Matthew Tod, Neil Mason (Foviance’s own and speaking again this year), Mike Grehan, Dave Chaffey, Jim Sterne himself amongst others. Best practice case studies for that year included Carphone Warehouse, Lastminute.com. Lovefilm, Royal Mail and Direct Wines.

Some of the titles of the sessions included: Getting beyond the basics, Catering to the masses online, Measuring your misery, Getting buy-in from the top down, and finishing up with – You thought this was going to be easy?

It struck me, even with the continuing mainstreaming of social media, how little has changed in many respects. Companies are still caught between paranoia and paralysis. I can measure everything, so I must. I do measure everything, but why? What I do measure, what is it telling me? What do I do with the stuff I am not measuring?

We are daily reminded that everything can be measured, so why is it still so difficult? People haven’t significantly changed between 2005 and 2010. We still buy and we still complain, and we will continue to do so. Social media and smartphones have simply made doing this a little bit easier, a bit more accessible and very much more public.

As a customer I can engage with a company in store, via email, web site, fixed phone, mobile phone, social media, chat, word of mouth… There’s no doubt that we have far more channels than we’ve ever had to communicate or engage with a company.

So where does it all seem to come unstuck?

Surely companies define their objectives, define what success or failure looks like, define what metrics they will use to understand this… the metrics are simply a tool to gauge progress or lack of. It seems to me that the problem with metrics has little to do with metrics and more to do with understanding what you’re trying to actually achieve. The fact that we can supposedly measure everything, isn’t a cause for paranoia, it simply gives us a richer toolkit of metrics to choose from.

A quick glance at the session headings for eMetrics London 2010 shows me that the conversation has moved on, with social media featuring prominently. And if nothing else, I’ll look forward to getting a new souvenir bag, as my one from 2005 finally fell apart shortly after seeing Jim again at the Travelling Geeks event last year.

In this issue we focus on cross-channel customer experience. I relay my own experiences from a recent practical master class with a number of senior marketers, Sean shares his views on information visualisation, Adam points the way ahead for mapping experience, while Guy rises to the challenge of cross-channel measurement.

If you enjoyed this latest newsletter, you might also enjoy reading and commenting on some of our consultants’ thoughts and opinions on our regularly updated blog page.

I would be very interested to hear from you directly with any feedback.

Paul

Paul Blunden, CEO, Foviance

In this issue:

Customer experience master class

Information visulisation

Mapping the experience

Conquering cross-channel customer measurement

First of all, one can wonder how these rankings are built. As it is rarely clearly stated, we can only suppose that the number of downloads of an app governs its position in the table. But again, over what period of time? Number of downloads in the last month, quarter, year? This brings in the process a lot of vagueness and can surely not be used as a success measurement tool. Moreover, as Jakob Nielsen suggests in his column about iPhone App (http://www.useit.com/alertbox/mobile-apps-initial-use.html), users download more app than they actually need and use, which corroborates the idea that the number of downloads is not representative of the usage and can be resorted to as success criterion.

Then comes the question of the popularity of an app. App owners can write comments on the app description pages of the AppStore and rate them on a scale from 1 (negative) to 5 (positive). As a user experience professional, I strive to get the voice of the customer heard by my clients but I don’t consider it to be a viable success measurement method. Personal opinions are very subjective and tend to be only expressed when something goes wrong or incredibly well. Moreover, most of the app ratings derive from the ‘rating prompt’ that pops up on the iPhone when a user decides to delete an app. This biases the results in a negative way.

So how should the success of your app be measured? From a business perspective, the response is simple: the success of an app should be measured just like the success of a website, i.e. by defining and tracking KPI. Most of the apps provide a web-based service, which implies a digital connection between phones and web servers, just like there is a connection between PC and web servers. This allows data to be captured, such as number of information requests, conversion rates, app usage duration, app usage frequency, etc. In terms of web analytics tools, the capture of mobile phone applications usage is only at an embryonic stage but the trend is on the up and some of the current tools on the market are already capable of monitoring app usage.

Boasting about an app being the most used rather than the most downloaded would surely be more credible to end users!

This post was originally posted by Xavier Klingenfus

On the contrary,  it is the belief of both my team and I that we must try, that sees Foviance opening for business in China this quarter, with a new office in Shanghai.

Our plan is a fairly simple one. Foviance is pretty good at international work and as a result we do a lot of it, mostly for western clients that want to test things in Asia. As a result we have gained quite a bit of knowledge and a fair few relationships in the Chinese market. If we could also generate some business in the local market for Chinese companies looking to market to the west, we should have a nice little business plan. Sounds simple doesn’t it? I thought so and yet the planning has taken nearly 18 months and we are only just ready now.

Initially we are focusing our efforts on core user experience services in web and mobile, which is where we believe the largest initial opportunity is. Usability is a growing discipline in China with a large and expanding UPA membership and an ever-increasing acceptance of the benefits. If everything I have read about the Chinese market is true, I believe that we will soon be providing our measurement strategy, customer insight and strategic customer experience capabilities. Initially though, we are staying focused to ensure we keep a handle on the quality of delivery and the market differences.

We know there will be plenty to learn. We have experienced the differences first hand testing various web and mobile services in China for western clients. There are differences in the way people use devices and the web of course but most startling are the cultural variations. It is little wonder that the field of usability is dominated by behavioural psychologists given the need to match interaction design with user behaviour.

International culture differences will add a completely new dimension to the work we conduct developing personas to help organisations design with their audience in mind. Some years ago we carried out work for a gaming company that was looking to launch a Mandarin website. One of the most surprising findings was that there was no translation for ‘lucky dip’. The term simply doesn’t exist and is a perfect illustration of the challenges western organisations face going to China. The reverse is true of course for Chinese businesses wishing to market their services in the west.

In the Star Wars film ‘The Empire Strike Back’, the character Yoda tells Luke Skywalker after he has tried and failed to move an impossibly heavy object (his space ship), “Try not. Do, or do not. There is no try”. I think his point was that if you approach something in a half-hearted way you stand a fairly good chance of failing. At Foviance we trust that our planning and steady execution will enable us to be successful in China but we are not naïve. It will be hard and we will have some success and some failure, but we will do it, because we believe it is important to the future of Foviance.

This article was written as part of our February newsletter

It’s been a busy couple of weeks on the conference circuit. WebTrends held their Engage conference in London and last week I was in Washington DC for the Emetrics Marking Optimization Summit . It was good to see WebTrends out and about in the market, showing off some of their latest stuff. They’ve been working hard on their messaging by the looks of it and getting a distinctive market positioning going. The core brand themes they talked about were “Power”, “Elegance” and “Openness”. By “Elegance” they were referring to the usability of the solution for the end user and it was interesting for someone who works at a user experience consultancy that they have built an in-house user experience team on ensuring that they have the right focus on making the data as accessible and digestible as possible. Information visualisation is definitely on the agenda at the moment. By “Openness” they are talking about the ability to get data in and out of the application via APIs. As someone who thinks that web data should be “liberated” and not locked into reporting tools, this is an approach I approve of.

Emetrics was the usual three days of input, stimulation and networking. Jim Sterne kicked off proceedings by challenging us to focus on turning web intelligence into business value and there were some great examples of how’s that being done in some of the presentations I went to. One of them was from Joe Megibow, VP, Global Analytics and Optimisation at Expedia. Megibow took us through some of the initiatives and changes that had been happening in analytics at Expedia over the past 6 months following a global re-organisation of the business. Some of the work they are doing to tackle marketing attribution problems was interesting, but what was more interesting to me were some of his thoughts about how to get analytics and optimisation initiatives higher up the food chain at organisations. Some of his advice included the need “to construct meaningful narratives”, i.e. the need to add insight and interpretation rather than just hand over the numbers. That resonated with me as I’m always asking my analysts to “tell a story” rather than just present a bunch of charts. Other useful bits of advice from Megibow were to “do less but accomplish more” and to “start small and communicate”. It’s clear to me that he’s focused on delivering business value and he said in response to a question that at the moment he wouldn’t grow his team because the business wouldn’t be able to necessarily get more things done from the insights they generated. Therefore the return on investment from the additional hires would be zero. At the end of the day, the return on investment from analytics is dependent on an organisation’s ability to execute on the findings.

A continuing impression I got from the conference was that measurement of some of the newer digital channels is still hard to do. When it comes to the measurement of mobile, audio, video, social media and so on, people are figuring out as they go along. So it was good to see some of the various approaches that companies were taking to the measurement of these various channels. What struck me also was that there are an increasing number of possible tools that can be used, particular in the social media monitoring and measurement space. But whatever, the technology the key thing is to have a process and that came through loud and clear in a couple of the sessions that I went to on social media measurement. A bit like the dotcom days 10 years ago, it’s tempting for companies to jump on the latest bandwagon without being clear about what their objectives are and what they want to get out of it. So planning is key and the measurement approach follows from that.

Sterne wrapped up the conference with a quick digest of the things that he had learned from the previous US conference back in May in San Jose. He went through the highlights of the “Analysis Symposium” that lead to his report on “101 things that you should know about analysis” . He summarised it as follows: “When doing analysis you should make it:

• About people
• Actionable
• Not about you
• Prioritised
• Intriguing
• Pre-determined
• Compelling (rather than precise)
• Palatable
• Relevant (to the individual)
• A treasure hunt

And finally you should make it happen. I don’t think you can say fairer that that.