Usability Testing

Since the advent of the mobile phone we have seen them move from being a large device used by a very select few to their current ubiquity, with some people owning two or more phones. How did we manage before we had them? However we did it is unimportant; the mobile phone is here to stay. This mobile evolution happened fairly rapidly and over a short space of time, which is due in no small part to the efforts of Nokia, who conquered the market in the late 90s and has seen off growing competition from other mobile manufacturers in the last couple of years. The legend of Nokia lives on and it has managed to keep hold of a sizeable share of the market, although the credit crunch seems to be loosening its grip, albeit marginally – Nokia’s Q4 market share figures for 2008 are down 3% from the previous year’s Q4 share of 40%. That said they are still leagues ahead of their nearest rival. I think the key to their success story can be linked to their heavy investment in user testing and focus on easy to use products. Nokia users are known to be steadfastly loyal and shriek at the thought of using another type of handset.

We face a number of challenges for designing usable interfaces on these shrinking devices. For one we need to contend with the increase in the number of applications and games that are freely available, mobiles are being used in many new and varying ways. Addressing how to make this interaction easy and engaging while on the move is a formidable task. The change from testing in labs to testing in the field will be an important one. Testing in lab conditions will highlight a high proportion of the issues but many more will be missed. Things like mid-day sun can be simulated to a certain extent but not walking and texting at the same time or looking after a child and trying to find someone’s contact details simultaneously. By using small cameras which can be attached to the handset which records participants’ interaction is one way. Another is to make use of embedded software which records users’ interactions, relaying the keystrokes and menu selections back to usability specialists to be analysed. However it is done, mobile testing is going alfresco. Far from being left out in the cold mobile manufacturers should take to the streets.

Since 2001, our organisation has evolved rapidly and continually in order to stay one step ahead of technology and consumer trends – we have evolved over this short span of time from focusing on usability, to the overall customer experience. Over decades the discipline has transformed from a fundamental attempt to understand how humans might interact better with mechanics, to a technologically advanced collection of processes and services developed to improve all facets of the customer experience.

Usability, as we know it, first emerged in the UK as a result of the intensive use of technology during the Second World War. People realised that adapting machines to meet the needs of humans, improved human-machine reactions, speed and performance. The science expanded into telecommunications, then computers, and the rest is quite literally history. All the digital technology we now take for granted was developed and refined as a result of pioneering usability testing.

Of course the analytical consultancy, research services, customer insight and experience management expertise we draw upon today didn’t leap directly from wartime mechanics. Studies of human-computer interaction (HCI) and other behavioural sciences led to a major focus on user-centered design in the 1980s and 90s. Consultants began to realise the importance of encouraging end users of products to participate in the design process, rather than simply being shown a range of finished designs for comment. A discipline with its roots in the military and academia had to realise that revenue-driven businesses rarely had the time for blank canvasses, blue-sky thinking and constant design revisions – they needed user centered design to drive mutually pleasing results as quickly as possible.

Customer experience consultancy as we know it today evolved directly for this realisation. Bringing the story right up to date, even the very latest ISO standard for human-centric design is now demanding a far greater emphasis on human involvement.

The revised version of the human-centered design standard ISO 13407, renamed ISO 9241-210, is now out for public comment until April. Its aim is to ensure interactive systems have usability at the heart of their design. The critical change to the standard is that the four key human-centered design activities are now requirements, not merely recommendations. As pointed out on Econsultancy.com, anybody claiming that a product or service adheres to the standard must now:

• Understand and specify the context of use (including users, tasks, environments)
• Specify the user requirements in sufficient detail to drive the design
• Produce design solutions which meet these requirements
• Conduct user-centered evaluations of these design solutions and modify the design taking account of the results

Essentially then, the new standard means organisations and ultimately their developers must have humans heavily involved in design evaluation to get the tick. It also emphasises the role of iteration in design and that humans should be consulted throughout the design process, not simply in testing results. We have to consider not just how we test, but when we test, and who we test with.

The practice of evaluating and refining usability has evolved considerably since we first made the transition away from surmounting mechanical obstacles and functionality issues and began instead to look more closely at the user, or customer. A more acute emphasis on the people who actually use the products and services we test, has led to us taking a more holistic view of interactions, interfaces and the entire experience.

That is why the many elements of a modern customer experience consultancy add up to so much more than just usability testing. The reason we employ innovative research techniques such as eye tracking or Electroencephalography (EEG) alongside tried and tested methodologies such as card sorting or ethnography, is because ultimately we want to provide the most valuable insights possible for the companies that ask us to help to make their customers happy.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

I have just been reading a booked called “Super Crunchers” by Ian Ayres. It’s an interesting book on how the use of data mining and predictive analytics is becoming more widespread across all aspects of our societies, and is increasingly shaping our lives. He cites a number of different examples where these empirical approaches are able to outperform human experts in their ability to accurately predict the likely outcomes.

I particularly liked his story of an econometrician who was able to predict the expected quality of Bordeaux wine based on a simple regression analysis of weather data. He was able to predict the expected quality of a particular vintage based on just three variables; the amount of rainfall in the winter, the amount of rainfall during the harvest and the average temperature during the growing season. What was interesting for me was not the fact that he was able to make these predictions, but the accounts of the resistance and even hostility that he got from the “wine establishment” for his predictions. The wine experts of the time were very threatened and affronted by the fact that their “art” and “expertise” could be reduced to a simple equation.

Ayers goes on to give a number of other examples in various industries where the growth of data and technology has allowed data mining and predictive analytical techniques to change the rules of the game, from baseball scouting to social policy development and medicine. Quite often in each of these fields there has been resistance to the ascendency to the use of these techniques from the established experts in that field, such as baseball scouts, policy makers, doctors and so on. They would not, or could not accept that such empirical methods could be better than the expertise they had developed over years of training and experience. However numerous studies cited by Ayers have shown that predictive analytics outperforms “experts” in the ability to predict an outcome correctly. That doesn’t mean that predictive techniques always get it right just that they get it right more often than the experts.

In the digital marketing field Ayers uses the example of A/B and Multi-Variate Testing (MVT). The point he makes is that the volume of data and the technology, now allows people to run repeated tests and trials to predict which versions of which element on a page is most likely to be successful in driving the desired outcome. Those of you familiar with the MVT technologies will know that the marketing stance behind them is often that they eliminate the need for subjectivity in the design process. You just come up with some alternative versions and see which one works best. It’s the ultimate tool for overcoming bias and subjectivity of the various stakeholders involved in site development. Who needs usability testing, right?

Ayers’ background is not as a statistician or an analyst but as a lawyer. You don’t immediately think of lawyers as being masters of the empirical universe and why would a lawyer be an expert in number crunching? The interesting point being a lawyer could be similar to being an analyst. Often you are trying to prove or disprove a hypothesis and looking for the appropriate evidence to support your theory or disproves somebody else’s and, for me, this gives rise to one of the fallacies about econometrics and predictive analytics that it is purely a scientific discipline.

Predictive analytics is often as much about art as it is about science. To build a good model you need to have a good understanding of the way that the “system” you are trying to model works. More often than not, at the beginning of the model building process, there is some subjective opinion about what are going to be the likely factors influencing the thing that you are trying to predict. So where do these opinions come from? They usually come from the people who are knowledgeable or experts in that particular field. We sometimes called this the “domain expertise”. If we take the example of the econometrician predicting the quality of wine, the econometrician was also a wine buff so he had some previous knowledge about what the likely factors were that could potentially affect the quality of a particular vintage. His skill was in quantifying it.

In the same way, some domain expertise is needed in the development of good tests. If we look at MVT then the technology can help you determine which the best page design to use is. If you test 4 different versions of an element (say a call to action), then you will get a winner. That “winner” may be the one that you started out with, but it’s still the winner. It doesn’t mean though that it’s the best one, it’s just the one that was best out of the various options that you looked at. There may be a much better option out there which you haven’t tested. Usability experts can potentially provide better insights into what versions are the best ones to test in the first place, and also help to understand why the results have come out the way that they have.

So we need the experts to help us build better models. That expertise may come from years of experience or knowledge gained from understanding the effectiveness of previous models. In either case, there’s room for both the science and the art.

London, UK, 2 February 2009 – Foviance, the expert in customer experience, has been selected by East Sussex County Council to test their website and provide usability recommendations. As part of ongoing improvements to online services, this study aims to make their information more accessible to East Sussex residents and other web visitors.Childcare services and help for older people are amongst the key areas of the Council’s website, receiving over 1.5 million visitors a year, which will benefit from the research. Read more…

By Arthur Moan

With so many reports of economic doom and gloom dominating our newspapers and inboxes, it is too easy for us to simply assume that all online retailers are having a negative time of it.

In actual fact, innovative brands that have worked hard to provide their online customers with an excellent cross-channel digital experience have managed to shine through much of the gloom, reporting some excellent figures in the face of widespread market turmoil.

Because my own role within Foviance brings me in close daily contact with a number of leading retail brands, I have been able to get a genuine feel for what has been happening, their ability to see decent returns on their customer experience investment, without having to rely on second hand reports. The recent achievements of some of these brands might surprise you.

The Shop Direct Group, which owns the Littlewoods brand, recently announced a bumper Christmas trade with sales up nine percent on the same period last year, with 60 percent of overall trading coming from online sales. This performance came as no surprise to me as the online team as Shop Direct has been focussed on enhancing the user experience throughout 2008 and have already committed to further usability testing entering into 2009.

Another client of ours, Sainsbury’s, also announced a 4.5 percent increase in sales over the same period last year. The grocery giant’s online sales rose by 27 percent during the quarter, boosted by a record number of orders in the week before Christmas. Like Shop Direct, Sainsbury’s conducted a lot of work with customers last year, looking closely at all feedback and acting to improve the overall digital experience of its brand.

Boden’s efforts to enhance its brand through innovative online retail methods that concentrated heavily on customer empowerment also delivered some excellent Christmas trade figures. Sarah Blair Gould, ecommerce manager for Boden, said: “Regardless of economic conditions, we believe it is absolutely essential that any innovative online brand makes genuine efforts to really get to know their customers, what they desire in a retailing experience, and what makes them feel valued and an important part of the brand’s core values. When times are tougher, it’s vital not to cut back on the extra frills that make customers feel that extra bit special. If anything, the more indulgent areas of an online retail business should be emphasized in a downturn in order to really underline the customer centric nature of the brand and retain loyalty.”

Of course I’m not claiming that all retailers delivering a good online experience managed to secure a bumper year end trade, despite the positive tone of many of the news items currently reported at IMRG, ‘the leading industry body for global e-retailing’. But it is true that many industry observers, including leading analyst firms such as Forrester, are now urging customer experience professionals to maintain their focus and to make the business case for new customer experience projects ahead the much sought after first green shoots of economic recovery.

Signals like these from some of our leading brands do serve to remind us that even in tough economic times, brands with a strong proposition and a careful consideration of the overall customer experience can perform well if they select online as their channel of choice and optimise that retail channel to the maximum benefit of their customers.

The deepening gloom surrounding the global credit crisis is providing a distraction for many people in business from carrying out a review of what their business has achieved in the past 12 months and how the market has changed. Foviance is no different, although we are fortunate in that we carried out a complete review of our business some 18 months ago as we felt it could be run better. This has enabled us to be slightly more outward looking than we might have been had we not done this and made the changes that resulted. So I am able to take a look at the last 12 months and consider what we achieved and how the market has evolved.Our year began with a face lift for our reception area and offices. Our usability labs have never been as busy as they have this year and we have had more than 1,000 participants through Foviance doors this year for various client research. With many of our customers spending long days with us viewing the research, we felt the place needed a new coat of paint and a more professional feel – despite being only two years old! Further improvements are planned for next year to the lab technology and it seems credit crunch or not, testing key user journeys remains a critical success factor for many businesses.

The other area of our business that we have been working hard to develop and really took off in 2008 is product testing. Since we started the business in 2001, we have carried out product research but it has come to us rather than us seek it out. We took Microsoft’s tablet PC all over the world in 2001 and since have tested everything from printer control panels to laptops and beyond. It seems to me the change we have seen is driven by two factors.

The first is the global market place that has developed, and many brands are now routinely leveraging our Asian, US and European connections to test their products in multiple markets. The second is the convergence in mobile and small portable devices between software and hardware from a user experience standpoint.

It seems to me that this year the willingness to adapt the hardware in order to maximise the user experience in the software has increased substantially. It is easy to credit the Apple iphone with this evolution but I believe it is the general increase in comfort and understanding of technology that has facilitated this. In my opinion, more is to come.

Looking back on 2008, I think businesses were just starting to ‘get’ cross-channel customer experience.
I use the past tense because I think for many, the credit crisis will (if it has not already) send them backwards. This is a mistake. I have met many businesses this year that had started to think about the customer experience they want to deliver across all their channels in the same way they think about the brand holistically. A planned and well executed customer experience means that when a customer goes online, visits the store or calls the call centre, they receive a consistently good experience. If they do they are more likely to convert, remain loyal and become advocates. The trouble is, being organised is more expensive upfront which is why I think we will see the progress made this year eroded.

We ended the year with the acquisition of Applied Insights which I was, and remain absolutely delighted about. “When the going gets tough, the tough get measuring” I am told and that certainly seems to be the case. Since Neil Mason joined us we have hardly seen him or the team, such is the demand from organisations that want to understand what is going on in their businesses.

So, not a bad year, we achieved many of our goals and I feel we are well equipped to cope with whatever the market throws at us next year. There is uncertainty from our clients about their businesses and markets but I think that can only help us, and our industry deliver more value and create new and exciting ways to do so. The year ahead will be tough for a lot of businesses, but I am optimistic that customer experience research will continue to play a vital role in the success of businesses in every sector.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

It’s been a busy week since I started a new job at Foviance. The reaction from people I know in the industry about Foviance and Applied Insights coming together has been very positive. People tell me that it makes sense, which is good news because it certainly made sense to me.

The reason why it made sense to me is that I passionately believe that analysis and insight lies at the heart of improving the online user experience. Working with one of the leading user experience consultancies in the UK as the person responsible for driving their analytical services capabilities forward is going to present some great opportunities to grow and to learn. It’s never too late to stop learning!

For me marketing has always been a blend of art and science and in the digital marketing space there is more science available to those who want to take advantage of it. For many organisations though it has taken some time for them to adopt the ability to improve the online customer experience through measurement and analysis. At times there is often a tension between the “creatives” and the “analysts”, whereas the reality is that both is needed and both need to be blended.

I think that the position where many organisations have got to know is that they have “adopted” the measurement and analysis side of things. They have plumbed in a web analytics system. They may be regularly measuring customer satisfaction. They may be routinely doing testing. They now have access to the science. However, what they haven’t managed to do is to integrate the science into the way that they do business.

Often decisions are made on the basis of judgement even when the data is available to them. But the trouble is, as someone once said, “Good judgement comes from experience. Experience comes from bad judgement”. Or as an old boss told me after I had made some cock-up: “Neil, all experiences are learning experiences. It’s just some are more pleasant than others”. One of the key roles of data, analytics and insights is to help us avoid having too many “unpleasant learning experiences”.

So the opportunity going forward is to blend the art and the science in a seamless approach to improve the user experience. Creative designers working alongside analysts to understand the impact of their design changes in a collaborative fashion.
Quantitative analysts such as web analysts working alongside qualitative researchers such as usability consultants to understand the user experience form all the angles. Not just looking at what users did but also understanding why they did it and what they felt about the outcome. This kind of integrated approach will need integrated thinking based on integrated data. Integrated thinking will come from the recognition from all the players that they only have a part of the solution and their instinct should be to go and seek out the other parts.

The difference between adoption and integration will come down to organisational culture and processes. This is a theme that I keep returning to in my consulting activities as well as in this column as I think this is one of the biggest challenges in the industry at the moment. People still worry too much about the technologies rather than worry about what they are going to do with the technologies. Organisations and their agencies will need to start thinking about how to build the science into the creative process in a systematic way and how to view the creative process as an iterative, cyclical process rather than just a linear process. The physical manifestation of this vision might be a roomful of designers, information architects, analysts, usability experts and brand marketers coming together to throw ideas around about what the user experience should look like. Each contributing their perspective and each contributing to the final outcome. Not on an ad-hoc basis for big projects, but on a regular basis constantly iterating the solution week after week. It might take a while to get there but I’m looking forward to the journey.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

As I write this, the Emetrics Marketing Optimisation Summit in Washington DC is drawing to a close and I am trying to process all the inputs and turn them into some outputs of the core themes and takeaways. Tough job, there’s been a lot of stuff to process.

One of the key messages that I picked up from a number of the sessions I attended was that people had moved on talking about the “what” to the “how”. So the talks focussed less on “What is web analytics?”, “What is optimisation?”, “What is multi-variate testing?” and more on “How do I embed web analytics into the organisation?”, “How do I create a testing culture?, “How do we move towards a performance driven organisation”. It’s like we know what the tools in the toolbox are now; we just need to figure out how to use them better and to get other people in the organisation on board. There were still some presentations that covered the “what” type questions but they seemed to mostly revolve around the newer technologies and the emerging measurement practices such as social media and mobile analytics.

A parallel theme that came through was the sense that some organisations or people within these organisations were hitting a glass ceiling. They had deployed the tools, generating the data, created the reports but were struggling to take it to the next level. They could see the opportunity but were not able to make the break through. Bill Gasman from Gartner outlined a number of requirements to move an organisation’s analytics capability forward and the first one of these was to have senior “C” level sponsorship. I’ve just finished reading Tom Davenport’s book “Competing on Analytics” and time and time again he also makes the point that companies that successfully deploy an enterprise wide approach to analytics usually have someone at the top making it happen. The question then becomes how you go about getting that support? I described one approach that worked for me in my last article. (www.clickz.com/showPage.html?page=3631145) and that seemed to be a view endorsed by Bill Gassman when asked the same question by someone in the audience. His answer was to start small and build momentum. It was interesting to observe that some of the issues we are encountering in Europe are not all that different to some of the issues being raised here in the US.

One of the highlights for me was watching Avinash Kaushik unveil the latest enhancements to Google Analytics. You sensed it was what the crowd had been waiting for. It’s not often you see a vendor being applauded for announcing feature releases. Of the various developments announced there were two that caught my attention. The first one is the new advanced segmentation feature. I’m a big fan of the ability to filter and segment your data and so any developments in this area are welcome. Providing a segmentation capability in a tool like Google Analytics will encourage property owners to look beyond the topline numbers and to start to think about their site in terms of different groups of visitors behaving in different ways. So hopefully people will start to look beyond teh bland averages of topline reports and start to drill down into their data.

The other feature that caught my eye was the announcement of a Google Analytics API to allow access to the underlying data. Details are still a bit sketchy at the moment but for me one of the features of a more enterprise level tool is the data integration capabilities. There are many hacks out there for getting data out of Google Analytics and hopefully the API will make this easier in the future. This seems to be recognition by Google that web analytics data can’t operate in a silo if it is to survive.

Finally the presentation of the conference for me was by Jason Carmel from ZAAZ. Jason’s presentation went by the title of “Effectively using kittens for optimisation and usability” (Go figure!) and in it he looked at how site optimisation using tools such as Optimost and Google Website Optimiser is complementary to user-centric design processes and usability based optimisation. He outlined the process by which the two can work together in site optimisation projects with the site optimisation tools basically telling you what’s working and the usability analysis showing you why it’s working and how to use usability experts to improve the quality of the site optimisation tests. It reinforced to me that you always will need more than one tool, in the toolbox to get the job done properly.