Usability Testing

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

As I write this, the Emetrics Marketing Optimisation Summit in Washington DC is drawing to a close and I am trying to process all the inputs and turn them into some outputs of the core themes and takeaways. Tough job, there’s been a lot of stuff to process.

One of the key messages that I picked up from a number of the sessions I attended was that people had moved on talking about the “what” to the “how”. So the talks focussed less on “What is web analytics?”, “What is optimisation?”, “What is multi-variate testing?” and more on “How do I embed web analytics into the organisation?”, “How do I create a testing culture?, “How do we move towards a performance driven organisation”. It’s like we know what the tools in the toolbox are now; we just need to figure out how to use them better and to get other people in the organisation on board. There were still some presentations that covered the “what” type questions but they seemed to mostly revolve around the newer technologies and the emerging measurement practices such as social media and mobile analytics.

A parallel theme that came through was the sense that some organisations or people within these organisations were hitting a glass ceiling. They had deployed the tools, generating the data, created the reports but were struggling to take it to the next level. They could see the opportunity but were not able to make the break through. Bill Gasman from Gartner outlined a number of requirements to move an organisation’s analytics capability forward and the first one of these was to have senior “C” level sponsorship. I’ve just finished reading Tom Davenport’s book “Competing on Analytics” and time and time again he also makes the point that companies that successfully deploy an enterprise wide approach to analytics usually have someone at the top making it happen. The question then becomes how you go about getting that support? I described one approach that worked for me in my last article. (www.clickz.com/showPage.html?page=3631145) and that seemed to be a view endorsed by Bill Gassman when asked the same question by someone in the audience. His answer was to start small and build momentum. It was interesting to observe that some of the issues we are encountering in Europe are not all that different to some of the issues being raised here in the US.

One of the highlights for me was watching Avinash Kaushik unveil the latest enhancements to Google Analytics. You sensed it was what the crowd had been waiting for. It’s not often you see a vendor being applauded for announcing feature releases. Of the various developments announced there were two that caught my attention. The first one is the new advanced segmentation feature. I’m a big fan of the ability to filter and segment your data and so any developments in this area are welcome. Providing a segmentation capability in a tool like Google Analytics will encourage property owners to look beyond the topline numbers and to start to think about their site in terms of different groups of visitors behaving in different ways. So hopefully people will start to look beyond teh bland averages of topline reports and start to drill down into their data.

The other feature that caught my eye was the announcement of a Google Analytics API to allow access to the underlying data. Details are still a bit sketchy at the moment but for me one of the features of a more enterprise level tool is the data integration capabilities. There are many hacks out there for getting data out of Google Analytics and hopefully the API will make this easier in the future. This seems to be recognition by Google that web analytics data can’t operate in a silo if it is to survive.

Finally the presentation of the conference for me was by Jason Carmel from ZAAZ. Jason’s presentation went by the title of “Effectively using kittens for optimisation and usability” (Go figure!) and in it he looked at how site optimisation using tools such as Optimost and Google Website Optimiser is complementary to user-centric design processes and usability based optimisation. He outlined the process by which the two can work together in site optimisation projects with the site optimisation tools basically telling you what’s working and the usability analysis showing you why it’s working and how to use usability experts to improve the quality of the site optimisation tests. It reinforced to me that you always will need more than one tool, in the toolbox to get the job done properly.

When it comes to deciding how many users to recruit for user testing, nobody seems to agree on an ideal sample size. Perhaps more precisely, nobody actually seems to know. This is probably because user testing straddles two seemingly antagonistic domains: business and science.Whenever a client asks me how many people I think will be needed for a particular project, the first thing that comes to my mind is the dreaded: “It depends”. In practice, I generally opt for the rather less elusive response: “Let’s talk a bit more about the tasks before we decide.” In truth, when it comes to sample size, “it depends” is probably more accurate!

First and foremost, sample size is dependent on the type of study. There are a few voices both in the usability and the academic worlds preaching about the ideal number of users in a standard usability evaluation. It is generally agreed that we get value for money with five to eight participants, because on average somewhere between 80 percent and 85 percent of problems are identified using those sample sizes. To unravel closer to 100 percent of problems we would need perhaps twenty people. The maths is simple: why spend 150 percent more on recruitment to get 15 to 20 percent more in terms of results?

There is, of course, some variation in these numbers, but generally, when it comes to standard evaluations, it is fairly easy to decide on sample size. Rich data is extracted, behaviour is observed and interpretation of the results relies on known best practice and experience. Things get a bit more complicated, however, if we use quantitative measures of behaviour, such as eye tracking or quantitative survey data. As someone with a lot of experience in eye tracking, I often get asked how many participants to recruit for such projects, and invariably, people are once more in danger of hearing the dreaded: “It depends”.

Clearly, sample size is related to the complexity of interfaces and tasks. The more complex, the more people we need to test as data variability increases. But crucially, sample size depends on the behaviour the test is set to measure. This, in turn, depends on what the objectives of the study are. For example, to know whether an advert is going to be noticed when users perform their usual tasks on a page, 20 people might be required. However, to know how long on average it takes people to look at the ad, more people are needed because of the huge variations between participants.

With surveys, sample size estimation is also somewhat less straightforward than with standard usability evaluations. Here, the information being collected is attitudinal data, which by its sheer nature can be slightly fuzzy. It all comes down to the size of the effect you intend to detect. Imagine you wanted to know whether people in London are taller than people in New York. If people in London and people in New York are actually pretty much the same height, you will need to measure a high number of citizens of both cities. If, on the other hand, people in London were particularly tall and people in New York were shorter than average, this will be obvious after measuring just a handful of people.

What sample size does not depend upon, is the size of the original population. Whether we are testing people that belong to the whole population of Europe or teenage boys that only wear Ecko clothes and speak with a South London accent, the factors weighed to estimate sample size should be: interface and task complexity, sensitivity of measure and effect size, and the variability between the users.

Of course, in any case, the more the merrier, but this is only possible in a world where resources, such as time and money, are infinite. In the real world, we compromise, and the trick is in being able to achieve a good balance between rigour and value.

The goal for forward-thinking retailers is to re-organise their entire businesses around the customer. The tougher challenge is how to make this change whilst continuing to grow, servicing customers and gauging the extent and pace of change necessary to meet employee and customer expectations without damaging the brand.

Multi-channel retailing provides an answer to this dilemma. If implemented effectively it can prove a catalyst for change across your business, moving away from product-centric operations to a more customer-centric organisation, thereby developing a model that meets the expectations of future customers whilst delivering profitability and growth. The opportunity lies in understanding how to win and retain those customers of the future demanding to buy what they want, when they want and where they want.

To achieve this, multi-channel retailers must first understand who their customers really are and then segment them into groups which share common characteristics. Only then can retailers understand the lifestyle touch points, wants, needs and influencing factors that their brand can be associated with. In order to achieve this, insight ‘personas’ are a must.

- What are personas?
Personas are fictional, but realistic, character sketches based on a break-down of the target audience into groups of users that exhibit common characteristics. They are created using knowledge drawn from customer experience research (qualitative), web analytics and survey data (quantitative), as well as interviewing influential stakeholders of the website.

- How are personas used?
Personas are used throughout a site review or re-design project. They help build a shared understanding of who will be affected by any site re-design, the relative importance of different groups to the site, and how different groups are motivated. In detailed design phases, personas can inform design decisions by allowing multi-channel retailers to explore a range of ideas, hypothesise the potential consequences on target audiences and select the best options.

I have worked with the largest and most successful retailers in the UK and globally over the past three years helping them achieve an enhanced online customer experience, and yet I often still take time to ask the question: “Who are your customers?” My clients know the answer to this based on research they have conducted and their perception of who they wish to target, however there is often a deeper requirement to understand more about the characteristics and buying behaviours of these customers.

How exactly does a retailer deliver the right online content for the core persona while satisfying the needs of secondary personas? The internet is the most measurable medium available and most organisations have rich data available to them in the form of web analytics, pre and post purchase online surveys and so on. This is a good starting point but further research must be done to truly develop an accurate persona, such as ethnographic research (this involves observing a customer or acquisition target in their natural environment), focus groups, usability testing and in a multi-channel environment, interviewing customers in-store.

Pulling all this data together is of course a skill in itself, as well as a core competence of Foviance.

I recently returned from a business trip with colleagues to attend and present at the Usability Professionals’ Association (UPA) International Conference 2008, held in Baltimore, Maryland during the third week of June.

The theme of this year’s conference was ‘Usability Through Holistic Practice‘. The UPA believes it’s important for people to understand just how broad a spectrum of professionals touch the user experience of a product or service and bring their own valuable perspective to the process, including marketing specialists, graphic designers, computer scientists, business analysts, psychologists, information architects, technical writers and others.

My colleague Catriona Campbell and I were fortunate enough to present an all-day tutorial around the subject, sharing our own approach to holistic usability and managing user experience.

Attending, and especially presenting at UPA08, gave me a fresh perspective that can be hard to gain when working nose-to-grindstone in the office. In the UK we’re sometimes guilty of thinking that we are somehow behind the curve compared to our American counterparts, when actually we are ahead in many ways, such as the manner in which we regard, measure and design customer experience. It’s not that innovation doesn’t exist in the US, it definitely does, but I believe our general attitude is more adoptive across the board.

Our workshop at the UPA followed the core principles of the Foviance approach to customer experience – combining qualitative and quantitative measurements, to fully understand the true user experience and what is to be done to enhance it.

We presented a broad array of elements and tools that could be employed to manage user experience, from more straightforward devices such as web analytics or satisfaction surveys, to more cutting edge approaches such as eye tracking and EEG stimulus. We also discussed emerging techniques such as benchmarking experience across competitor sites and products using remote usability testing with quantitative numbers of participants and appropriate tools, such as WebEffective from Keynote Systems. Once identified as the right elements or tools needed to achieve a particular project’s aims, any or all of these then needs to be combined in order to deliver a holistic view of customer experience.

Some of the attendees, who ranged from independent usability people to owners of customer experience at large corporations, admitted past experience of having the right tools but encountering problems bringing them all together. As the conference progressed, people searched us out to let us know they had been able to grasp this bigger picture.

For those that attended our workshop, the conference clearly demonstrated that we have the tools and technologies to manage and measure the whole user experience from end to end. Not only that, but we are able to prove categorically the business case for investment in user and customer experience – something that is going to be increasingly important in the ‘credit crunch’.

By Marty Carroll

In any fast-moving industry, it’s usually possible to get a glimpse of the future by studying the past. The foundations of the modern usability and customer experience industry were built from the principles of HCI, or human-computer interaction, an area of study which kick-started ongoing investigation into the interface between technology and people across a broad variety of platforms and channels.

When the HCI industry gathered momentum in the 1980s, it was primarily concerned with making desktop software easier to use. By the time the mass-market leapt online in the 1990s and branches of HCI had begun evolving into usability, its proponents were employing ‘discount usability’ methods that made some headway to improving usability but were not allowed to slow down development timescales.

During the 1990s web design methods and agency productivity centered on technology itself rather than the eventual consumers of the information, and this attitude dictated the design process. By 2001, a more user-centric approach had begun gaining currency within the web design community and we were ahead of the pack in recognising the value of usability testing. We believed it was important to conduct usability studies in the early stages of web design, helping to create sites that were more in keeping with the needs of end users. We understood the importance of the bottom-line needs of businesses, but argued successfully that aligning design with what was best for its users would ultimately be good for business too.

Back to the present day: I believe it is time that similar user-centric design techniques used so successfully for the web are now applied to a wide range of other services across multiple channels. The experiences customers have with different channels are summed to make up the customer’s impression of the whole brand, and forward thinking companies need to be thinking about customer experience holistically. It doesn’t matter that the web experience is great if the call centre keeps people on hold forever, for example. Similarly, it doesn’t matter that goods are shipped immediately if they’re poorly packaged and arrive smashed into pieces.

We are applying the same rigour with which we approach webdesign across all other customer ‘touch points’, to establish where there are breakdowns. We have found that when we take this user-centric design approach, our clients improve their customer experience, boost retention and satisfaction
of calls, increase customer ‘lifetimes’, and the whole business is impacted positively and profitably, just as it was with the earlier days of user-centric web usability.

Some people say that unlike time and turnover, customer experience is intangible and impossible to measure. We disagree. With the right metrics, and the processes to record them, you can measure the quality of the customer experience. The Apple Store, which offers customer support surgeries and training, shows how a good customer experience can be an important part of the brand. While there are many companies that see customer experience as a cost, those that thrive will be the ones who choose to see it as a worthwhile investment.

The usability industry has come a very long way during the eight years we have been in business.

Back in 2000 site owners rarely allocated budgets for any kind of usability input into site design. Unsure of the benefits, but intuitively suspecting it could add value, they used all manner of subterfuge and creative accounting in order to run usability tests in an ad hoc fashion. As a result, usability was deemed the very antithesis of creativity and its practitioners lacked any real appreciation of the design process. Read more…

It has recently become common practice to bundle eye tracking technology as standard with usability testing. Not only is this somewhat surprising, it is also very disappointing, because it sends out the misleading message that eye tracking is simply an attractive accessory to usability testing, in the same way that a pair of earrings might complement a necklace.

At a time when usability testing has practically become a commodity, usability companies should be striving to provide clients with customised solutions that answer questions directly and resolve specific problems. Such an approach requires a level of sophistication that hinges on choosing the right set of tools and applying them appropriately, rather than using multiple tools for the sake of the aesthetic impact of the deliverable. Read more…

In the 90s, the webdesign team comprised a designer, responsible for the look and feel, and a developer, who did the coding behind the scenes. Who was thinking of the end user? Theoretically both of them were, but it was nobody’s job. Designers often created wacky interfaces to express their personalities and (somewhat optimistically) expected users to persevere and learn how they worked. The coders tended to have a technical background, and sometimes considered a user interface that is bug-free to be working and users who couldn’t operate it to be stupid. When the team did consider what users needed, it was based on intuition rather than research.

The creative team of the future will also include somebody who takes responsibility for the user experience. The ethnographer is there to empathise with users and ensure that the finished website meets their needs easily. The best way to do that is to watch users in their natural surroundings. It’s like watching wildlife. If you study pandas at the zoo, you’re observing the zoo as much as the bears.

We recently undertook a project for Royal Mail, which wanted to optimise its online postage services for small businesses. Rather than inviting people in to talk to us or sit in the lab, we visited small businesses and shadowed them as they processed their post. We could ask them questions as they were working, and find out what inspired them to use particular carriers. Our subjects also kept diaries which we used to quiz them later.

Usability testing is an ideal tool when subjects can easily articulate their needs, or when an interface is already available for testing. But ethnographic research is essential to find out how people use a service, including those parts that are not online, or to learn more about the users’ needs when they are hard to generalise and explain.

While visionary companies employ ethnographers and usability specialists today, in future, they will be a part of every successful design team.