Usability Testing

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission. 

The past few weeks I have been looking at the need for organisations to focus more heavily on the user experience and to more rigouroulsy understand what their customers want online and how best to deliver that to them. Good customer insight is core to that process and insight comes from a range of systems, methodologies and techniques. Last time  I looked at the use of quantitative approaches to customer insight and this week I want to look at some of the more qualitative approaches. Read more…

We’ve all been there. Sitting round a conference room table discussing with our colleagues about the design of the website, the flow of a particular user path or the layout of a particular page. Opinions differ on what would work best, whether the call to action button should be red or blue, square or round, flat or bevelled. We all know best, because we’re experts. Aren’t we? In some cases it may not matter how expert we are, because the loudest voice will win or the most important person’s opinion will be the one that counts. Read more…

By Lucy Carruthers

Chris Lake of Econsultancy recently published a blog called 50 swinish ways to annoy web users. It’s an amusing read, but what struck me is that the first 7 items related to forms of advertising on the web (Auto-play, interstitials, death by Adsense, death by banners, pop-ups, etc.)Advertising on web sites is an interesting conundrum. The participants in our studies are often quite vocal in their annoyance of them, yet online advertising generates big money for companies so we must be clicking on them. In fact, the IAB have recently released a press release stating that in the US Internet advertising revenues surpassed $23 Billion in 2008, reaching a record high. So like them or loathe them, adverts are a big part of our online world.

Our clients often ask how best to approach advertising on their site. Is brighter and bolder better? Should adverts be contextual and personalised? Should they wait until the end of a customer journey to upsell or is that too late? How can they create a design that provides the best balance between increasing their revenue without infuriating their customers at the risk of damaging their revenue?

I recently tested a high-fidelity black and white prototype for a financial client that contained placeholders for adverts. On pages containing non-personal content, the advertising wasn’t questioned by the participants as it appeared relevant to the main content, and also offered potential savings and discounts to the users. The participants were less keen to see similar material on pages where the main content was personal to them; however, their attitude was that as long as they weren’t distracted from their tasks they weren’t too bothered. But, as soon as the same participants saw the same pages but with the graphic design overlaid their attitude immediately changed; they were enraged to see ‘advertising’ amongst their personal account information despite the savings promoted. The graphic design had made the adverts far more prominent and as far as the participants were concerned they were intrusive and out of place.

So what can organisations do?

Firstly, organisations should see how their competitors are doing it and determine what works and what doesn’t. Online banking, for example, is an area where adverts are becoming ubiquitous and I think some banks have got their placement right, while others rudely interrupt the users journey.

Secondly, they should look for innovative new ways to advertise while keeping the users goals and needs in mind. Pixazza, a company in the States, have developed a means of converting photos on web sites into interactive advertisements. James Everingham, who is behind the concept, says that many adverts are an “obnoxious intrusion” and that “ads work best when they don’t resemble ads but rather a feature that helps shoppers find items they may already be interested in buying”.

Lastly, it’s crucial to test any designs with users and to test iteratively. Testing with early prototypes can catch key issues before it gets expensive to make changes, and testing once the final design has been overlaid can ensure that with the graphic design the site still meets the users needs and expectations. The graphic design can have a major impact on both on the usability of the site and the user experience; and satisfaction levels for the overall customer experience.

Although remote testing will never be a complete replacement for traditional user testing, it does have its role in the user experience professional’s tool box – and it’s becoming an increasingly important one.

Traditional one-on-one website usability testing reveals a rich insight into any suitable site evaluated. It uncovers lots of qualitative data while providing a perfect opportunity to watch users’ behaviour, interpret their actions, and engage in conversations that delve into their opinions and difficulties. So when is remote testing appropriate?

There are times when small, qualitative samples are not suitable. Remote testing can instead provide a large volume of quantitative, statistically significant data. It can provide proof of issues on a large scale, or produce the figures needed for a business case. Remote testing and traditional user testing can also be used in tandem. Remote testing can provide statistical data to back up findings from a usability study. Or, perhaps, findings from a remote usability study can be expanded and explored during a one-on-one study.

As more brands become global, it is more important than ever for companies to optimise their international offering. There is no better way to understand the quality of an international presence than to test a company’s website in the international community. Current financial instabilities have heightened this importance, despite making funding of such testing harder to achieve. Remote international testing eliminates a large proportion of these costs – certainly a good deal of travel – while potentially enabling a single team to conduct all tests.

Of course not all types of remote user testing should be viewed in the same way. Certainly a clear distinction must be made between remote moderated (using screen sharing software) and remote unmoderated testing (using in-depth customer experience research tools such as WebEffective), and studies should be designed with this in mind.

Although it is conducted via some form of screen sharing software, remote moderated testing is very similar to traditional user experience testing. A moderator conducts the tasks in a similar way to a traditional user test, and is on hand to answer or ask questions at any time. They are even able to observe certain facial expressions and behavioural nuances via a webcam. Emphasis must be placed on ‘think aloud protocol’ to ensure that nothing is lost from the visual cues that cannot be seen.

Remote unmoderated testing is an entirely different story. Because there is no moderator present, any communications with participants must be really clear. There is no chance to ask questions, and if instructions are misunderstood this could easily lead to inaccurate results, possibly turning the whole study into an expensive mistake. The study should be written with no room for error and participants must be prompted to give full answers. There are pros and cons for this unmoderated form of remote testing. On one hand, participants without moderation will have less social anxiety about providing unhelpful feedback and therefore may be less inclined to write a lot. On the other hand, the lack of a moderator may give the participant more freedom to voice their opinions on more sensitive topics, or where the participant would feel embarrassed if identified.

Understandably, the person running an unmoderated remote study has even less control over their participants’ environment than with a moderated remote study. This means there is less control over distractions, or interruptions from the study which can affect completion times, but over a large enough sample this should even out.

It should be remembered throughout that with any international testing, evaluators must recognise and be sympathetic to differences in participants’ language, social conventions and mental models in order to be culturally sensitive and collect useful results. It is also advisable to understand variations in laws surrounding the treatment of human subjects in different countries. As the need for remote user testing increases, it is essential to understand the need to plan studies that always uses methodologies that best suit the individual project.

Here at Foviance, we are experts in both moderated and unmoderated remote user testing, and will be releasing the results of unmoderated industry studies with our partner Keynote later in the year. We’ll share our analysis with you as soon as it is available.

In addition to international customer experience testing for websites and ecommerce stores, Foviance is also involved with global projects that evaluate the suitability of and reactions to products across multiple regions.

Our global examination of the varying regional customer experiences with netbooks is a great example of what can be achieved, and avoided, with thorough usability testing. Netbooks are compact, portable laptop computers designed primarily to power web browsing, electronic messaging and wireless communications. They are extremely desirable objects, especially when coveted from afar, but as we discovered, the actual user experience of these devices varies wildly depending on the country the consumer comes from.

I was part of team tasked with evaluating four different brands of netbook with end users in the UK, US and Japan. We allowed each group of individuals to gain an initial impression of the devices, by providing them with the level of look and feel insight you might gain in a shop environment. Following this introduction, we then allowed our international test groups to get to grips with the netbooks so that they could evaluate usability through a series of tasks. The idea was to see how likely they were to buy a netbook before and after assessing their practicality. By the end of the testing, a clear picture emerged that on the whole, netbook usability doesn’t live up to initial impressions.

Keyboard entry proved a problem in Japan in particular, where users were completely turned off by the idea of netbooks in just an hour and a half. Of course different cultures had different impressions, depending on their country’s level of design centricity and emphasis on usability. Our Japanese testers enjoyed the modern slick looks but found the devices unusable, whereas our US group thought the devices worked okay but didn’t like the fingerprints left on the shiny surfaces.

This close-up and personal testing is logistically hard to perform. We used our usability partners to ensure we had two testers with each individual, monitoring reactions and engaging with the testers on a personal level. We needed to buy each product locally to ensure language and layout compatibility of keyboards and screens, and we had to rely on the knowledge of local agencies to recruit from the most appropriate customer segments. At all times we tried to make the testing interactive and fun, allowing testers to sit down, pick the things up, use them as they would hope to in their own lives, and role-play. We found the most effective testing embraced a combination of ethnography and lab testing, whereby testers may not have been in their own environment, but were certainly at ease.

As a result, our manufacturer clients were blown away by the results, and were stunned by the dramatic swings in opinion before and after usage – particular with regards our Japanese group. These testers simply couldn’t get used to the keyboard, which made Japanese style data entry – focused largely around the enter key – feel clumsy, slow and impractical. It was a critical issue that our manufacturers hadn’t really considered at all.

Our product testing unlocks problems arising from cultural or regional differences and makes them transparent to vendors and designers. We provide a level of expert local insight in diverse markets that manufacturers simply can’t replicate adequately themselves. With our help, clients can re-evaluate products so that they fully live up to user expectations in different countries around the world and secure the sales returns that the core technology truly deserves. Companies that invest in their customers, inevitably reap the greatest cultural and economic rewards.

This is an abridged version of a conversation between Angus Cormie, head of e-Business EMEA Consumer for Dell, and Foviance Lead Consultant, Clare Mitchell Crow. This is the latest in a series of regular interviews with senior figures from the world’s most respected businesses, focusing on the ways organisations manage their customers’ experience.

Angus, please tell us a little more about your experience and role at Dell?
Sure. I previously worked for Phillips and 02 in a variety of commercial roles, getting into the online space around 11 years ago through the 02 mobile portal venture Genie. I ran the 02 online portal for a couple of years before leaving to gain some practical marketing and retail experience. Since then I have been back in the corporate world, initially with T-Mobile and then with Dell, where I manage its website and the online team for the consumer division within EMEA.

Foviance has been working for Dell for around two years. Could you talk about the work you have done with customer experience?
With Dell and in previous roles I have done a lot of work with usability, largely broken down into two areas. Firstly, testing new concepts and ideas from a web experience perspective, and secondly, ongoing usability activities, setting up regular robust reviews of customer experience and usability analysis. Regular reviews give us enormous levels of feedback that are very different from ad hoc research for new product concepts and so on, but both activities are extremely valuable. With both Foviance and with our in-house teams we tend to look at path analysis and general usability. Ad hoc testing helps us see things we may not be focusing on, but the only way to make customer experience testing part of the culture and ethos of the company is to make testing regular, consistent and measurable. A quarterly programme of activity allows us to ask the same questions over again and track responses and analysis from one quarter to the next with measurement. It’s a big ask, and it is time consuming and expensive, but gives us the biggest return.

Dell recently reorganised its business focus with regards to customer categories. How does this affect your own role?
When I first joined Dell the European organisation was very much a sales organisation. Consumer was the first division to become a global business unit and it’s only now that the other segments are doing the same. Despite this globalisation, Dell has actually managed to give the regions more focus and more engagement with our global colleagues. The online team within Europe has had plenty of opportunity to influence and drive change and to be key players in what actually goes on with the way we manage our business. We’re positive these changes are a force for good. Within consumer we had some concerns about how we would make that transition from a business focus to a consumer path, but we’ve worked to adapt the whole experience of entering our site, selecting products, and customising them, so that it is much easier for consumers. We drove change that brought more consumer focus around language, tone of voice, and specifically the online path and navigational elements, and these changes were successful and adopted globally. The next stage will be to make it more end-to-end and journey driven through all the different customer experiences and touch points.

Dell is very active in social media with Twitter, Facebook and more. Are you seeing big growth for Dell in those areas?
Well, the UK certainly needs to catch up, with the US being very much the pioneer in this. They’ve done huge amounts in building the blogs and the forums and online activity generally – everything from IdeaStorm, the Dell online forum where customers can post their ideas, to our very active Twitter account in the US, which promotes our Dell Outlet returned and refurbished products and has helped us pass our first million dollars on that. We’re doing the same now and have set up five or six Twitter accounts in EMEA to promote certain things. So we’re very much getting into social media activity here, but it is also at the heart of what Dell had been doing very well for several years. We’ve benefited hugely as a business by using social media tools to join in on the conversations our customers are having about Dell. It’s a strategic imperative to do more in this space.

What innovations have you see during these tougher economic times?
We are actively exploring customisation and personalisation technology, certainly in terms of the products, but particularly regarding the web experience, enabling behavioural targeting. We’re also placing a greater emphasis on testing generally at the back end. This includes both beta testing and multi-variant testing. I’d rather spend time with my team looking at lots of smaller parts of the overall customer journey and get a greater gain, than spending a year conducting a single major redesign. We’re also, like other companies, moving away from being a purely transactional company. Social media tools and content will help with this and allow our customers to engage with us about what they really need and make the right decisions. Overall I’d say don’t get too complicated – simplicity is a model that works well.

What relationship does customer experience have with financial success?
Clearly a lot of what you do with customer experience is very tacit knowledge driven, based on experience you have in figuring out the best solution for individual customers. Sometimes we want to make changes that take us out on a limb a little bit, but having a discussion with finance teams about the possible impact of that is always going to be a difficult decision. So we work hard to find ways to test and measure stuff before we actually do it. However we have found that our own personal experience does count for a lot.

As you move more into social networking and in-store as well as online, is there someone who will take overall charge of customer experience?
There’s certainly no single owner of customer experience. It’s a diverse responsibility spread across the business. That includes teams and centres of expertise in the US that are globally responsible for online, but there are also very strong lines for day-to-day management on the ground in the regions. It would be great if there was a single owner, as that would help with the development of a customer experience focused culture, and that comes from leadership down. We’ve definitely become more consumer focused than ever before and think a lot more about it.

Who offers a great customer experience, in your opinion?
A hard question! From a telco background, I think companies have to learn that it’s not just about selling a product, it’s about developing and building a relationship and keeping those customers and engaging them post-sale. We’re certainly going to take some learnings from the telco side. Me personally, I like those ‘wow moments’ – going through the web and finding some functionality, or copy on a website, or a thank you screen, or anything that makes you think: “You know what? That just made that whole process so much simpler; exactly what I needed.” A passion about the detail of experience along the whole journey through a website will ultimately result in a web experience which is very strong indeed.

The enterprising people at Nokia have set up a $5 dollar comparison website. This interesting site allows you to see pictures that have been uploaded by people around the world showing (you guessed it) things that cost $5. The website aims to explore the relative value of $5 and is asking people to upload their own images. Nokia admit that the amount is an arbitrary one but are most interested in exploring how low cost communication can be brought to the masses (3.3 Billion) still without mobile communication. Being able to tap into this considerable market would be very profitable for whoever gets there first.

It is an interesting concept in these frugal times, but is it one that is going to provide an insight into how to break into this untapped market? I am not convinced. I think it is a novel way to while away 5 or 10 minutes but that’s really as far as it goes. The most interesting thing about the site is what it highlights – the value of 5 dollars around the world. But of course communicating value it is not always as simple as displaying a picture of what it’s worth. Others have tried to highlight value in different ways, Boden for example have a formula that purports to measure the value of their clothes: value = price (£) / times worn.

However, highlighting the value in what we do as an industry is a different story. Companies do not always realise that investing in usability or the user experience can reap huge rewards. But I think, to be fair, a lot of companies are coming around to this way of thinking. However, with all the belt tightening and credit crunching that is going on at the moment, some have opted to try and save by cutting spending on evaluations. This could well be to their detriment as more prudent companies recognise that now more than ever is no time for cutting corners when you are trying to get an edge in a competitive market.

There are a number of ways to stay ahead of the game – analytics can help you measure success and failure and point to areas that cause users to drop off, not to mention understanding who is visiting your site and why. If this is carried out both pre and post user testing, its real value is clear. By combining analytics with user testing or an expert review you can get a more holistic view of your online offering. For cross-channel insight you can think about throwing emotional research into the mix. If your belt tightening is starting to make you feel the pinch maybe it is time to consider some of the above!

Foviance’s Gerry Duffy and Consultant Ronan Tighe are giving a presentation on ‘Keynote’ and remote usability testing at City University’s Open Day. The event will take place on Tuesday the 31st of March at City University’s campus in the Human Computer Interaction Department, London.

Registration and more information