Testing international waters remotely

Although remote testing will never be a complete replacement for traditional user testing, it does have its role in the user experience professional’s tool box – and it’s becoming an increasingly important one.

Traditional one-on-one website usability testing reveals a rich insight into any suitable site evaluated. It uncovers lots of qualitative data while providing a perfect opportunity to watch users’ behaviour, interpret their actions, and engage in conversations that delve into their opinions and difficulties. So when is remote testing appropriate?

There are times when small, qualitative samples are not suitable. Remote testing can instead provide a large volume of quantitative, statistically significant data. It can provide proof of issues on a large scale, or produce the figures needed for a business case. Remote testing and traditional user testing can also be used in tandem. Remote testing can provide statistical data to back up findings from a usability study. Or, perhaps, findings from a remote usability study can be expanded and explored during a one-on-one study.

As more brands become global, it is more important than ever for companies to optimise their international offering. There is no better way to understand the quality of an international presence than to test a company’s website in the international community. Current financial instabilities have heightened this importance, despite making funding of such testing harder to achieve. Remote international testing eliminates a large proportion of these costs – certainly a good deal of travel – while potentially enabling a single team to conduct all tests.

Of course not all types of remote user testing should be viewed in the same way. Certainly a clear distinction must be made between remote moderated (using screen sharing software) and remote unmoderated testing (using in-depth customer experience research tools such as WebEffective), and studies should be designed with this in mind.

Although it is conducted via some form of screen sharing software, remote moderated testing is very similar to traditional user experience testing. A moderator conducts the tasks in a similar way to a traditional user test, and is on hand to answer or ask questions at any time. They are even able to observe certain facial expressions and behavioural nuances via a webcam. Emphasis must be placed on ‘think aloud protocol’ to ensure that nothing is lost from the visual cues that cannot be seen.

Remote unmoderated testing is an entirely different story. Because there is no moderator present, any communications with participants must be really clear. There is no chance to ask questions, and if instructions are misunderstood this could easily lead to inaccurate results, possibly turning the whole study into an expensive mistake. The study should be written with no room for error and participants must be prompted to give full answers. There are pros and cons for this unmoderated form of remote testing. On one hand, participants without moderation will have less social anxiety about providing unhelpful feedback and therefore may be less inclined to write a lot. On the other hand, the lack of a moderator may give the participant more freedom to voice their opinions on more sensitive topics, or where the participant would feel embarrassed if identified.

Understandably, the person running an unmoderated remote study has even less control over their participants’ environment than with a moderated remote study. This means there is less control over distractions, or interruptions from the study which can affect completion times, but over a large enough sample this should even out.

It should be remembered throughout that with any international testing, evaluators must recognise and be sympathetic to differences in participants’ language, social conventions and mental models in order to be culturally sensitive and collect useful results. It is also advisable to understand variations in laws surrounding the treatment of human subjects in different countries. As the need for remote user testing increases, it is essential to understand the need to plan studies that always uses methodologies that best suit the individual project.

Here at Foviance, we are experts in both moderated and unmoderated remote user testing, and will be releasing the results of unmoderated industry studies with our partner Keynote later in the year. We’ll share our analysis with you as soon as it is available.