The more the merrier?

When it comes to deciding how many users to recruit for user testing, nobody seems to agree on an ideal sample size. Perhaps more precisely, nobody actually seems to know. This is probably because user testing straddles two seemingly antagonistic domains: business and science.Whenever a client asks me how many people I think will be needed for a particular project, the first thing that comes to my mind is the dreaded: “It depends”. In practice, I generally opt for the rather less elusive response: “Let’s talk a bit more about the tasks before we decide.” In truth, when it comes to sample size, “it depends” is probably more accurate!

First and foremost, sample size is dependent on the type of study. There are a few voices both in the usability and the academic worlds preaching about the ideal number of users in a standard usability evaluation. It is generally agreed that we get value for money with five to eight participants, because on average somewhere between 80 percent and 85 percent of problems are identified using those sample sizes. To unravel closer to 100 percent of problems we would need perhaps twenty people. The maths is simple: why spend 150 percent more on recruitment to get 15 to 20 percent more in terms of results?

There is, of course, some variation in these numbers, but generally, when it comes to standard evaluations, it is fairly easy to decide on sample size. Rich data is extracted, behaviour is observed and interpretation of the results relies on known best practice and experience. Things get a bit more complicated, however, if we use quantitative measures of behaviour, such as eye tracking or quantitative survey data. As someone with a lot of experience in eye tracking, I often get asked how many participants to recruit for such projects, and invariably, people are once more in danger of hearing the dreaded: “It depends”.

Clearly, sample size is related to the complexity of interfaces and tasks. The more complex, the more people we need to test as data variability increases. But crucially, sample size depends on the behaviour the test is set to measure. This, in turn, depends on what the objectives of the study are. For example, to know whether an advert is going to be noticed when users perform their usual tasks on a page, 20 people might be required. However, to know how long on average it takes people to look at the ad, more people are needed because of the huge variations between participants.

With surveys, sample size estimation is also somewhat less straightforward than with standard usability evaluations. Here, the information being collected is attitudinal data, which by its sheer nature can be slightly fuzzy. It all comes down to the size of the effect you intend to detect. Imagine you wanted to know whether people in London are taller than people in New York. If people in London and people in New York are actually pretty much the same height, you will need to measure a high number of citizens of both cities. If, on the other hand, people in London were particularly tall and people in New York were shorter than average, this will be obvious after measuring just a handful of people.

What sample size does not depend upon, is the size of the original population. Whether we are testing people that belong to the whole population of Europe or teenage boys that only wear Ecko clothes and speak with a South London accent, the factors weighed to estimate sample size should be: interface and task complexity, sensitivity of measure and effect size, and the variability between the users.

Of course, in any case, the more the merrier, but this is only possible in a world where resources, such as time and money, are infinite. In the real world, we compromise, and the trick is in being able to achieve a good balance between rigour and value.