Mariana Da Silva

Mariana's bio

Mariana DaSilva joined Foviance in 2007 after 2 years at UCL researching online interactive search, following a PhD in Cognitive Psychology and a Postgraduate Diploma in Business Innovation. She has helped many global brands improve their online customer experience, particularly in the Gaming sector.

“I really enjoy working in the gaming sector. It’s incredibly multifaceted and full of interesting characters, both on and offline! Over the past two years I have tried to learn how to use all available tools in measuring customer experience so I can mix and match them to provide the perfect solution every time a client comes to me with a problem.”

As geographically separate regions of the earth are brought ever closer together by the pervasiveness of the World Wide Web, it is only natural that businesses attempt to extend their reach beyond national boundaries via their online presence.

However, ‘internationalising’ a site is a far more involved process than merely translating it, and without intelligent international research with native users, it is not possible to truly localise a site. Read more…

You might well have come across multivariate testing techniques before in your explorations into customer experience measurement, but for the uninitiated, here is a brief definition that puts the methodology into context.

Multivariate testing, or MVT, is an experimentation process by which a series of possible design variables are tested at once to see what effect, if any, they have on website performance. It’s a complex form of split, or A/B testing, employing algorithm-based software and constant monitoring of web analytics data. Small changes are made to single variables (such as the position of a menu, the colour of a background) and the impact of each change is measured. From series of changes, optimum design configurations can be narrowed down as a result of measurable evidence. With MVT it is also possible to experiment with structural, business rule and database driven elements, as well as cosmetic changes. We can even employ advanced rule-based targeting capabilities, including targeting by geographic location, traffic source (such as search engine versus email campaigns), cookies, and more.

Read more…

In-game advertising and in-game product placement are both very effective ways for companies to get their brand out there in front of millions of consumers immersed in a world that remains relatively unregulated – especially in comparison to TV, for example.However, the jury is still out over placing adverts or product names in games. The difference between right and wrong is likely to depend on how this is done, rather than on whether it is done at all. Get it wrong, and you risk seeing irritated customers leave in flocks, with practically no chance of their return. This is particularly problematic in a world that is based on a very strong sense of community and word-of-mouth. Even the most anonymous of gamers can ultimately make or break a game’s popularity with a few incisive comments or reviews.

Last year, the Consumers’ Experience with In-Game Content & Brand Impact of In-Game Advertising Study, a study conducted by Nielsen on behalf of IGA Worldwide, the leading in-game advertising network concluded that integrating adverts into gaming environments boosted consumer awareness and opinion of brands. 82 percent of respondents even said gaming was just as enjoyable with ads as without.

Certainly modern consumers tell us they want to have fun, escape and be more empowered. To accommodate such needs, game developers are under pressure to produce environments that are as realistic as possible. Any in-game advertising has to play be these same rules. Consumers get increasingly grouchy about advertisement that is simply thrown at them, but this is not the same as saying consumers don’t want advertising at all. Executed sensitively, adverts can add to the gaming experience by creating more realistic settings, further unlocking fun and escapist qualities. Developers might even argue that achieving the correct ambiance in a medium where immersion is key even necessitates overlaying real product names and logos onto virtual characters and environments.

Getting it right also involves consideration of context. Car manufacturer billboards on Gran Turismo tracks make perfect sense. Having the blood-stricken monsters in Dead Space walk around wearing branded clothes is likely to cause more of a stir. The suitability of the streets of Burnout Paradise as places to display Barak Obama’s campaign billboards is less clear. But marketers should certainly be aware of the following growing ‘rules’:
• Ads should be found where gamers want them and if they want them
• Ads must make sense to the target audience
• Ads should be easy to notice but simple to ignore

Interactivity and personalisation is the obvious way forward for fun and empowerment. Getting Lara Croft to change into a Rip Curl wet suit just before she dives into the sea in search for hidden treasures is not likely to offend gamers and may even give them a sense of reality. It also gives gamers a chance to have their favourite character engage with a brand they recognise in real life, adding to the gaming experience and to the gamers’ sense of identity and individuality.

Currently, the biggest game development houses have no need to open their doors to brands in order to finance their creations. However, financial return should not be the sole reason for a dialogue between the two to exist. Ads can easily disrupt the immersive experience with potentially catastrophic consequences to a game or gaming brand, but they may equally contribute to that experience if executed properly. With the right research, game developers and brand owners could develop positive relationships that are crucially based on the gamers and their user experience, and not just their business model.

When it comes to deciding how many users to recruit for user testing, nobody seems to agree on an ideal sample size. Perhaps more precisely, nobody actually seems to know. This is probably because user testing straddles two seemingly antagonistic domains: business and science.Whenever a client asks me how many people I think will be needed for a particular project, the first thing that comes to my mind is the dreaded: “It depends”. In practice, I generally opt for the rather less elusive response: “Let’s talk a bit more about the tasks before we decide.” In truth, when it comes to sample size, “it depends” is probably more accurate!

First and foremost, sample size is dependent on the type of study. There are a few voices both in the usability and the academic worlds preaching about the ideal number of users in a standard usability evaluation. It is generally agreed that we get value for money with five to eight participants, because on average somewhere between 80 percent and 85 percent of problems are identified using those sample sizes. To unravel closer to 100 percent of problems we would need perhaps twenty people. The maths is simple: why spend 150 percent more on recruitment to get 15 to 20 percent more in terms of results?

There is, of course, some variation in these numbers, but generally, when it comes to standard evaluations, it is fairly easy to decide on sample size. Rich data is extracted, behaviour is observed and interpretation of the results relies on known best practice and experience. Things get a bit more complicated, however, if we use quantitative measures of behaviour, such as eye tracking or quantitative survey data. As someone with a lot of experience in eye tracking, I often get asked how many participants to recruit for such projects, and invariably, people are once more in danger of hearing the dreaded: “It depends”.

Clearly, sample size is related to the complexity of interfaces and tasks. The more complex, the more people we need to test as data variability increases. But crucially, sample size depends on the behaviour the test is set to measure. This, in turn, depends on what the objectives of the study are. For example, to know whether an advert is going to be noticed when users perform their usual tasks on a page, 20 people might be required. However, to know how long on average it takes people to look at the ad, more people are needed because of the huge variations between participants.

With surveys, sample size estimation is also somewhat less straightforward than with standard usability evaluations. Here, the information being collected is attitudinal data, which by its sheer nature can be slightly fuzzy. It all comes down to the size of the effect you intend to detect. Imagine you wanted to know whether people in London are taller than people in New York. If people in London and people in New York are actually pretty much the same height, you will need to measure a high number of citizens of both cities. If, on the other hand, people in London were particularly tall and people in New York were shorter than average, this will be obvious after measuring just a handful of people.

What sample size does not depend upon, is the size of the original population. Whether we are testing people that belong to the whole population of Europe or teenage boys that only wear Ecko clothes and speak with a South London accent, the factors weighed to estimate sample size should be: interface and task complexity, sensitivity of measure and effect size, and the variability between the users.

Of course, in any case, the more the merrier, but this is only possible in a world where resources, such as time and money, are infinite. In the real world, we compromise, and the trick is in being able to achieve a good balance between rigour and value.

Over my years of working as a user experience consultant, I’ve come across numerous examples of what I call the “nonsense usability dilemma”. It happens when the needs of the customer diverge from the perceived requirements of parts of the business. It goes something like this: “If a site is easy to use, visitors will not stay for long and that’s no good”.

One of the biggest causes of this is the belief that it’s vital that visitors to the site are exposed to all the content. This argument is used to defend convoluted user journeys, where site visitors end up disorientated and frustrated. It’s the online equivalent of changes in supermarket design, implemented under the impression that confused and lost customers buy more products.

In a previous engagement, a business division within a major client had mis-interpreted the objective that ‘the interface should provide the best experience possible but also ensure users are not pushed to leave too early’. They had taken this to mean that the user should be delayed from achieving their final objective by being exposed to as much ‘interesting’ content as possible along the way.

On another occasion, I can remember sensing disappointment after a few minor changes on a website (suggested by us) significantly improved conversion rates but reduced the number of products explored on the average visit.

The truth is that users want to find things quickly. The number of page impressions per visitor is not, by any means, a direct measure of a good user experience. If an average user stays in a site for 10 minutes and visits 30 pages but doesn’t buy anything and never comes back, how does this help the business achieve its objectives? Furthermore, in these days of Web2.0, they may also post negative comments about the site, which is likely to cause some damage to the brand.

It has recently become common practice to bundle eye tracking technology as standard with usability testing. Not only is this somewhat surprising, it is also very disappointing, because it sends out the misleading message that eye tracking is simply an attractive accessory to usability testing, in the same way that a pair of earrings might complement a necklace.

At a time when usability testing has practically become a commodity, usability companies should be striving to provide clients with customised solutions that answer questions directly and resolve specific problems. Such an approach requires a level of sophistication that hinges on choosing the right set of tools and applying them appropriately, rather than using multiple tools for the sake of the aesthetic impact of the deliverable. Read more…

Eye tracking is a powerful tool because it can identify behaviours that users cannot articulate, or may not even be aware of. However, one of the most common ways of implementing it in usability testing can render its results meaningless.

Research tells us that the way the eyes flit over a webpage reveals what the user is thinking. For example, repeated fixations on an object usually mean the user is trying to understand it or work out how it fits with the page or the task. Long dwell times could mean something is taking time for the user to understand. Organised scan paths show that the user’s attention is efficiently distributed, and therefore so is the information.

With this knowledge, a usability consultant can combine eye tracking data with other data – such as task completion time, error rate and backtracking rate – to answer a wide range of questions: Which of two designs is more efficient? Why are users not using a button? What elements in the page do users process first? Do users look at irrelevant parts of the page?

But one of the most common implementations of eye tracking in usability testing involves asking participants to think aloud, and explain their actions throughout the test. It provides the facilitator with an insight into the reasoning and motivation behind participants’ actions, but it can also affect the user’s attention and inflate task completion times. As a result, the eye tracking data loses its reliability.

The answer is to use retrospective think-aloud protocols, where users can explain their actions after they have completed them so that the eye tracking metrics are not biased. To get the best from eye tracking, test participants must be allowed to act first and think later.