User testing is a crucial component of successful product design. Without the insights that on-the-ground testing yields, designers can only guess at how users will actually interact with their products.
But the reality is that not all UX research methods are created equal. Without a firm understanding of testing biases and a healthy dose of observational research, your user tests might unintentionally serve up skewed results—and end up leading the UX of your product astray.
For example, we recently tested a product where users struggled to complete a task—let’s say it was entering scores from a quiz. In attempting to complete the task, our test users clicked all over the screen and ran into plenty of speed-bumps that were clearly difficult to navigate. However, at the end of the process, they all rated the product highly, saying that it had been easy to use. Had we relied on self-reporting alone, we never would have identified the pretty serious workflow issues that still needed to be ironed out.
Understanding the Role of Bias in UX Research
To understand exactly why the structure of user tests is so important (and why we so strongly recommend observational research), you first need to have an understanding of biases as they relate to UX research methods.
All people are naturally biased. That’s no surprise; a quick scroll through the news feed on your favorite social media app will likely bear this statement out. But in the context of user testing, biases are much subtler than things like political preferences and religious affiliations. In fact, the biases that impact user testing are so subtle that the people involved likely won’t perceive them at all.
Users are often eager to please, eager to seem smart, and quickly forget when things have gone wrong if they achieve a goal. If we as testers aren’t careful, these very pleasant tendencies can give way to misleading biases. The following three biases are especially important to understand since they often cause users to unwittingly change their behavior in a testing environment.
- Social Desirability Bias: Users don’t want to be judged for not understanding something or for using a product “incorrectly,” so they may unintentionally modify their behavior in an attempt to look more proficient. The best way to minimize this bias is by putting your test users at ease and reassuring them they aren’t the ones being tested. Remind them that you are testing your own interface, so if something isn’t working, it’s a problem with the interface and not the user. Stay neutral as test participants move through the test session, and make sure to ask open-ended questions that don’t imply right and wrong answers.
- The Hawthorne Effect: While observational research is key for a number of reasons, the very act of being observed can cause participants to change their behavior in subtle ways. Since the effect of this bias often goes down over time, the best way to reduce it is to plan for longer observation sessions in the hopes that users will forget you’re there. You can also attempt to ameliorate the Hawthorne Effect by directly addressing it with test takers. Tell users to pretend you aren’t there as they go through the testing session. You can also ask them how they might handle things differently if they weren’t being watched. You may be surprised at how much insight users can bring to this question once it’s brought to their attention.
- Hindsight Bias: Hindsight bias refers to the fact that people tend to subconsciously filter memories of past events through the lens of present knowledge. In the context of user testing, self-reporting is less reliable than observational research because of this very phenomenon, with users often reporting that an interface was easier or more manageable than it actually was. Only by observing users during testing does this important gap come to light. What people say isn’t always what they mean, so we need to watch them in action to collect a broader range of data points and help fix experiences that aren’t reported as issues.
It would be impossible to eliminate all the biases that might come into play in a testing environment. Still, having an awareness of the existence of the most common biases—and how best to counteract them—goes a long way toward reducing the problem.
The Importance of Observational UX Research Methods
By now it should be pretty clear that you don’t want to make UX decisions based on verbatim and self-reporting alone. But hindsight bias isn’t the only reason to prioritize observational studies as part of your UX research methodology.
We want our products to fit into the lives of our users. People won’t change their behavior to use your product, and there’s a lot more that goes into someone’s day than their strict interaction with your software. You need to understand how your product fits into users’ lives by watching them in action.
Observing Users in Their Own Environment
We can’t know how users live their lives unless we observe them, ideally in their own environments and contexts. For example, if your product is used to help manage coursework and assignments, then you need to have an understanding of how and when it’s being used by instructors and students. Only by observing your product in action might you learn that instructors typically assign coursework while looking at an on-screen calendar, meaning that your product’s screen needs to be responsive enough to work alongside another app. Careful observation allows variables like this one to become known entities that your product can then accommodate.
In-person, in-context observation is the gold standard for user testing. But, of course, there are a number of reasons why it’s not always feasible, from budgeting to timing and logistics. The good news is that a lot can be learned by simply observing users click through prototypes in remote moderated sessions. And this is all the more true when you have an understanding of how to coach testers in a way that reduces bias.
EdTech companies face a number of pressures when it comes to developing and releasing new products, and it can be tempting to save time by minimizing user research. But not knowing enough about users is the biggest danger that many EdTEch companies face. If you short-change observational UX research, you are guaranteed to design a product that is less responsive to users’ needs and therefore less successful—and you may not even know why. So, when it comes to user testing, measure twice and cut once. Do that, and your stakeholders and users will thank you.
