EdTech products must meet the needs of the students, teachers, and administrators they serve, regardless of ability. The goal of inclusivity doesn’t just make sense because it is the right thing to do. Or because it is in keeping with the goals of educators more broadly. It also makes sense because it leads to a better-quality product — for everyone.
For all of these reasons, more and more EdTech companies are embracing inclusive design. That’s a good thing. But the focus on design may unintentionally obscure the need for similar practices elsewhere in the product development process. The reality is that no digital product can be truly inclusive if the UX testing that helps to define it is exclusive. If your UX testing is structured in a way that eliminates a whole group of target users’ feedback, your product will suffer.
So what does it mean for UX testing to be inclusive? At Openfield, we’ve been asking that question for a long time. Along the way, we’ve made a few simple changes to our UX testing protocol. These changes allow us to draw more accurate data from a broader swath of users.
Our new, more inclusive user testing process doesn’t just lead to more reliable test results. It also enables us to design products that are more meaningfully shaped by and for the full spectrum of their intended users.
The Need for Inclusive UX Testing
UX researchers rely on a set of best practices to structure UX testing. Their goal is to conduct testing in a way that allows a broad range of users to provide their honest feedback about a product.
To that end, researchers take great pains to ensure that their test results are reliable and unencumbered by bias. For example, they avoid asking leading questions and strive to provide a consistent testing experience from one test session to the next.
These best practices form a solid foundation. But they don’t necessarily account for the needs of users with disabilities or other special circumstances. When it comes to making UX testing truly inclusive, the devil is in the details. And those details depend on the particular needs of your users as they relate to their ability to successfully participate in your test sessions.
This is illustrated by the adjustments we’ve made to create an effective testing environment for English-as-a-second-language instructors and students. ESL teachers and students can be found in just about any educational environment. But for STEM products geared toward higher education settings, ESL users often form a much larger percentage. We discovered that for this group, the usual approach to user testing wasn’t sufficient.
Traditionally in a moderated user test setting, researchers verbally describe the background, scenario, and each task that test participants must complete. Moderators may use a script to ensure that their language is consistent from one test session to the next. However, all conversation and instructions are verbal.
For obvious reasons, this reliance on verbal communication can be problematic for ESL users. On the one hand, they might misunderstand the instructions without realizing it. On the other hand, they might be aware that they don’t fully understand the instructions but feel too embarrassed to ask for clarification. Either way, the result is a test session that doesn’t accurately reflect the users’ ability to successfully navigate a product.
When there is a language gap or a misalignment in a users’ understanding, researchers have a few options. They can throw away the data and find another participant who is easier to talk with. They can make best guesses about the data and risk accuracy in reporting. Or they can lose time in the session trying to explain the scenario and tasks to the users — potentially introducing additional bias from setup not given to past users.
None of these options is good. And all of them risk ruling out a whole category of users who will eventually judge your product based on how well it meets their needs.
The Path to Inclusive User Testing
The path to inclusive UX testing begins with a basic understanding of your users and their particular needs as they relate to testing.
As soon as we realized the standard UX testing protocol didn’t work for many of our ESL test users, it was time to adjust our process. We needed to find a way to present testing instructions that minimized the language gap.
Our simple solution was two-pronged.
First, we introduced slides with testing instructions and background information written on them to accompany our conversation. Second, we started using a reading-level checker to make sure that all of our written instructions were at or below the eighth grade reading level. This ensures that we only use common, easily understood language to convey our instructions. We’ve adopted both of these new practices for all of our user testing (not just testing that includes a large percentage of ESL users).
In practice, it’s very simple. When we moderate user tests, we still verbally explain the test instructions. But we also project the clear, simple written version of the instructions on a slide. We display the written instructions throughout the entire test session. That way, as users are going through the test, they can refer back to the instructions as needed.
The interesting thing is that these small tweaks didn’t just enable us to conduct more meaningful test sessions with ESL users. They actually improved our testing across the board. That’s because they provide another layer of clarity and consistency in terms of how we present information to our test users. They reduce confusion for non-ESL users who have difficulty hearing as well as those who process written information more easily than auditory instructions.
Of course, this new process doesn’t solve every inclusivity challenge related to UX testing. However, our experience does demonstrate that small tweaks can make a big difference. It also proves that when you make the effort to address the needs of users with special challenges, everyone benefits.
