You already know that user testing is critical to the success of your product. But for many EdTech companies, actually sourcing test users is a perennial challenge. You see, when it comes to user testing, not just any users will do. You can either build a bespoke set of actual users of your product or you can utilize a service that brings their own preselected group. In order to get meaningful insights, you need to start with the right users. To make matters even more complicated, the definition of “right” changes depending on your research goals.
In their quest to put together the perfect mix of test users, some EdTech companies struggle to go beyond their core group of established power users. Others have a broad database of current users but aren’t sure how to identify non-users. And still others find themselves relying on a network that lacks the right level of diversity to support their research objectives.
For all of these reasons and more, you may be thinking of outsourcing a preselected panel of users from an online user research service. Before you pull the trigger, stop and ask yourself: Do you understand the pros and cons of preselected panels? When are they a good idea? And when are they actually counterproductive? Keep reading for answers.
How to Find Users for Usability Testing: Preselected vs Custom User Panels
Most EdTech companies start with custom-built user research panels for their UX testing. That is, they hand-select the individuals who participate in any given user test. Some of the largest EdTech companies have amassed a large, demographically diverse pool of ready-made test users from which to pull on demand. But most other companies don’t have that luxury. Often, each round of research finds them scouring their professional networks and querying their most active users to put together a fresh user panel.
Preselected user panels, which are typically supplied by a third-party user research service, attempt to solve that problem. With a preselected user panel, EdTech companies can purchase access to a much larger pool of test users. Depending on the service, they may be able to apply any number of demographic filters — including age, geographic region, and job title — to quickly fine-tune a panel’s attributes.
The Pros and Cons of Preselected User Panels for EdTech Testing
Preselected user panels can be both valuable and convenient. But before you engage one, it’s critical that you understand both the benefits and the possible pitfalls.
Preselected User Panel Pros
First, let’s look at the pros. Preselected panels allow you to:
- Move faster. Rather than scrounging around for test users, you can quickly pull together a panel as needed — and get moving on your next round of user research.
- Balance your “usual suspects” with a pool of non-users. Many EdTech products know how to enlist active users to participate in their UX research initiatives. But in most cases, it’s important to test your product on a mix of established users and those who have never encountered your product before. Lining up non-users is often the most challenging part of building out a user panel. With preselected user panels, you can quickly and reliably augment your roster of known users to create a more balanced overall panel.
- Ramp up the diversity of your test user pool. If you only rely on your own in-house panel, you may be introducing unintentional biases by virtue of the demographic limitations of your organization’s network. For example, you may only include participants who are located in the midwest, associated with four-year institutions, or who already know about your product and are interested in your work. A preselected panel of recruited users offers the possibility of greater demographic diversity.
Preselected User Panel Cons
Now for the cons. Preselected user panels may introduce the following risks:
- “Professional” test-takers can skew your results. When it comes down to it, preselected user panels are basically a self-selected group of professional usability test takers. They’ve applied for the job of being a test user. In some cases, they may have had to prove their ability to follow instructions well, use technology capably, and provide clear feedback. They may even have received training about how to key in on certain elements and interact with various user interfaces. Not only that, but preselected test users may participate in hundreds of user tests. In short, though they may not know your product, preselected test users are “power users” in their own right. If you rely too heavily on them, you may introduce unintended biases — and get skewed results. For example, when vetted by a preselected panel, your product may appear to be more user-friendly than it actually is for the broader public.
- Preselected user panels may not be targeted enough to support your research goals. Anytime you conduct user research, you want to include at least a percentage of your product’s customers. But if you rely solely on a preselected user panel, you very likely won’t have the ability to include users who are familiar with your product. They may be out there, but you likely won’t be able to screen for that attribute. In fact, depending on the service you use, you may not even have the luxury of presenting your product to testers who fit the profile of your target audience.
- Professionally recruited test users may be incentivized to say what you want to hear. Test users are typically compensated for their time. But for preselected test users, it’s not just a one-time research stipend. It represents an ongoing paycheck for the subjects and therefor may introduce conscious or unconscious bias. Which means that when they are included in your product’s study, they are reporting to you. Like all “employees,” they may try to please you by telling you what they think you want to hear.
Preselected User Panel Best Practices for EdTech UX Research
Preselected user panels really can be useful. But only if you approach them the right way.
First things first. In general, you should never use preselected user panels on their own. Rather, start with a list of your own product’s existing users. There’s almost no UX research scenario in which you won’t want to include at least a few of them. Once you have a core group of your product’s users, you can round it out with a preselected user panel.
If you do decide to use a preselected panel, make sure to vet the service you use to ensure the panel will meet your needs. Ask them:
- How do you recruit test users to your service? Do they have to apply and prove they are “good” at participating in user tests in order to make it on your panel? (Application-based recruiting practices may promote built-in biases. Ideally, you want some sort of organic method to attract users and get people to sign up.)
- How big is your pool of users, and how frequently do you cycle through them? How often does each individual user participate in a UX research activity? (Remember, you don’t want a panel made up of people who are constantly giving feedback.)
- How diverse is your pool of users? Are they geographically centralized? Do they represent a range of ages, areas of study, and educational institutions? What about technological literacy?
- What sort of screening tools are available, and how granular can I get in terms of selecting a panel’s attributes? (The more granular the screening tools, the better your ability to customize your user panels.)
Finally, keep the goals of your UX research in mind — as well as your longer-term goals for your product. Consider your current users. Now, think about what you want your audience to look like 3-5 years from now. If the two groups are different, start testing with your future market in mind now! That may mean reaching out to users of a competitive product or prospective customers who previously turned your product down. And it could also mean leveraging a preselected user panel with the appropriate filters in place.
Bottom line? Preselected user panels can have a place in your EdTech product’s UX research plan. But they should never fully replace the hand-selected user groups you currently rely on.
