Remote Moderated Usability Tests: Why to Do Them

Under normal circumstances, few teams have enough time and resources to perform as much in-person usability testing as they’d like. Due to recent quarantine and travel restrictions, almost all UX teams are suddenly finding that in-person methodologies are not currently possible.

Acting under the (correct) assumption that any user data is better than no data, many teams are turning to remote usability testing. One of the most popular variations of remote testing is unmoderated usability testing.

Unmoderated usability testing (also known as asynchronous testing) is a popular way to get a product tested by users without traveling or breaking the bank.  It usually involves using one of the many available services (such as What Users Do), setting up some tasks, and waiting for the data to be collected. 

This method has some substantial benefits: 

• No recruiting (if you’re using the built-in panels of users that the remote-testing services provide)
• No moderation skills needed
• Easy test setup 
• Fast results
• Low cost

However, the data obtained from such remote unmoderated sessions is often less detailed than that from in-person testing. This difference is accounted by the serious drawbacks of unmoderated remote testing:

• Short test sessions (the length is limited by the testing service’s rules, but usually is around 20 minutes, compared with 1–2 hours for in-person studies)
• No way to clarify slightly ambiguous or unclear instructions
• Inability to ask users to elaborate on a comment or continue using an area of the design
• Unrepresentative testers
• Variability in participants’ motivation and commitment to the test 
• High chance that testers will multitask or get distracted by their environment

In contrast, remote moderated usability testing combines the advantages of both methods (in-person and remote unmoderated): it can deliver high-quality findings (comparable to in-person testing), but is convenient and inexpensive (like remote unmoderated). The benefits of remote moderated usability testing include: 

• The facilitator may change, skip, and reorder tasks as needed.
• The facilitator may ask follow-up questions or for clarification if needed.
• The participant is less likely to spend time on activities unrelated to the test.
• The situation may feel more natural than talking out loud to oneself.
• The test sessions can be longer (usually about an hour) and leave room for in-depth exploration of a design.
• The team can watch the test at the same time and discuss the findings immediately after the session. 

We acknowledge the two most resource-intensive parts of moderated remote testing, which are:

• Recruiting users: You can’t take advantage of a built-in panel. However, you can also recruit users to match your target audience. Use a recruitment agency, your own user database if you have one, your website, social networks, and friends and family.
• Setting up the meeting software: It can take extra time and effort to ensure that it works for your test participants, your observers, and you.

Even with these hurdles, remote moderated testing is easier than most people think. Check out our checklist with descriptions for each step.

Three Usability-Testing Formats Compared

When to Use Remote Moderated Usability Testing

If your team’s UX-research resources are severely limited, remote unmoderated usability testing may be your only option. This is often the case for quantitative usability studies, which often require more than 30 participants. In such situations, remote unmoderated usability testing is certainly better than no usability testing.

However, remote moderated usability testing may be a better fit than remote unmoderated testing or in-person moderated testing if:

• You want deep insights and rich data
• Your participants are busy, geographically distributed, or otherwise cannot travel to a testing location
• Your researchers have enough time to meet with each participant individually

Give Remote Moderated Testing a Try

Remote unmoderated testing has the benefit or being fast, inexpensive, and easy. It can get some great insights, and should be part of every UX researcher’s toolbox. 

However, remote moderated testing can give you significantly more useful, interesting, detailed findings than you’ll get from remote unmoderated tests. It takes a bit more coordination, but the small amount of extra effort is well worth the beneficial impact this methodology will have on your research. If you’ve never tried it, there’s no better time than now.

Full-Day Seminar on Remote Studies

For detailed help planning, conducting, and analyzing remote moderated user testing, check out our full-day seminar: Remote Usability Testing.