These days, we’re spoiled for choice when it comes to remote user research. The vast array of tools available at many different price points can be overwhelming — especially since many of the descriptions of tools are virtually indistinguishable.
Every remote-research tool promises to deliver user insights, but they do so in very different ways. If you’re trying to choose a tool, this list can help you understand exactly what you’re getting and make sure the service you pick is a good fit for your research needs.
All of the user-research tools compared in this article allow you to do studies that are:
- Remote: Participants can be located anywhere, the entire study is completed online.
- Unmoderated: Participants complete the study on their own, without a researcher guiding the session.
- Task-based: Participants receive instructions to complete specific tasks.
- Behavioral: Users’ actions are recorded by the tool so you can tell what people did and whether they successfully completed the tasks.
- Interactive: Participants can test on live sites or interactive prototypes rather than just seeing a static image.
- Do-it-yourself: You can plan and carry out your own studies, without using the tool’s research-consultancy services.
Together, these qualities allow you to conduct studies that are similar to in-person usability testing, but without the moderator meeting individually with each participant. Unmoderated testing is often a good option when you have limited time or budget or when users are geographically dispersed.
2 Types of Data in Unmoderated Usability Testing
It’s important to understand that different types of data that can be collected by various tools. Some tools record unstructured qualitative data in the form of video recordings; some tools collect highly structured quantitative data about tasks, and some tools can gather both of these types of data.
| Type of Data | Qualitative | Quantiative |
|---|---|---|
| How it is collected | Video recordings capture sceen activity and think-out-loud narration by the test participant | Metrics are recorded for dimensions such as time spent, success rate, satisfaction, and perceived difficulty |
| What it reveals | What participants did and why they did it | How common certain problems, behaviors, and opinions are among participants |
| Challenges | Unstructured video recordings are time-consuming to watch and analyze. | Metrics do not reveal causes of behavior; low participant motivation or inaccurate self-reported data can cause misleading metrics |
| Useful when you need to... |
|
|
Make sure you have a clear idea of what you hope to achieve through your research. Then you’ll be able to decide whether you need qualitative recordings, quantitative data, or both.
Tools and Data Types
The chart below lists 15 different tools which can be used to conduct unmoderated usability testing. The position of each tool in this chart indicates both the type of data collected by the tool and how long the tool has been in existence. Generally speaking, tools which have been available longer are more mature, with more robust features. Also, though there is never any guarantee, a longer-lasting service is less likely to go under in the middle of your study and make any already-collected data evaporate into a lost corner of the cloud.
The diagram above indicates several unmoderated-testing tools which combine both types of data collection. The features listed for these tools are quite similar, so it can be difficult to distinguish between tools by reading their descriptions. Despite these surface similarities, there are important differences between these services, which are easier to understand if you’re aware of the history of each system. UserZoom and Loop11 initially focused on quantitative metrics, and later added qualitative recordings; while UserTesting, Userlytics, and Userfeel initially focused on video recordings and later added quantitative metrics. As you might expect, the tools’ original functionality tends to be more robust, while the newer features are more limited. (These distinctions are represented in the chart above by the placement of each tool’s name, which is positioned closer to its original data type.)
It’s also worth noting that the metrics-only tools included in this diagram, Maze and KonceptApp, are both designed to be used for testing prototypes and are not suitable for testing live websites or applications. Although they can simulate interactions, such as letting test participants click a link and move to another screen, this behavior requires you to actually build or import an interactive prototype.
Feature Comparison
Once you’ve determined the type of data you want to collect, review the precise capabilities of the tools you are considering. Some features which may be important to the design of your study are listed in the table below.
| Recruiting | Study Design & Setup | Qualitative Data | Quantitative Data |
|---|---|---|---|
|
|
|
|
Features that you may need to consider when selecting remote unmoderated usability-testing software
As a starting point for your comparison, we’ve prepared a list of tools and features we were able to confirm for each tool. This spreadsheet provides a detailed feature comparison for 15 tools for unmoderated user testing.
When to Use NONE of These Tools
This article has focused on tools for unmoderated usability testing, but that's not always the right research method. For example, moderated usability testing (whether in-person or remote) is more appropriate for evaluating an early-stage prototype or to identify usability issues in interface or tasks that are so complex that it’s necessary to provide personalized directions and ask followup questions to fully understand users’ behavior. Also, all participants in unmoderated studies are people who were willing and able to install a browser extension or application and carry out a fairly complicated online interaction. If your target audience includes a lot of users who wouldn’t opt to participate in this type of research, you’ll need to use other methods to find and observe these people.
Finally, some research questions are better answered with a completely different type of study, such as an A/B test, 5-second test, interview, field study, card sort, or tree test. (Some services — most notably UserZoom — support a wide range of such research methods.) You should always figure out which research method best addresses your question before choosing any tool.
Full-Day Seminar on Remote Studies
For detailed help planning, conducting, and analyzing remote unmoderated user testing, check out our full-day seminar: Remote Usability Testing.
Share this article: