Usability testing with a small number of participants is an incredibly efficient way to improve an interface. By observing real people attempt to use a design, we see the interface from the users’ perspective. Since average users lack insider knowledge about how the system is “supposed” to work, they often encounter problems that the creators of the design didn’t foresee.
Discovering these unexpected problems is the point of doing usability testing. But the very fact that they are surprising, and not problems that stakeholders anticipated, often leads to doubts about the whether the issues observed are actually representative of what ‘real’ users would encounter.
Stakeholders who have doubts about whether they should trust findings from usability- tests often assert that test participants are not representative or that there aren’t enough test participants to take the results seriously.
When you encounter these objections, here are some techniques you can use to help teams understand why they should take qualitative usability testing seriously. In the simplest cases, skepticism may be simply a consequence of a lack of experience with the methodology, and can be successfully addressed by clearly explaining the method, and communicating findings in a compelling format.
Prepare Stakeholders with Proactive Explanation
First and foremost, make sure to explain your research method before conducting the usability sessions. Describing the process in advance is much more persuasive than trying to defend it after people have already become skeptical. Prepare your own explanation, or share videos and articles that cover what usability testing is and why you need only a few participants. Acknowledge the limitations of this method upfront: mention that some questions, like whether people believe a certain brand is reliable, could not be accurately answered by talking to just 5 users. But usability testing isn’t about beliefs or preferences — it’s about discovering how normal users behave in the specific context of using a particular design for a particular task.
Prepare stakeholders for surprising findings by explaining:
“We’ll probably see people encounter problems that we don’t expect, because our familiarity with the design makes it hard for us to spot some issues.”
“Since all users lack our team’s insider knowledge about the design, it’s likely that problems caused by this difference of perspective will affect many users and will show up within the first few test sessions.”
“Rather than trying to prove exactly how bad each problem is by repeating tests with a lot of users, we will focus on quickly discovering common issues, so we can fix them and improve the design.”
Share this context with members of the team in the early planning stages, especially if they are not familiar with usability testing. Even teams that are familiar with usability testing benefit from reminders that they are not like their users.
Hearing Is Believing
Sometimes stakeholders dismiss test results because they view them as just one more opinion — the opinion of the researcher who wrote the report.
The whole point of usability testing is to hear from users. Insights and analysis from a skilled researcher can add great value to a report, but always make sure that users’ direct feedback, in their own words, comes through loud and clear. You can prominently incorporate direct quotes from your study participants; video clips are even more compelling (if you are able to record sessions and have time to edit the footage).
The most persuasive method of all is to arrange for teams to actually directly observe the usability-testing sessions. This practice allows them to see for themselves that the test participant is a real person and develop empathy for the user’s struggles with the system. When team members directly observe sessions, they can also see that the researcher is not leading or influencing users.
Addressing Objections that Participants Aren’t Representative
To avoid objections that test participants are not representative of “real” users, make sure to test with users who are actually representative — and share how you recruited them in advance with your stakeholders.
Test with Real Users
If you are testing a live website or application, recruiting actual visitors or users is an effective way of finding representative participants. Today’s sophisticated analytics tools (such as HotJar or Ethnio) make it easy to target specific types of users, such as people who are using a particular section of a website or a certain feature of an application. Lists of past or prospective customers are another great source for realistic test participants.
Get Buy-In on User Profiles Before Recruiting
If you can’t recruit participants directly from the actual system, you may need to recruit participants from other sources. But you can still ensure that your participants are representative by carefully defining which traits are important characteristics of your target audience. If you have personas, refer to them. Ask the stakeholders who will be receiving the research findings for their input about who the target audience is. Then screen potential test participants by asking them questions which selectively identify people with all the essential characteristics of the target users.
Whichever method you use to find participants, describe it to your stakeholders every step of the way: during the planning stages of the study, by distributing participant profiles to stakeholders who observe test sessions directly, and as part of the final report.
Objections that There Aren’t Enough Participants to Prove a Problem
Proactive explanations of the usability-testing methods go a long way towards building credibility and confidence in test findings. But there may still be concerns that a small number of people experiencing a problem doesn’t necessarily mean it’s a common issue — especially if only a single participant in the study encounters the problem. That participant could be an “outlier,” who has unusual expectations or habits that are not shared by many other users.
The truth is, seeing at least 2 people struggle with the same issue does inspire more confidence in the validity of that problem. But that doesn’t mean you should discount issues that only one participant experienced. (Statistically, there’s not much difference between 1-of-5 vs. 2-of-5 in terms of confidence intervals.) Instead, gather more information to better explain the context of that single instance.
Analyzing Findings from A Single User
When only a single test participant experiences a problem, first consider the following:
- How much would it cost to fix it? If there’s an easy fix, it may be less expensive to just fix it rather than invest time in trying to document the problem, let alone spending time in team meetings debating the issue endlessly. (This option works best when you already plan to test of the next design iteration, where you can check whether the ‘fix’ introduces any new problems.)
- How serious was the problem for that one person? Serious issues are typically those that prevent user from completing a task, cause great frustration, or relate directly to key design goals. If any of these are true, but it’s not an easy problem to fix, then further investigation is warranted before committing to a design change.
Gathering more information about single-participant usability findings does not necessarily require more user testing. One starting point to consider is competitive analysis: if a review of competitor designs reveals that most do not have this problem, then fixing it may be important in order to meet user expectations.
Check Independent Data Sources to Estimate Frequency
Existing data sources such as usage analytics or support requests are excellent for estimating the frequency and potential impact of a problem.
For example, review support requests and compare how many of them relate to the ‘outlier’ issue, compared to other issues observed in testing. This information can provide a relative estimate of the real prevalence.
If you don’t have specific data about how many users encounter a problem, you may be able to determine what proportion of users interact with that feature and how valuable those users are to the organization’s goals. For example, imagine one participant in a study misunderstood how a sorting feature worked, and because of that she failed to find a product. If you have analytics with event tracking set up for that sort function, a quick check of that data can reveal what percentage of real customers use the sort function. (Since not all users will experience the problem, the percentage of people affected will be somewhat lower than the percentage who use the feature.)
Compare Test Observations with Usability Theory
Consider the extent to which your single-user usability issue can be explained by existing knowledge about user behavior and what makes user interfaces easy or difficult to use. For example, check the 10 usability heuristics which have proven themselves over decades. Or look at published usability guidelines based on substantial user research with a broad range of other designs. While such broad UX theory doesn’t specifically address your individual design, you can generalize from previous research.
If the issue you observed with a single test participant can be easily explained by existing usability theory, you have a stronger case for believing that it’s a real usability problem. If, on the other hand, the theory predicts that your design ought to be easy and not cause problems, then it’s possible that your unlucky test participant was indeed an outlier and the issue would be unlikely to repeat with any frequency.
Unfortunately, the state of UX theory is such that it can’t conclusively decide the matter one way or the other. Human behavior is so variable, and the usability of a design is further dependent on a myriad of small contextual details, that we can never know with 100% certainty whether something is good or bad. But this doesn’t mean that we know nothing about UX. We have substantial amounts of conceptual insights from decades of prior research about what makes user interfaces easy or difficult for certain categories of users and certain types of tasks, and you can use this knowledge to interpret individual empirical observations.
Also, relating your test observations to the body of externally validated UX knowledge can be a useful way for you to explain your findings to your team. Make it clear that it’s not just you who think that the design has a problem, but that many others have found similar problems in testing other designs. And put a name to the problem. These tactics help you rise about the “tiny sample” objection.
Conduct Quantitative Usability Testing
For potentially serious problems that cannot be placed into context with existing data sources, testing with just 5 users may truly not be adequate to confidently recommend a course of action. In this case a quantitative usability study can provide more confidence in what proportion of users are likely to encounter a problem. If the problem can be tested in an unmoderated remote test, collecting data from 20 users needed for an accurate quantitative study can be reasonably affordable.
Quantitative research may also be worthwhile if an organization has deeply rooted skepticism about qualitative methods with small groups. Measuring the incidence of qualitative findings with a larger sample of users can prove the reliability of qualitative methods within the specific organization’s designs and users. To capitalize on the investment in qualitative research, thoroughly document and publicize the project within the organization, with particular attention to the correspondence between qualitative and quantitative findings.
Conclusion
It’s pointless to discover design problems if the team doesn’t believe the findings and therefore doesn’t do anything to fix the issues. Effectively addressing doubts about usability test findings is an essential part of user research, and should be incorporated into every research plan, from start to finish.
Share this article: