It’s usability-study day. Your first participant walks in. You brief him, explain the session, and dive into the study. Only to realize that the participant does not understand what you’re asking him to do. It’s not a problem with the site design — it’s a problem with the study itself. You’ve just begun your study and you’re already digging yourself out of a hole, trying to quickly determine how to salvage the session. Now what?

When running qualitative usability studies, we often recruit only a handful of users. Small numbers of users can lend great insights into design and usability strengths and weaknesses. So if users struggle not with your site design, but your study design, it can be difficult to get the necessary results. Session data may need to be thrown away, depending on the severity of the problems.

Pilot testing can go a long way to alleviating such problems. In a pilot test, the usability practitioner (and team, if possible) runs through a session or two in advance of the main, scheduled study. Typically only a small number of sessions are needed to prepare for the full study and make sure everything is in order. The point of the test is to test the study itself, running 1–2 sessions to help ensure that the full study goes as smoothly as possible

Pilot testing is particularly important if you are:

  • New to running a usability test. Better to have your first try with a session that you can throw away if needed.
  • Testing in an unfamiliar subject area. If this is your first test of a site aimed, for example, at rocket scientists or nuclear physicists, and you aren’t an expert in the area, pilot tests can help prepare you. (Also do work ahead of the study to familiarize yourself with the topics and terminology, for instance by meeting with subject-matter experts as well as designers and developers.)
  • Running a remote, unmoderated study. Any time that study instructions need to stand on their own, they need to be tested to try to limit possibilities for misinterpretation. If you are running a diary study, unmoderated online study, or study conducted via email, all communications, from recruiting information to confirmation emails to study instructions to follow-up questions, should be pilot tested. These instructions need to stand on their own, as no one will be there to answer questions or make clarifications if a participant runs into a problem.
  • Running a quantitative study. Larger scale studies are typically done so that statistically significant results can be calculated. In such studies, each session needs to be run the same way. As such, a solid script must be created and tested.
  • Testing a high-visibility project. Even if you’ve worked in usability for years, there may be the occasional high-visibility project that requires some extra care. Maybe the results are going straight to the CEO. Maybe the test is on the company’s premier product. Of course, every usability project is important, but some may be a bit more important than others.
  • Doing a one-shot research project. If you’re doing many rounds of iterative testing, then the damage is limited if you make a few mistakes in the first test. The second study will be better. But if a design project only gets a single dose of user research, you want to get it right the first time, because there won’t be a second time.

Even veteran usability practitioners can benefit from running pilot tests. The longer you work in the field, the better you can get at writing strong tasks and instructions in the first place, but it never hurts to run a test plan past a participant in advance. Does every usability study require a pilot test? No, but it’s extra insurance that the study will run smoothly, leaving the team able to focus on the results, rather than the study itself.

Benefit #1: Dress Rehearsal

Pilot testing is a dress rehearsal for the study. It’s a great dry run to make sure the facilitator and team are prepared for the study. Are the materials printed? Consent forms copied? Payment prepared? Site ready and functional? Checklists can help, but running through the study in a low-pressure setting first is a good way to double check that the team is prepared.

Benefit #2: Test the Tasks

If a user is derailed by a poorly written task, it takes precious time away from studying the interface. The facilitator first needs to realize that the participant didn’t understand the task as it was intended, and then has to spend time redirecting or reinstructing the user. Any on-the-fly changes run the risk of accidentally giving users clues about how to complete an activity, or making participants feel they did something wrong. Tasks that can seem perfectly clear to you and your team can be confusing or misleading to participants. Find and fix issues with tasks before your study to help ensure a smoother test and stronger results.

Benefit #3: Timing

It can be hard to know how much time to schedule for user testing. Will participants complete 3 or 7 tasks during your testing session? Running through the tasks yourself and allowing more time than it took you is a very rough way to estimate task time. Running through the full study with 1 or 2 pilot users can help you better estimate how long users might take, allowing you to prioritize tasks effectively for the real study.

Benefit #4: Data You Can Use (Maybe)

If all goes well, or pretty well, in a pilot session, you’ve gotten a jump on the rest of the study. Run the session as though it’s the real thing, and if there are no major hiccups, the data can be used. Even if one or two tasks go awry, you may still be able to salvage information from the sessions. Don’t be too quick to dismiss any learnings just because it was a pilot test. At the same time, don’t be overeager to use the session as part of your final learnings if the session does not go well or something goes wrong at the beginning that might affect the reliability of the rest of the session (such as leading the user or overexplaining an offering or interface.)

Tips for Pilot Studies

To get the most out of a pilot session, schedule it at least 1 day in advance of the scheduled study, and longer if you are testing instructions for an unmoderated study where you might need to pilot instructions several times before they are solid enough to use. This allows you and your team time to make any necessary changes. (Assume there will be changes, because there always are.)

Recruit participants who match your target profile for pilot studies. This means that any feedback — about the study or about the site being tested— will be more relevant. In a pinch, recruiting someone who doesn’t quite fit the profile is typically better than not running a pilot test at all, but the results from those sessions then would not be applicable in the final study.

Does running a pilot study take extra time? Of course. Materials need to be ready earlier, extra participants need to be recruited, and time needs to be scheduled to run the sessions. The payoff, however, is that the final study will run more smoothly, making it easier to get the results that matter to your team and, ultimately, your product.