Usability testing in the field is an effective and quick way to learn about users and their context of use. Like other types of field research, it typically takes place where users live, play, or work. The following lessons learned from our own studies can help you avoid common problems.

  1. Do a pilot study session in order to debug your materials and understand how much you might get done during the allotted time.
  2. In the likely case that you have too many questions or tasks and not enough time, either prioritize and make some of the last ones optional or plan how to best rotate questions and tasks among participants so you get good coverage of your research issues.
  3. Consider making an editable script for each session, so that you can take notes into the script. A tablet with an ink app can be handy, or bring a superlight laptop. Use a checklist for important things to take with you.
  4. Your questions should evolve or change over the sessions as you learn. Prepare any stakeholders and observers for this process and explain why in this situation (unlike for other types of research such as surveys or field studies) consistency is not essential. (These are decidedly not measurement studies.)
  5. The onsite project manager may be able to arrange incentives and a host gift for you. This help can be extremely useful when your research takes place in a different culture or country than your own.
  6. Monitor recruiting closely to make sure you aren’t getting all superusers, trainers, administrators, or support-escalation people when you actually need normal users. Don’t let well-meaning helpers stack the deck with “good” (but unrepresentative) or expert participants in a misguided attempt to ensure that people perform well in the study.
  7. Recruit diverse participants in terms of ethnicity, role, gender, and experience with the task domain and the system. Try to include at least one person whose native language is not the language of the interface.
  8. Everyone will want to watch. Limit the number of observers per session. Don’t overwhelm your participants by crowding them out of their cubes. If needed, schedule the observers too. The more the better, but make sure you are running the show so they won’t prevent you from getting the data you need.
  9. Weigh locations carefully. Think about whether you need to conduct research sessions in each person’s normal workspace or to move it to another venue, such as a conference room — for example to get enough privacy and quiet. Sometimes a noisy environment is essential in order to recreate all the distractions that people will normally have to deal with when using the system.
  10. If you are able to pay the incentive at the time of the research session, the consent form could also serve as your signed receipt. That way you can pay cash, so that no one will worry whether you’ll really send them a check in the mail or not.
  11. Observers and stakeholders will want to talk to the participants. Provide guidelines for observers, including when and how other people can ask questions. You may need to reword some questions before they get asked in order to remove biases. Be ready to alert people who cause problems through a prearranged signaling system. (Passing notes works well for communicating with observers during sessions.) Don’t let your research be derailed by other people if at all possible. Reserve 5 minutes at the end of the session for observer questions. Sometimes a quiet discussion after a session is necessary in order to regain cooperation. Stakeholders will very likely have other (and better) opportunities to find out more, but you might have only one shot at doing the research. If power struggles appear during the planning phase, build in extra participant-question time for the stakeholders at the end of each session and extra debriefing time.
  12. When conducting research at a business location, reserve a conference room or another private area, if possible, for the researchers and observers to occupy when not in sessions. A whiteboard and a projector might come in handy too. Make snacks and drinks available. Ensure that research participants can’t overhear the team talking.
  13. Encourage and welcome observers, especially if they are usability research skeptics. Stakeholders who observe research studies can become your biggest advocates for change later, and it saves a lot of explaining time to have them onsite. Having a few extra people along can also be quite handy when problems crop up and you need people working in parallel with you, running errands, or intervening when political or emotional issues intrude.

During the Study

  1. Make sketches. Consider sketching notes and ideas on copies of the user interface screens (and even taking environmental photos).
  2. Keep separate copies of the original images and documents, so you can have as many of them as you wish to annotate.
  3. Date documents for version control.
  4. Number participants and their documents so you won’t be attaching names to data.
  5. Take good notes, even if you are allowed to make recordings. Recordings take just as long to listen to (or longer) than the original session, and recordings sometimes fail. Capturing observations and insights in real time can be crucial.
  6. Don’t rely on people to remember to send you promised material after the session. Get permission for someone to email them one reminder if needed.
  7. Pay attention to everything in the environment.
  8. Don’t rely on your memory for anything. Note your questions, ideas, insights, to-do items, and concerns as they arise.
  9. Debrief observers and any onsite research team after each session.
  10. Make debriefing notes so they can become the source for preliminary top findings.

After the Study

  1. If you have recordings that need to be shared with stakeholders, add data-confidentiality instructions and warnings at the beginning of each video or audio file. Release recordings only to a responsible person who fully understands the need to keep research participant data safe and anonymous and the need to destroy raw data and personally identifiable information as soon as it’s no longer needed.
  2. Set expectations for when you’ll deliver results.
  3. Compile and share preliminary top findings as soon as possible, while everything is still fresh in your mind.
  4. Thank everyone who helped make the research effort successful. If allowed, bring a business gift for the host(s), such as a UX book or great office supply item (or make plans to go out for a meal together).

Conclusion

When preparing for field research projects, collaborate with stakeholders to make a research plan, ensure you take the equipment and supplies you need, review these tips, and then relax and have a great usability study.

We can teach your team a full-day course on how to do ethnographic field research. (Also available as a 2-day course, for further depth.)