In usability testing, we use realistic tasks to reflect what users do on the site. To complete some of these tasks, users may have to provide personal information during testing. This personal information may be as simple as a name or email address, or as complex and potentially sensitive as financial or health information.

Users are often hesitant to give personal information to unknown websites, even in a testing scenario. And you as a moderator need to respect the user’s privacy. Couple this with the fact that one or more moderators and observers may be watching the users’ sessions, and that those sessions may be recorded, and the users’ hesitations can be magnified.

So, what to do?

Consider if It’s Necessary

The need for participants to use their own information varies with each design and with the goal of the testing. In quantitative testing, we rarely ask users to use their own data, as it can make the experience too variable and increase measurement error. The goal of quantitative testing is obtaining measurable results. But in qualitative testing, researchers need to assess their needs and decide on a case-by-case basis whether there is any benefit in using real user data.

If entering personal information is only a means to an end (for example, because the user needs to gain access to a section of the site), but does not determine what information is shown, using fake information will usually suffice. For instance, in a test of a telecommunication site, providing users with an address that qualifies for the services being tested may be more important than using participants’ own information.

However, do not dismiss the need for real data in user testing too quickly. Watching users type their own information can reveal more diverse issues in the interface than if all users enter the same fake information. For example, variations in the users’ names or choice of passwords can inform the design. Users will also be much more aware of any errors in the data or interface if the information used is their own.

Another reason for using real data is to make study participants more likely to relate to the assigned tasks and more involved in the study. In usability testing, we ask users to suspend disbelief and complete specific tasks in a specific order. Fake information can remove them even more from the process and make them less invested in the task. The more realistic the experience for users, the better the chance that they will engage with the design.

Provide Fake, but Realistic Information

If the information isn’t essential to the testing or if you’re running a quantitative user study, provide users with information necessary to proceed through the site tasks. This information should be as realistic as possible.

Don’t try to be funny. Providing a silly name or a celebrity name in place of a real name can be a distraction.

Consider how you provide the information for the user. For instance, in a real purchase, a user needs to look at a credit card in order to enter the credit-card number, expiration date, and a security code that may be on the back of the card. In testing, if this information is presented in a labeled list, you may make it easier to enter data than it would be in reality.

Determine what information can be the same for all users and what information needs to differ. Signing up for a newsletter on a live website won’t work if each user enters the same fake email address, unless you can set up an account on the backend that won’t trigger duplicate-email errors.

Think about what needs to be set up ahead of the study. A fake email address won’t work if users must check that email account and respond in order to activate a registration.

Ask Users for Consent During Recruiting

If participants will have to provide personal information, make sure they agree to it during the recruiting process.The more sensitive the information requested from the users, the more details you should offer at recruiting. For example, if the user is asked to fill in only a first and last name, special notice will likely not be needed. If a user will be asked to provide a financial, health, or family-related information, ask about their willingness to do so while recruiting, with an explanation of why the information is needed, how it will be used, and what you will do to protect the data.

Ask Users to Bring Real Data

In some tests, users may have to enter more information than they can be expected to remember or know. For example, in a test of an application for automobile insurance, a user’s driver-license information may be necessary. In a test of a healthcare application, participants may need to know the names or dosages of their current prescription medications or names of their healthcare providers.

In such cases, make sure to ask users during recruiting if they are willing not only to use their personal data, but also to bring the additional information they’ll need for the study.

Provide a Reasonable Alternative

In some situations, it may be possible to provide a workaround to avoid the need to share information if users are reluctant to do so. In an ecommerce study, we gave participants a budget to purchase an item of their choice.

During recruiting, we asked users if they were willing to use their own credit card, and told them it was not a requirement for participation. At the time of the study, before starting the purchasing task, we asked users again whether they were ok with using their credit card. Some who had previously said no now consented, as they had better understood what the study was about. Others decided against using their own information. For anyone who did not want to use personal data, we provided a gift card to use in the study.

Follow Through on Privacy

If a participant uses personal information in a study, ensure that the information remains private. In highlight videos, blur personal information. In reports, blur user data in screenshots or examples. At the end of the usability session, clear all cookies and cached information, including form data from the browser; do this in front of the participant, explaining what you are doing. If, during the study, users printed anything that contained their information, let them take it with them or shred it at the study location.

Better Results with Real Data

Results from qualitative usability testing will be more robust with realistic user data than with fake data. While interfaces can be tested with dummy information, the level of insight can deepen with users’ real information. Details about the success or failure of processes, forms, feedback, and information can be discovered through the variety of information used when participants enter their own data.

 

Update added March 2016. After this article was published, two papers that reach similar conclusions were brought to our attention. While we were not aware of these papers when this article was written and reviewed, it’s nice to have an independent confirmation of some of our recommendations: 

T. Zazelenchuk, K. Sortland, A. Genov, S. Sazegari, M. Keavney. 2008. Using participants' real data in usability testing: lessons learned. In CHI '08 Extended Abstracts on Human Factors in Computing Systems (CHI EA '08). ACM.

A. Genov, M. Keavney, T. Zazelenchuk. 2009. Usability testing with real data. J. Usability Studies 4, 2, 85-92.