In an effort to save time and money, some companies use their own staff as user-testing participants. Although this method is convenient and cheap, for most situations it is inappropriate and leads to incorrect results that cannot be trusted to reflect your real audience’s needs and behaviors.

Why Some Companies Recruit Employees for User Testing

Here are a few reasons why it is tempting to engage in this practice:

  • Recruiting convenience: Internal participants are more easily accessible than external users and also more willing to volunteer their time in order to benefit their company. You won’t need a recruiter to find participants — you likely know where to post an ad for a study and you can trust the responders to be responsible and show up for the planned session. After all, their reputation is at stake. Plus, you probably have a myriad of internal tools that you can use to communicate with coworkers.
  • Lower Cost: Companies have different rules about paying internal employees, but often the compensation process is simple and may not even involve money. (Cafeteria vouchers seem popular.)
  • Confidentiality: In some situations, conducting a usability test on an unreleased product creates confidentiality issues, especially for revolutionary new products that you don’t want anybody to know that you’re even working on. Employees have already signed nondisclosure agreements and are less likely to breach them than external participants.
  • Attracting attention: Inviting employees to be usability-test participants could increase interest in the project. Those test participants that become familiar with the process can advocate for your work and possibly lead to more UX support.

Why You Should not Recruit Employees

Asking employees for feedback can affect both how sound your data will be and how the employees might feel about the company. Your data may reflect employees’ biases instead of real-user needs and behaviors:

  • Employees have more prior knowledge about the company than external participants. They know the company’s jargon and its business goals better than nonemployees, and can use that information in a user test to perform better — or for sure differently — than an outsider would do. You won’t observe authentic behavior, which is the main goal of user research.
  • Study participants may have prior knowledge about the project: they may have been directly or indirectly exposed to the project and may have already defined attitudes toward it.
  • Employees have higher motivation to use your product than many outside prospects might have. Employees are likely to think that whatever your company does is meaningful, whereas outsiders might dismiss your company — or even your entire industry — and have very little motivation to engage with your design.
  • Employees’ perspectives of the company are clouded by their opinions on the best course of action. They might make judgments about how feasible and desirable your project is, and how much time, and how many resources it would take to implement any changes suggested by your research. The feedback obtained from the study will likely reflect these opinions.
  • Employees might feel pressure to complete the task; they may spend extra time on each task because they are invested in the welfare of the company.
  • Internal participants may directly or indirectly know people involved in the study. Their feelings for these team members may influence their behavior during the study. Even when participants don’t personally know the team, they might feel a strong connection to it and fear disappointing the designers or insulting their work.
  • An employee’s relationship and history with the company can influence the results of the study. Employees may have a strong positive or negative view of the company that may bias their opinion of the product.
  • Participants may feel uncomfortable being watched by their coworkers or even feel the need to impress high-level executives observing the study.

The last 30 years’ experience with usability studies has conclusively demonstrated the methodological benefits from testing with customers instead of employees. A recent study conducted by Yahoo! confirmed our recommendations: its results indicated that employees are more critical of competitor products and spend less time on competitor sites when compared with outside usability-test participants. These behaviors are consistent with brand loyalty: employees rated the company’s own site 2% higher than a competing site, whereas outside users rated the company’s site 15% lower than the competing site. The outsiders were 18% slower than the employees in performing tasks on the company’s site. Thus, both preference and performance data were skewed, and the research would have been misleading if conducted purely with employees.

All these reasons point to why studies with employee participants will lead to inconclusive results and biased data. The irony is that, in spite of their poor reliability, results from such studies may actuality enjoy higher credibility in the company, because they reflect views of people that the company already trusts and perceives as “its own.” Don’t fall into that trap: by taking employees studies ad litteram and acting upon their findings, you risk wasting a lot of time and money without fulfilling the needs of your actual audience.

When Should You Test with Employees?

There are a few cases where it’s appropriate to recruit internal participants for a usability study:

  • If you are testing an intranet or an application developed for your own employees, then you can legitimately use your employees as test users, because they are the exact target audience. Even in this situation, you want to minimize biases, so find people that are not involved or familiar with the project and minimize discomfort by asking that all direct managers or high-level executives observe in a different room.
  • If you want to pilot a user test, recruiting internally is a great method that can point out weaknesses in your test plan or tasks. If you invite observers or stakeholders for the pilot session, be clear about the scope of pilot testing: to give you feedback about the test structure (to improve on the subsequent test sessions with real users), and not to draw conclusions about the interface.
  • If your company is resistant to provide UX support, this method could be a skunkworks-style technique suited for the transition from UX maturity level 2 to level 3, to demonstrate the need for usability testing. In this case, it is better to recruit people with little existing knowledge of the company history such as interns or new hires. Human-resources people or support staff are preferable to developers, researchers, designers, or marketers who already may have a lot of knowledge about interfaces. Conduct the study to highlight challenges that could be resolved through usability testing with a few target users.

Recruiting employees as test participants is not a replacement for recruiting externally. Limit its use to these few scenarios because it can prevent your organization from recognizing the benefits of recruiting externally. When it comes to usability testing, there is nothing that can truly compare to recruiting real users.

(Learn how to do your own user testing with sound methdology in our full-day training course Usability Testing.)

References

Joanne Locascio, Rushil Khurana, Yan He, and Jofish Kaye. 2016. Utilizing Employees as Usability Participants: Exploring When and When Not to Leverage Your Coworkers. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 4533-4537. DOI: http://dx.doi.org/10.1145/2858036.2858047