One common question I get when teaching User Interviews, a full-day course at our UX Conference, is how many people do I need to interview? Unfortunately, there isn’t a golden number. In this article, I’ll highlight some factors that will help you decide.
Common Misconceptions
Some UX professionals assume that the recommendation to test with 5 users applies to interview-based studies as well. In fact, for many exploratory-research studies, 5 participants are too few. Others have been taught to recruit 5 people per persona, a rule of thumb to ensure that the sample is representative and large enough. However, this rule can result in many more interviews than necessary — especially when you have 5 or more personas. An experienced interviewer can extract in-depth insights from a much smaller number of interview sessions.
For quantitative research studies, it’s possible to perform sample-size calculations to tell us how many participants to recruit in order to confidently make generalizations to the wider population. But interviews are a qualitative research method. Qualitative research aims to understand the human experience in detail, not to determine how many people have had a given experience or express a particular need. (That’s what a follow-up quantitative study can tell us.) It’s not possible to know exactly how many participants is enough to gain this in-depth understanding.
Saturation
In qualitative research, the sample size is often determined by the point at which saturation is reached.
Definition: Saturation in a qualitative study is a point where themes emerging from the research are fleshed out enough such that conducting more interviews won’t provide new insights that would alter those themes.
(There are slightly different interpretations of saturation depending on the school of thought, but that’s beyond the scope of this article.)
After a certain number of interviews, there are diminishing returns — little new information is learned about the topic of study. This is the point where saturation is reached.
The same phenomenon of saturation occurs when recruiting users for a usability test. When modeling the usability issues uncovered from 11 studies, Jakob Nielsen and Tom Landauer found that testing with 5 people uncovered 85% of the usability issues in the interface being tested, and with each additional new participant added to the sample, fewer and fewer new usability issues were noted.
However, in an interview-based study, we’re looking to understand people’s experiences and needs rather than discover issues in an interface. Thus, because there’s more variability in the kinds of information we’re aiming to collect, the point of saturation is often higher than for user tests. Thus, 5 interviews are often not enough.
Unfortunately, there isn’t consensus amongst academic researchers on how many interview participants are needed to reach saturation. For example, Mark Mason analyzed abstracts of over 2,000 PhD theses that utilized qualitative interviews and found that the sample size ranged from as little as 1 to as many as 95. Although the median sample was 31, the standard deviation was large: 18.7. Over the years, researchers have offered different sample-size recommendations based on their own experience. These recommendations range from as little as 5 to 50. Marketing researchers Abbie Griffin and John Hauser analyzed and modeled the number of customer needs uncovered from their interviews on the topic of portable food-carrying devices and estimated that 20–30 interviews achieve saturation by uncovering 90–95% of all customer needs. Researchers Greg Guest, Arwen Bunce, and Laura Johnson conducted 60 interviews with women in Africa on the topic of sexual health and performed a thematic analysis after sets of 6 interviews. They then analyzed the number of new codes and code alterations after each round of analysis. The researchers observed that, of the 36 high-frequency codes (that led to the development of themes) that were added, 34 were added after 6 transcripts and 35 were added after 12 transcripts. Additionally, after analyzing 12 transcripts, the codes created remained complete and stable, despite subsequent interviews. These findings led them to conclude that saturation had occurred by the time they had conducted and analyzed 12 interviews, but they acknowledged that if they were interested in high-level overarching themes, a sample of 6 would have been sufficient for their project. (If this was a UX project, we would stop after 6 interviews for practical reasons. Although we may miss one theme and some further detail, the time taken to conduct and code 6 additional interviews could be better spent on researching other pressing problems.)
Two Large Factors Affecting Saturation
How many participants you need to interview to achieve saturation depends on:
- The breadth and scope of your research goals
- The diversity of the study population
A research project that is very exploratory and targets a diverse study population will require more interviews to reach saturation than a study with a small-scope research goal and a homogeneous study population. For example, a study looking at the experience of the general population accessing healthcare might require a sample of 20–30 (or more) people to see saturation. This is because the population is very varied (from young to old and healthy to sick) and the scope of the research (the experience of accessing healthcare) is quite broad. On the other hand, a study looking at the experience of patients with Type II Diabetes undergoing treatment could require as few as 5 interviews since it has a smaller scope (receiving treatment for a specific disease) and a more-homogeneous study population (each participant suffers from the same disease and will receive the same or very similar treatment).
Of course, there are some additional factors that can affect saturation. These include:
- How experienced the interviewer is: Experienced interviewers can extract more insights by asking well-designed probing and followup questions and can uncover more themes when analyzing the data.
- How much expertise recruited participants have: People with a lot of experience in a particular domain will be able to share more information about it.
- How structured the interviews are: If interviews are unstructured, then the same topics or questions may not always be covered in each interview, making saturation harder to achieve with smaller samples. If the interviews are semistructured and the same few open-ended questions are asked of each interview participant, it’s more likely that overlaps will occur and saturation will be reached early on.
Considering these factors can give you an idea of your sample size.
Start Small and Analyze as You Go
Since it’s hard to know in advance when saturation will be reached and since discoveries are often too short, it’s best to start with a small representative sample (say 5-6), and analyze your interviews as you go. If you’re still learning new things and generating new codes, you can recruit a few more participants until you reach the point where your themes are complete and little new insight is being discovered with each new interview.
Make sure you recruit people who are your target audience and put an emphasis on relevant characteristics that could affect your research questions. For example, if you’re researching the end-to-end experience of booking vacations online, relevant characteristics could include income, frequency of travel, and age. These characteristics could affect people’s preferences, attitudes, and behaviors.
Of course, it’s good to get a mix of genders, ages, and ethnicities if you can, but since interview-based studies have small samples, it’s unlikely that your sample will match the general population proportionately on all demographic factors – and that’s okay! Some characteristics will be overrepresented in the sample. For example, if you care about inclusive design, you should have people with low digital skills and accessibility needs “overrepresented” in your sample.
Lastly, if you need to set a sample size before starting your interviews (for budgeting or because a stakeholder wants to know), give a range. In this way, you can set expectations that, if you reach saturation early, you can stop.
Summary
How many interviews are enough depends on when you reach saturation, which, in turn, depends on your research goals and the people you’re studying. To avoid doing more interviews than you need, start small and analyze as you go, so you can stop once you’re no longer learning anything new.
References
Abbie Griffin and John R. Hauser. 1993. The Voice of the Customer. Marketing Science, Vol. 12. No. 1.
Greg Guest, Arwen Bunce & Laura Johnson. 2006. How many interviews are enough? An experiment with data saturation and variability. Field Methods, Vol. 18, No. 1, 59-82.
Mark Mason. 2010. Sample Size and Saturation in PhD Studies Using Qualitative Interviews. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, Vol. 11, No. 3. https://doi.org/10.17169/fqs-11.3.1428
Nielsen, Jakob, and Landauer, Thomas K.: "A mathematical model of the finding of usability problems," Proceedings of ACM INTERCHI'93 Conference (Amsterdam, The Netherlands, 24-29 April 1993), pp. 206-213.
Share this article: