Open-Ended vs. Closed-Ended Questions in User Research

Introduction

When conducting usability studies or field studies, it’s a great idea to ask lots of open-ended questions. Typically, researchers ask questions before, during, and after research sessions. It’s easy to focus on what you want to know rather than on how you ask, but the way you ask questions matters a lot in terms of what and how much you can discover. You can learn unexpected and important things with this easy technique.

Definition

Open-ended questions are questions that allow someone to give a free-form answer.

Closed-ended questions can be answered with “Yes” or “No,” or they have a limited set of possible answers (such as: A, B, C, or All of the Above).

Closed-ended questions are often good for surveys, because you get higher response rates when users don’t have to type so much. Also, answers to closed-ended questions can easily be analyzed statistically, which is what you usually want to do with survey data.

However, in one-on-one usability testing, you want to get richer data than what’s provided from simple yes/no answers. If you test with 5 users, it’s not interesting to report that, say, 60% of users answered “yes” to a certain question. No statistical significance, whatsoever. If you can get users to talk in depth about a question, however, you can absolutely derive valid information from 5 users. Not statistical insights, but qualitative insights.

How to Ask Open-Ended Questions

Why Asking Open-Ended Questions Is Important

The most important benefit of open-ended questions is that they allow you to find more than you anticipate: people may share motivations that you didn’t expect and mention behaviors and concerns that you knew nothing about. When you ask people to explain things to you, they often reveal surprising mental models, problem-solving strategies, hopes, fears, and much more.

Closed-ended questions stop the conversation and eliminate surprises: What you expect is what you get. (Choose your favorite ice cream: vanilla, strawberry, or chocolate.) When you ask closed-ended questions, you may accidentally limit someone’s answers to only the things you believe to be true. Worse, closed-ended questions can bias people into giving a certain response. Answers that you suggest can reveal what you are looking for, so people may be directly or indirectly influenced by the questions. Don’t ask, “Does this make sense?” Ask, “How does this work?” and listen closely to discover how well the design communicates its function. Note users’ word choices, because it might help to use their terms in the interface.

Tips

Start open questions with “how” or with words that begin with “w,” such as “what,” “when,” “where,” “which,” and “who.”

Don’t start questions with “was” (an exception to the “w” tip) or other forms of the verbs “to be” and “to do.”

In general, avoid “why” questions, because human nature leads people to make up a rational reason even when they don’t have one. We normally ask “why” only about ratings, to tease out more open-ended feedback. Say “Please tell me more about that,” instead.

Aim to collect stories instead of one- or two-word answers.

Even when you must ask closed-ended questions, you can ask an open-ended question at the end, such as, “What else would you like to say about that?”

Adding Other __________ to a set of multiple-choice answers is also a good way to get open-ended feedback.

When to Ask Open-Ended Questions

• In a screening questionnaire, when recruiting participants for a usability study (for example, “How often do you shop online?”)
• While conducting design research, such as on
  • Which problems to solve
  • What kind of solution to provide
  • Who to design for
• For exploratory studies, such as
  • Qualitative usability testing
  • RITE (paper prototype) design research
  • Interviews and other field studies
  • Diary studies
  • Persona research
  • Use-case research
  • Task analysis
• During the initial development of a closed-ended survey instrument: To derive the list of response categories for a closed-ended question, you can start by asking a corresponding open-ended question of a smaller number of people.

When To Ask Closed-Ended Questions

• In quantitative usability studies, where you are measuring time on task and error rates, and you need to compare results among users
• In surveys where you expect many (1000+) respondents
• When collecting data that must be measured carefully over time, for example with repeated (identical) research efforts
• When the set of possible answers is strictly limited for some reason
• After you have done enough qualitative research that you have excellent multiple-choice questions that cover most of the cases

Bottom Line

Whenever possible, it’s best to ask open-ended questions so you can find out more than you can anticipate. Test your questions by trying to answer them with yes or no, and rewrite those to find out more about how and what. In some cases, you won’t be able to accommodate free-form or write-in answers, though, and then it is necessary to limit the possibilities.