Quick quiz. Which phrasing of a follow-up question will result in more accurate results in your next user research?

  • “I saw you were having difficulty with the navigation. What happened?”
  • “Why did you have difficulty with the navigation?”
  • “What was easy or difficult about getting to the content you wanted?”

The third question will get you the most reliable response, but why? Because each of the first two questions is leading – meaning that it includes or implies the desired answer to the question in the phrasing of the question itself.

Let’s see what is problematic about the first two questions.

Question 1: “I saw you were having difficulty with the navigation. What happened?”

Problem: The interviewer rephrases what was observed, which may not be an accurate representation of the user’s experience. To the interviewer or observer, it may have looked like the respondent was struggling with navigation, but she may have been deciding what information was most important, confused by the task, or exploring various areas of the site. The question also names a user interface element — the navigation — which is a term that users may or may not fully understand, relate to, or normally use.

Question 2: “Why did you have difficulty with the navigation?”

Problem: Again, this question implies the answer and assumes that navigation was the problem. It also puts the blame on the user, rather than on the site. It focuses the question on the user’s actions as opposed to the elements in the site that may have contributed to the user’s actions.

Question 3: “What was easy or difficult about getting to the information you wanted?”

Improvement: This question steers the user to the topic of interest — moving around the site and finding content — without suggesting terms or feelings to the user. The user can say it was simple to move around or difficult, without disagreeing with the interviewer.  Here the interviewer offers a general frame for the topic of the question, rather than suggesting a response.

Much of user research is observational — we watch what users do. But we also listen to what users tell us, and in many instances, we request clarification about what they tell us. We ask follow up questions after tasks. We may prompt users to share more information in the moment. We start a session by asking for or confirming some basic information about users.

Honest, unbiased participant feedback is critical for user research. When we ask questions, we want to learn more about the user’s actions. Why was this piece of content clear? Why did an interface element cause difficulties? Leading questions are a problem because they interject the answer we want to hear in the question itself. They make it difficult or awkward for the participant to express another opinion. This is particularly true in a usability-study interaction, where often the interviewer is the “authority” in the room and many participants will not want to disagree.

Leading questions result in biased or false answers, as respondents are prone to simply mimic the words of the interviewer. How we word these questions may affect the user response and also may give them extra clues about the interface. We may end up with inaccurate feedback that may or may not truly reflect the user’s experience, mental model, or thought process. Or, even worse, we may alter that user’s behavior for the rest of the session. For example, an unexperienced facilitator asked “What do you think this button does?” in a session and made the user realize that the text she was pointing to was in fact an active link.

Leading questions ultimately rob us of the opportunity to hear an insight we weren’t expecting from the user. The more leading our questions are, the less likely the user will comment in a way that surprises or intrigues us, or makes us think about a problem or solution in a different way. They may be good for “validating” designs, but are definitely bad for testing designs.

Keep in mind that sometimes the best question is not a question at all, but a redirection to help users continue their thoughts. When we do want to ask questions, how can we avoid leading the user?

What Makes a Question Leading

You may be familiar with the idea of leading questions from courtroom dramas, as lawyers call out: “Objection! Leading the witness!” There are many ways in which we can prime a user to simply repeat or confirm our bias, agenda, or assumption. Some practitioners may do it intentionally, trying to get confirmation for their own theory about what does or does not work in a design. However, most of us do it unintentionally. To gather user insights, we should ask open-ended questions — questions designed to elicit explanations from users, rather than single-word yes, no, or multiple-choice answers. How we ask these questions is essential to the value and validity of the feedback we receive. Here are some common traps to avoid:

  • Do not rephrase in our own words.
    • Participant: “I notice this picture here…”
    • Researcher: “You mentioned that the picture was helpful. What about it did you like?”
    • Improvement: “You mentioned the picture…?”
  • Do not suggest an answer.
    • “How well would this save time for you during your workday?”
    • Improvement: “How might this affect your efficiency, if at all?”
  • Do not name an interface element.
    • “The related links on the side of the page here — where would those lead?”
    • Improvement: “This area on the side of the page… [point to area]. What is that?”
  • Do not assume you know what the user is feeling.
    • “When you were struggling with this task, what was happening?”
    • Improvement: “What was easy or difficult about completing that task?” 

Practice Makes Perfect…or at Least Better

Some questions that we ask participants are prepared ahead of time — for example, we may start or end all research sessions with a list of standard questions. In that case, we can easily review those and rewrite them as many times as necessary to make them neutral.

It is much more difficult to make up questions on the fly without impacting what the user says. Everyone makes mistakes in phrasing now and then, and it is difficult to ask a question that is unbiased in every way. However, being aware of the problem is a good way to start fixing it. Watch out for instances where you ask leading questions, or observe others doing so, and see how it impacts the user’s response.