Introduction

"Echo. Boomerang. Columbo." No this is not a bastardized version of the NATO phonetic alphabet, rather it’s a handy way of remembering 3 safe and productive approaches for interrupting or answering users during usability tests and other research studies.

Facilitation Techniques for Handling Test Users’ Questions During Usability Studies

How does an unprepared usability-test facilitator respond when the user asks her a question or offers an indecipherable comment? Usually, not well. Admittedly, facilitating a usability study is not a natural way to interact with other human beings. So it is totally understandable why most of us have trouble facilitating, making classic mistakes such as:

  • Commence nervous chatter or panic and fumble for words, worrying about the negative impact they may be making as they speak.
  • Treat the test session as a conversation rather than an observation.Talking too much, at inappropriate times, or leading the user can affect what he does and says, which can invalidate part or all of the research findings. Interviewing methods are different from observational methods. While interviews can reap fruitful information about people’s desires and impressions, direct observation of live user behavior is an invaluable research method when trying to learn what people actually see, act on, and how they interact.
  • Go the complete opposite direction and stay completely silent, succumbing to fears about interrogating the user. Sitting completely mute is probably better than saying too much, but it's not an advanced facilitation practice as it doesn't enable gathering the most possible information during a study. Probing at the right times, without making mistakes (such as leading the user or asking closed questions which beckon only yes or no answers) can reap rich responses and lead to insights about exactly why a design is or is not useful and usable.

There is no need to panic or blather on when a user asks a question. Instead try the echo, boomerang, or Columbo techniques described here.

Echo

With the echo technique, the facilitator repeats the last phrase or word the user said, while using a slight interrogatory tone. Using the exact word(s) that the participant used ensures that the facilitator does not bias the participant by making a suggestion or describing anything in the interface. Instead, she just parrots and probes in a benign way. Here are two examples of good echo technique:

User: This table is weird, well, hmmm, not sure what, uh…

Facilitator: Not sure what?

Or

Facilitator: Table is weird?

Say these few words while using a tone that makes it clear that the phrase is a question. This will naturally put the user in the mindset of answering the question by elaborating on what he meant by those same words.

Echo Audio Example (Note that I play both the user and the facilitator.)

Boomerang

With the boomerang technique, the facilitator formulates a generic, nonthreatening question that she can use to push the user’s question or comment back to him. Examples of such nonthreatening questions are “What do you think?” or “What would you normally do?” So if a user asks a question such as, "Do I have to register to buy this?" the facilitator should not say, "Erm. I guess so," or whisper, "No, that’s okay." Rather, the facilitator should try the boomerang technique. For example:

User: Do I have to register to buy this?

Facilitator: What do you think?

Or

Facilitator: What would you do if you were at home now?

Or

Facilitator: What would you do if you were really doing this on your own?

The facilitator does not answer the user’s question; rather, she bounces the user's question back to him. The idea is to remind the user that he is to try and work out the issue as he might if he were not in a research environment.

Boomerang Audio Example (Note that I play both the user and the facilitator.)

Columbo

With the Columbo technique, be smart but don't act that way.

Peter Falk immortalized the character of Lieutenant Columbo in the late 1960’s television series in which he caught a myriad of criminals, mainly by enticing them to underestimate his investigative skills. Columbo seemed forgetful and inarticulate, but he was actually perceptive and astute.

While facilitators in a usability study aren’t trying to catch anyone as Columbo was, they are trying to craft tasks and questions in a way that coaxes people into saying what they think and into doing what comes naturally. Facilitators can take a page from Columbo’s handbook and act less like experts, and more like investigators. One way to do this is to ask just part of a question, and trail off, rather than asking a thorough question. This may sound strange, but it achieves the following:

  • Saying fewer words means the facilitator is less likely to teach or sway the user.
  • Forming less complete and perfect questions and pausing can cause the user respond more quickly, so he doesn't have to wait for the facilitator to finish her thought. And some users try to help the facilitator by answering before a question is fully formed.

So if a user asks a question such as, "If I close here will I lose my work?” the facilitator may be tempted to, but should not say, "I am not sure” or “Try it” or “I don’t think so.” Rather, the facilitator should try the Columbo technique. For example:

User: If I close here will I lose my work?

Facilitator: Uhm, you are wondering if [pause] you might [pause.]

User: I am just not really sure if I should pick "close" or "cancel" or "ok." I guess I don't know the difference between these buttons.

Columbo Audio Example (Note that I play both the user and the facilitator.)

A facilitator may also apply the Columbo technique effectively when she wants to initiate asking a post-task question. For example, she may be tempted to but should not ask the user, "Did you see the filters on the left?" Instead she might try this:

Facilitator: I was wondering if you could look at the page [pause] and this section [pause and motion to an area in the interface.]

Deciding Whether to Speak to the User

Don’t mistake any of the tips mentioned in this article as license to interrupt the user any time he makes a sound. Instead, follow these guidelines before talking to the user:

  1. Decide whether what the user said was a real question that you actually need to answer, or it was a rhetorical question, or just thinking aloud.
  2. Determine whether the noise or comment that the user made was indecipherable, or whether it was actually enough to draw a fair conclusion from.
  3. Consider whether you will truly benefit from probing the user further, or whether you have enough information from just observing what he is doing.

Key events which signal an appropriate moment to interrupt a person during a test session occur when he has:

  • offered some comments
  • asked the facilitator a question or is otherwise seeking guidance from the facilitator
  • naturally interrupted his own flow in some way.

Conclusion

When in doubt about whether to talk to the usability test participant, count to 10 silently. Then decide whether you should say something. If yes, try using echo, boomerang, or Columbo techniques for speaking with the user. Remembering these 3 practices can help you learn more from your users, and improve your own test-facilitation skills.

More Facilitation Advice

Many more guidelines for improved test facilitation techniques in the full-day training course about Usability Testing.