From Research Goals to Usability-Testing Scenarios: A 7-Step Method

For some usability studies, there may be an obvious and limited set of interface elements that should be tested. In many cases, however, the system is complex, new to the researcher, or the list of candidate features for testing is long. This article shows a 7-step method that helps everyone focus on the most important elements to test first, while also prioritizing everything of concern for later research.

The 7 Steps

1. Determine the most important user tasks.
2. Discover which system aspects are of most concern.
3. Group items from 1 & 2, then sort issues by importance to users and organization.
4. For each top issue, condense the information into a problem statement.
5. For each problem statement, list research goals.
6. For each research goal, list participant activities and behaviors.
7. For each group of goals, write user scenarios.

Our checklist for planning usability studies describes the larger process.

It’s important to involve stakeholders in these planning steps. After conducting user studies, researchers can sometimes encounter resistance to the findings: someone says the researcher recruited the wrong participants or tested the wrong elements. Going through part of the test-planning process as a team ensures that everyone understands and supports the research goals, priorities, and methods.

If you have only limited access to your stakeholders, you can do steps 1 and 2 in advance, steps 3–5 can be done in a one-day workshop with the stakeholders, and then the UX researcher or UX team can do steps 6 and 7 alone.

For each product, you might do this end-to-end process only once, because the process generates important mutual understanding, many useful research issues, and potential metrics. New features and issues that come up from time to time can be easily added into the artifacts resulting from this one-time process.

1. Determine the Most Important User Tasks

What do people need to do? Make a list of the tasks that users must be able to accomplish with the system.

Normally, every organization has a list of the top user tasks, because this list is instrumental not only in evaluating the usability of your website, device, or application, but also in tracking improvement over time. Ideally, you should use a systematic way of compiling the set of top tasks, by having users rank all possible tasks that the system can support.

But even if you haven’t yet formally engaged in this exercise, you can still determine the top tasks by looking at traffic data from website analytics, search-log queries, survey data, a competitor analysis, or other types of user data. And sometimes the important tasks are well-known or obvious. For example, on an ecommerce website, one of the most important tasks is buying a product. This task is composed of a number of subtasks:

• Top task: Buy a product
  • Look for a desired product.
  • Compare similar products and decide which one to buy.
  • Put product in the shopping cart.
  • Complete the checkout process.
  • Be sure the purchase is complete.
  • Understand when the product should arrive.

Other important tasks on an ecommerce website likely include: return a product, ask customer service for information, and look up a previous purchase.

2. Discover Which System Aspects Are of Most Concern

What are stakeholders most worried about? Often organizations have concerns that motivate them to seek usability testing. Examples include making sure users understand new features, reducing training and support costs, or discovering why people do (or don’t do) something. Look for issues that make users unhappy, problems that cost the organization money, and tasks that take too long to accomplish.

If stakeholders are not nailing these lists to your door already, you can discover top concerns and burning questions by interviewing members of product teams, customer-support staff, sales people, social media representatives, documentation specialists, and trainers. Other evidence may exist, such as search logs, FAQs, journey maps, and materials made by support and training staff that point to issues that might be useful to explore in testing. Doing a design review of the system is another good way to find issues that would be useful to test.

3. Group Items from 1 & 2, then Sort Issues by Importance to Users and Organization

Group related user tasks, stakeholder concerns, and questions. Sticky notes, a whiteboard, spreadsheet, or card-sorting system can help with this process. Your list of grouped issues will probably be too much for one research study, but it will be useful over time, so keep everything. It’s important to include user tasks, because stakeholder concerns don’t always encompass what users want and need to do; yet top tasks must be usable and often need testing.

If possible, prioritize issues as a team, with representative stakeholders.

Name each group of related issues, then put their names in a spreadsheet to sort and prioritize. You can use various methods to decide priorities; here’s a simple example that takes into account both user and business goals.

A spreadsheet and a point system can help with ranking: Sort the named issues from most important to least important, considering both the concerns of users and the costs and benefits to the organization.

Rank issues by importance to both organization and users. When there are many stakeholders or many issues, consider voting with tokens (sticky dots or coins, for example) or using a survey platform’s ranking system.

The objects of this step are to:

• Capture everything of importance
• Prioritize the list of issues quickly
• Create consensus

Don’t obsess over the method of scoring. List the top issues in a spreadsheet, with one column for importance to users and another column for importance to the organization. If the number of issues is large, rank only the most-important 10 or 15 issues in the table. Add up the user and organization numbers for each issue, and then sort the rows by total score.

Together, choose the issues to focus on for the next usability study. Keep in mind that although the total scores are important for prioritization and consensus, sometimes a test should include issues that score 0 for users but high for the organization — for example, expensive problems or new features.

Tips: If you require more accuracy — for example when the list of tasks is very large, the frequency of testing is very low, or changing the product is very expensive — you could survey users and stakeholders separately in order to discover and rank top tasks and concerns. (See our 2 min. video about ranking techniques for UX projects.)

4. For Each Top Issue, Condense the Information into a Problem Statement

Problem statements help everyone focus on what’s important and help you tie research goals to issues everyone wants to solve. Maintaining a list of known problems helps justify and get buy-in for research activities. Problem statements also suggest metrics you can measure to track improvement over time. Sometimes these statements will also include questions. Related issues can be combined into one problem statement, so you may end up with fewer problem statements than top issues. If that is the case, you will need to prioritize the problem statements — using either the issue priorities determined in step 3 or some other method.

Encourage stakeholders to participate in this step as well. It’s much easier to get people to accept and act on the results of a user study when they have been part of the process.

• Explain that the purpose of the meeting is to help decide what is most important to test first, and how.
• Share the detailed list of issues from step 3 with everyone.
• Write the issue names on a big whiteboard (or share a document for writing text together, if your team is distributed), leaving blank space under each for the problem statements.
• Combine related issues into problem statements; for example:
  
  Problem: People call customer service because they can’t find instructions on how to set up their networked lighting system or because they don’t understand the instructions. Calls are expensive for the company, and customers are angry when they call.
   
• Prioritize problem statements for testing, either by combining issue scores (as in step 3) or by voting.

5. For Each Problem Statement, List Research Goals

Problem statements suggest goals for the study. For example, given the problem statement above, here are some corresponding study goals:

• Discover where in the process people need instructions.
• Learn why people can’t find the instructions.
• Determine the best locations to put the instructions.
• Find instructions that need improvement.
• Find opportunities to improve the product in order to minimize instructions.

Some research goals can’t be addressed with usability testing. In those cases, note which methods would be better to use (for example, a survey or analytics) and set those goals aside.

6. For Each Research Goal, List Participant Activities and Behaviors

List the user activities and behaviors that you need to observe in order to address each of the research goals.

Research goals and problem statements can be quite abstract and high level. In order to tackle them in usability testing, you will need to observe users attempting to complete activities that are related to these goals and problems. This step helps you identify (1) types of user activities that are related to your research goals; (2) user behaviors that signal success or failure for a particular activity; (3) possible steps and problem-solving strategies that people could use in doing the activity.

For example, for the goal “Discover where in the process people need instructions,” some likely activities for the process would be:

1. Unbox the lighting system
2. Assemble the lighting unit
3. Connect the unit to the network

Some possible participant behaviors that signal success or failure might include:

• Asking questions (note their exact wording)
• Getting stuck for more than a minute
• Wanting to contact someone for help (note who and how)

Some problem-solving strategies people may use to complete the activity include:

• Looking for instructions in or on the box
• Looking for instructions on the website (note search queries and navigation paths)

After you have done this breakdown for each goal, group goals that have all or many of the same activities and behavioral observations so you can try to write one scenario that covers several goals.

In this example, it’s likely that the listed activities and behaviors also cover the goals: “learn why people can’t find the instructions,” “determine the best locations to put the instructions,” and “find instructions that need improvement.” The final goal for this problem statement, “find opportunities to improve the product in order to minimize instructions” will probably be met by ideas that occur to you and observers during the study.

Planning at this level of detail has four main advantages:

• Focuses scenarios (step 7) on what you hope to observe
• Helps observers focus on what’s important to note during the sessions
• Allows you to decide on the granularity of the success and failure criteria
  For example, instead of having one, monolithic user activity (set up the lighting system), you can consider whether to assign points to (or just track success and failure for) some or all of the subtasks that the participants complete successfully.
• Helps you decide in advance how far you want to let the participants go, how extensive your test environment needs to be, and what you might need to prompt participants for

For example, in this situation, you need to decide whether you want to:

• Listen to participants make calls and find out how well that process works for them
• Watch them create support tickets to discover how difficult filling out the form may be
• Capture the questions they want to send to support (how?)
• Let them look through the support website (for how long?) or just find the support landing page
• Prompt or encourage the participant to do or not do any of these things (what’s the best way to phrase what the facilitator should say?)

7. For Each Group of Goals, Write User Scenarios

Scenarios are the instructions that your test participants read during the study. For our example list of research goals (step 5) and the corresponding activities and behaviors (step 6), here’s a good user scenario:

Please show me how you would set up your new Acme Home Illumination Kit, and explain what you are thinking and doing in detail while you put it together.

With any luck, participants who attempt this scenario will find and read instructions as needed and they will think out loud by way of explaining, so you can understand their difficulties with both the system and instructions.

Because the example situation is quite straightforward, this scenario seems obvious and unworthy of the steps to arrive at it. (However, all scenarios should encapsulate your thinking and planning process into a simple statement that participants can understand quickly.)

Nonetheless, this method can untangle big and complicated systems, tame and educate a room full of stakeholders having competing priorities, generate a prioritized list of research problems to work on over time, help organize the test plan, the facilitator script, and observer study guide, generate usability metrics, and help you think through the logistics and contingencies for your usability sessions.

Tips:

• Some goals might lead to several scenarios. Some scenarios might cover several goals. As you work through this process, think about how you can combine goals into scenarios without making them too long or too complicated.
• Similarly, across problem statements, some duplication will naturally occur when making lists of goals and activities. After you make the lists, consider whether it makes sense to remove some steps or to combine or sequence scenarios in order to reduce repetitive activities or to reduce biasing effects (when doing one scenario teaches people a lot about another scenario). Sometimes it makes sense to watch each person do many searches, but you might need to watch other activities only once or twice for each participant.

Guidelines for Writing Scenarios

We wrote entire articles to help you write effective scenarios and avoid common errors in the specific task instructions. Briefly:

• Focus scenarios on outcomes your users would be motivated by.
• Write scenarios in a way that should cause people to do the things you need to observe.
• Don’t instruct people to operate the user interface.
• Don’t mention labels found in the interface. (An exception might be “search,” when you’re testing search, but consider whether you want to find out if people notice or locate the search tool on their own before you mention it.)
• Use general words your participants should understand, not technical or branded terms.

Next Steps

Once you’ve generated scenarios for your research goals, the next steps usually involve:

• Engage stakeholders in observing the study sessions to help you focus on the most important issues and to build consensus and buy-in for your user research activities.
• Decide what to measure and how to score it.
• Pilot test your session to debug the scenarios and your process, and to determine how many scenarios to attempt per session.
• Include the information generated from these 7 steps in your research plan for all of the scenarios in your study. Document the rest of the information for future research studies.
• Invite stakeholders to your study sessions and encourage them to help you take notes.

Conclusion

This 7-step procedure makes it easy to decide what to test, how to test it, and how to write comprehensive user scenarios. The lists generated at each step help you to prioritize research goals, to get buy-in for your research sessions, to organize your study sessions and materials, and to track important issues over time.