Time with Users: Set Personal and Company Goals

How do you become good at user experience? How do you get a glorious UX career? By doing UX. Experience is the way to grow UX expertise. There is not that much theory that’s really important for UX practitioners. It’s a valuable contribution to society to teach people the necessary UX background info, because you can’t learn the rest without building on a solid conceptual foundation. But those foundational 10% of UX skills are as much as you can learn in classrooms. 90% of UX skills sit on top of the foundation and come from on-the-job learning.

However, not all experience is created equal. If you just design away in a vacuum without feedback from users, you won’t learn much. (Also, your products will be horrible, but that’s not my topic today.)

It’s like learning to play the violin: fiddle away all you like, but you only get better through feedback from somebody who knows how the music should really sound. In user experience, only when you find out whether a design actually works for the target audience do you learn how to do better the next time. It’s not a valid criterion whether you personally think that the design is good. User interfaces and human behavior are both so complex that the interaction between the two does not lend itself to pure guesses based on personal opinion. You need data to learn what works, what doesn’t work, and what users really need. Can’t guess. Don’t do it. You can’t learn UX from within your own echo chamber.

User Exposure as a Teaching Tool

The way to learn is simple: watch what users do. Preferably with as many diverse user interfaces as possible: not just your own design, but the competitors’ designs as well.

Lee Duddell (head of UX at WhatUsersDo — a useful multinational remote user testing service) recently predicted that most UX organizations would set user-exposure targets in 2014. Sadly, I don’t think this has happened, though he did hedge the prediction by limiting it to “mature UX organizations” of which there are precious few in the world. However, the point is good: what gets measured gets done. So instead of just saying “I want to learn about user experience” or “our company should have better UX,” a person or company should set a specific goal for how many hours of user observation are expected next year.

At the end of the year, add up what actually happened and compare with the goal.

What Counts as User Exposure?

Any time spent watching another person interacting with a user interface to perform a task. This interaction can be observed live, or time-delayed through a video recording. The observation can be in a usability lab, at a customer site during a field study, or as a remote usability study. These details don’t matter as long as you’re observing real user behavior with a live site or an actual design (including a design prototype).

It’s easier to say what doesn’t count:

• Talking to users or interviewing them (nice, but not exposure to actual use)
• Focus groups (ditto)
• Surveys (ditto)
• Using products yourself (yes, this is real use, but you are not an average user)
• Reading the report from a usability study (a great learning resource, but a summary is not the same as your own direct observation of live user behavior)
• Summative findings from a quantitative user study, such as card sorting, tree testing, and so on (only counts as user exposure if you watched each individual user while they carried out the test)

What if you’re testing two users at the same time? (Sometimes called pair testing, dyad testing, or constructive interaction.) Do two users using one computer count double? I would say no, because you’re only observing one stream of interaction, even though two people are generating it collaboratively. (More research is needed to nail a possible multiplier for pair testing: maybe two users for an hour should count as 1.2 hours of user exposure because you’re getting a richer set of commentary. Pending such research, I would stay with a measure of one hour of exposure per hour of pair testing.)

For sure, being a room monitor during a Multiple-User Simultaneous Test (MUST study) should not count as having observed the number of people in the study. Since you’re moving from one test station to the next during the study period, you’re really only observing one participant a time, even if you’re moving between, say, 12 test stations. So in one hour, you’ve seen one hour’s worth of single-user behavior, even though this is composed of 5 minutes from each of 12 people. Thus, during this session with 12 users in the room, you’ve gained one user-exposure hour, not 12 hours.

User-Exposure Goals

Here are my suggested annual goals for personal user-exposure hours for different classes of personnel. For a company or group you would add up the hours across your staff to come to an appropriate overall user-exposure goal.

As for the last line in the table: high-level executives will obviously only want to spend time watching users in companies that depend strongly on user experience or on users’ online presence. Somebody who runs a coal mine shouldn’t spend time in a usability lab. But the CEO of a computer-game company should spend more time observing gamers than the high target in my table, since a game is all UI.

For each job category I have indicated a range of desired user exposure. Junior staff or people new to a job should aim toward the upper end of the range. In contrast, highly experienced staff can more safely be at the lower end of the range. (The range for UX managers is particularly wide. This is because people get promoted to this role from a variety of prior backgrounds. The less user exposure a newly minted UX manager had in her previous job, the more is needed in the new job.)

The reason experienced people need fewer hours per year is that lifetime user exposure accumulates. You may not remember the exact details of what a particular user did in a study 10 years ago, but you remember the patterns. And the patterns of user behavior are what’s important in terms of analyzing new user interfaces.

Here’s another way of looking at it: The more you already know, the less you will improve from learning even more.  (This is an instance of the power law of practice, that stands true for all learning processes. More about it in our class on the Human Mind and Usability.) If you’re a newbie with only 100 hours of user exposure under the belt, 100 more hours will double your lifetime exposure. But if you have accumulated 1,000 user-exposure hours in your career, another 100 hours will only be 10% more. And your accumulated insight will increase even less (maybe by 5%) because there will be substantial overlap between the new users’ behaviors and those behaviors you had seen before.

We have empirical evidence to support the notion that user-experience professionals improve less and less as their accumulated prior experience increases. Salary surveys show that the average increase in UX professionals’ real salary is as follows:

• First 5 years of UX career (years 1–5): 8% per year
• Next 5 years (years 6–10): 4% per year
• After 10 years’ UX experience: 2% per year

A company would quickly go bankrupt if it paid employees more than they produce. So we can reasonably infer that the annual improvement in a UX professional’s skills must be slightly bigger than the stated percentages for compensation raises. Accordingly the extent to which somebody gets better at user experience is high in the beginning and then tapers off. After 10 years people still get better at UX, but only by about 2% per year. This is an empirical fact, and my explanation for this data is that any additional experience adds less and less to the person’s cumulative UX knowledge.

Thus, if you’re still at the early stages of your user experience career, you should set aggressive goals for your annual user exposure: you’ll learn a lot, relative to the fairly little you currently know. (And your income should grow commensurately.) On the other hand, if you’re very senior, additional exposure will add relatively little to your existing store of UX insights. You still need some user exposure to stay current, but not as much as the junior staff needs.