Consider the following scenario: you're advising the project manager of an upcoming design project (that hasn't started yet) and are told that there's time and budget for exactly one usability activity. What should it be? And when should this one activity be scheduled in the project plan?

Take a minute to consider your answer before reading mine. What would you do in this liberating — yet frustrating — situation? You have the choice between all the usability methods in the world, but you are limited to doing only one thing.

(For our purposes here, you can't cheat by saying that the "one thing" is a prolonged iterative study with many phases even though that would be better; I really want a single, focused usability step.)

Now I'll present my two preferred solutions and then discuss the tradeoffs between them.

Option 1: Field Study before Starting the Design

Usability professionals always complain that they're called in too late to do much good. And rightfully so: Once the feature set has been decided, you're basically left with lower-value methods to clean up the UI.

In my scenario, you're asked for usability advice before the project has begun; it's thus possible to actually conduct user research up front and use the results to determine what should be designed in the first place.

So, you can go to the users' office, home, the factory floor, the hospital, or wherever people do the type of activity the project aims to support. Play the proverbial fly on the wall: observe what people do in real life.

While on the field study, remember the key methodology lessons:

As I've always said, users are not designers, so you should take their feature requests with a grain of salt. Instead, watch for opportunities to introduce game-changing features that will improve user performance by an order of magnitude or more. People don't know to ask for these new things, but you can spot prospects for great design when you observe users wasting time on workarounds — or simply giving up on even trying to accomplish desirable tasks.

Option 2: Early User Testing of a Low-Fi Prototype

Alternatively, you can wait until an early design has been created and make a simple paper prototype and test it with a handful of users. You can then revise the design to fix the myriad usability problems the test revealed.

At this early stage, it's still possible to make major changes to the user interface architecture, move things around, redesign features, and even remove features that prove useless or too confusing. You might even be able to introduce a few new system features, since you're testing before anything has been coded.

The ability to redesign the functionality and not just the surface UI is one of the biggest reasons to use paper prototyping.

Another reason is that it's 100 times more expensive to make a change after a system has been fully implemented than to make the same change early in the design process when the system is just a set of drawings. Pragmatically, management is much more likely to accept your redesign recommendations if you make them while change is cheap.

If you wait until everything is done and then conduct a usability study to "validate the design," it'll be too expensive to change anything substantial — and you'll be left with minor fixes, like making icons easier to recognize.

Because user testing shows you how users think and behave, the findings will also help your team as they progress to finish the product. Ideally, it's best to get even more findings by running additional rounds of user testing on these more-refined designs, but let's stick to the rules of this exercise and limit ourselves to a single test.

Risk Management Decides Choice of Method

Which of the two options should be your one shot at improving the hypothetical project's user experience? Let's start by looking at the gains you can expect from each choice:

The choice seems quite clear; after all, a 1,000% gain is much more profitable than a 38% gain. So, even if the field study is somewhat more expensive than the simple paper prototype test, it seems like you should pick Option 1.

Not so fast. Read the above bullets more closely: field research has only the potential to discover a hugely profitable radical innovation. It's a much riskier option.

I do believe that field studies are typically more valuable than a single round of user testing. But that's only true if you can trust the design team to deliver a high-quality user interface without major usability problems.

Most companies don't really know much about usability, and teams routinely create designs littered with UX disasters that stump users and/or drive them away from the site.

If the final user interface is terrible and makes the features virtually impossible to use, it won't matter whether the field study inspired the team to invent awesome features. If the target audience can't use something, it might as well not exist.

This is why I ultimately come down on the side of recommending one round of user testing as the safest approach. With user testing, you're sure to get a product that users will actually use. Without it, you might still have a success — but you're more likely to ship a dud.

Now, if you know that your project organization has a high UX methodology maturity, then I might reverse my recommendation in favor of a field study. In this ideal case, the team will have already conducted intensive user research on past projects and internalized the usability lessons. Also, your company will have well-documented UI standards and design patterns to draw upon.

In the real world, few companies are advanced enough to safely ship a design without user testing. More than 90% of teams would be better served with the safer (if less exciting) option of user testing instead of field studies, if they can do only one thing.

We have one last clue to estimate organizational maturity: the fact that we were limited to a single usability activity. Companies that truly believe in usability will typically plan for a sequence of usability activities throughout the project that includes both iterative design and several other user-centered design methods. Different types of user research supplement each other and create more value than repeatedly doing the same one thing.

Of course, there are cases where even the most customer-focused company needs a fast design without much usability input. But in most cases, having time for only one usability activity is a symptom of fairly limited usability maturity, meaning that the company cannot be trusted to deliver a usable design without having it cleaned up through user testing.