It takes 39 hours to usability test a website the first time you try. This time estimate includes planning the test, defining test tasks, recruiting test users, conducting a test with five users, analyzing the results, and writing the report. With experience, Web user tests can be completed in two work days .

In a recent project at the Technical University of Denmark, Rolf Molich and Christian Gram collected data from 50 teams of students who conducted usability tests of commercial websites as part of a user interface design class. The average time spent by each team was 39 hours. Furthermore, the students had sat through 15 hours of lectures on user test methodology. These numbers are an upper estimate of the time investment the first time you decide to user test your site.

With experience it is possible to conduct much more rapid user tests of a site. Good test tasks can be written in one or two hours, recruiting can be outsourced  to a recruiting firm (at a cost of less than $1,000 for five users), the actual test can be done in a day, and the results can be analyzed in a few hours. If you are a member of the design team, then there is no reason to write an extensive report which nobody will read, so reporting can be done in a one-hour meeting supplemented by a summary that takes 2-3 hours to write. In total, a discount usability study takes only two work days once you know what you are doing.

Even though experts can do the work more efficiently (and usually with better results), it is encouraging that utter beginners could complete a full Web usability project in less than one week. This truly proves that "limited budget" and "lack of time" are not valid excuses for inflicting difficult sites on your users.

The students tested nine large Danish sites: seven sites from major Danish corporations as well as the University's own site and the university library's site.

After rating the usability problems for severity , the study found that each site had an average of

  • 11 usability catastrophes
  • 20 serious usability problems
  • 29 cosmetic problems

In this study, a "catastrophe" was defined as a usability problem that prevented the user from completing a task. A "serious" problem was one that slowed down users significantly but did allow them to complete their task. A "cosmetic" problem delayed users slightly or annoyed the users as indicated by their verbal comments.

 

I am not at all surprised by that major commercial sites contain this many usability problems. In fact, Molich and Gram note that the sites probably had even more problems that were not found in the study since the students only tested a small part of each site (though presumably they focused on the most important parts).

Most of the usability problems found by the Danish students were similar to problems we have seen in studies of English-language sites over the last five years. This similarity is due to the fact that the study addressed domestic usability (Danish users accessing Danish sites) and not international usability (Danish users accessing, say, American sites). A few problems related to international usability were observed since some sites contained a combination of Danish and English content. Some sites switched language without warning when an unsuspecting user followed certain links. Multilingual search also caused problems since it was not clear to users whether an English-language search page would include the Danish pages in its search scope. Some other issues in multilingual search were discussed in my August 1996 Alertbox on international usability  but it is striking how little is known about this topic.

In a study of 15 large commercial sites  in the U.S., Jared Spool and colleagues found that users were only successful 42% of the time when asked to find specific information. When asked to rate "overall ease of use", people scored these sites 4.9 on a 1-7 scale (7 best); somewhat better than the neutral rating. This latter result highlights why it is not sufficient to simply ask people whether they like your site: people tend to be polite and give relatively high ratings even when the site is unusable.

These statistics form an interesting baseline for your own usability studies :

  • the average site has 11 usability catastrophes (design elements that prevent users from completing test tasks)
  • on average, users are only able to complete 42% of the test tasks
  • users' average subjective rating of websites is 4.9 on a 1-7 scale

If you did not use any systematic usability engineering methods in the development of your site, your score will probably be along the lines in this list; often worse.

Test Coverage

Web usability problems fall into two categories:

  • Site-level usability : home page; information architecture, navigation, and search; linking strategy; internally vs. externally focused design; overall writing style; page templates, layout, and site-wide design standards; graphical language and commonly used icons
  • Page-level usability : specific issues related to the individual pages: understandability of headlines, links, and explanations; intuitiveness of forms and error messages; inclusion or exclusion of specific information; individual graphics and icons

A usability test with 5 users will typically uncover 80% of the site-level usability problems plus about half of the page-level usability problems on those pages that users happen to visit during the test. The reason for the lower coverage of page-level problems is that different users will visit different pages, so most pages will be tested by less than the 5 users it takes to find 80% of the problems in a design. A test with 2 users typically finds half of the usability problems in a design, so that is my estimate of the proportion of page-level problems found.

Of course, on a large site, most individual pages will not be visited by any of the test users. Thus, the main goal of user testing a site should be to find the site-level problems. You should obviously take note of the page-level problems so that they can be fixed on those pages that the users happened to visit. The most important outcome of finding page-level usability problems is a better understanding of the extent of page-level problems on your site. Also, the specific set of problems can serve as a starting point for developing a list of typical page-level design pitfalls that need special attention in the design of future pages.

It is possible to run specialized user tests of particularly important pages to increase coverage of their page-level usability problems. For example, it is often a good idea to test registration pages and the pages where users actually buy products or download software. It is possible to substantially increase the number of users who are able to complete their transactions.

For most other pages it is necessary to increase page-level usability through other methods like heuristic evaluation  and by training content developers in principles of good Web usability. One good way of increasing the page-level staff's understanding of Web usability is to invite them to observe a few user tests. Even when other people's pages are being tested, one can still learn a lot about how users interact with Web pages from simply observing one or two tests.

The estimates of usability problems found with different numbers of test users comes from a project I did with Tom Landauer. For more information, see my book Usability Engineering  and Landauer's book The Trouble With Computers: Usefulness, Usability, and Productivity .