Intranet Usability Shows Huge Advances

During the few years between our two rounds of intranet testing, the world experienced a dramatic improvement in intranet usability.

Our simplest usability metric is success rate, which measures whether users can complete their tasks with the user interface. In the first study, the average success rate was 74%; in the second study, it was 80%. An increase from 74% to 80% might not seem very big, but that's because the first study's success rate was already fairly high, and thus it was hard to increase it further.

Our public Internet tests have shown dramatically lower success rates than what we've found on intranets. In our book, Prioritizing Web Usability, we present detailed results of a study in which we asked users to perform tasks on the Web without first taking them to any particular website. Such a task tests how people typically use the Web, and it had a success rate of only 60%.

Intranet success rates are typically about 33% higher than the Web for two reasons:

• There is no doubt about where to go. The company's intranet is the company's intranet — users don't have to worry whether they've reached a fraudulent site or a low-credibility source the way they do on the Web. The first problem in using the Web is to find a good website for your current question; intranets don't present this obstacle. (Yes, users have to find their way around their intranet, but that's equivalent to navigating a website, not to finding a website and judging whether it's going to cheat you.)
• Employees experience with their company's intranet continually increases because there's only one site. In contrast, users flitter among websites and don't spend much time on any given site. As a result, people rarely learn the intricacies of an individual website's interface and are thus more easily baffled.

The success rate variability among intranets was substantially smaller in the second study than in the first. The standard deviation relative to the mean was 15% for the first study, but only 8% for the second study. In other words, according to this common variability measure, intranet variability was cut almost in half. This further indicates the benefits of intranet usability guidelines.

Our first study was conducted before any intranet guidelines were available. (Indeed, the purpose of this first research was to identify the first set of intranet usability guidelines.) Without such guidelines, designers had to rely solely on local observations of their own intranet and couldn't benefit from usability findings from other intranets. Given such scant usability knowledge, design decisions were highly variable.

In contrast, we conducted the second study after we'd published general intranet usability guidelines in the report from our first round of research. Working from this broader knowledge base smoothed over the differences in the information available within each company. Of course, there was still some variability left because the resources allocated to intranet design teams varied and different design decisions had different effects.

Overall, however, the more we know about intranet usability, the less intranet design is a matter of guesswork. Hopefully, the publication of our new report edition, with much more detailed findings and additional guidelines, will further reduce the variability in intranet quality.

Intranet User Research

We conducted two rounds of international research into intranet usability, running 28 user tests with employees at 27 companies (we tested one company's intranet twice: once at its U.S. headquarters, and once in another country to collect information about international usability for multinational intranets).

We conducted our tests in the U.S. (20 tests), Europe (6 tests), Asia (1 test), and Canada (1 test). Our first round of research in 2002 focused on traditional lab-based usability testing. The second round of research repeated most of the lab-based test tasks from Study 1, added new tasks to account for recent changes in intranet use, and also included field studies.

In all cases, we conducted the user research on-site at the company or organization whose intranet was being studied, using representative employees as the participants. In the lab-based sessions , we asked users to perform a series of standardized tasks that spanned a range of common intranet use cases. Assigning prepared tasks allowed us to compare usability performance across intranets and to assess how usability had changed during the years between Study 1 and Study 2.

For the field study sessions , we observed representative employees as they went about their normal work. We interfered as little as possible and didn't assign them any specific tasks. During these sessions, people at different companies obviously did very different things with the intranets, making formal comparisons difficult. But field studies are still invaluable; they show us a wider range of behaviors than we see in the laboratory and let us assess the impact of changing contexts and other ethnographic variables.

Increased Knowledge about Intranet Usability

The following table compares the reports with intranet usability guidelines that we published as a result of our two rounds of research:

(*) One company's intranet was tested in two countries to collect additional information about international usability of a multinational intranet. Thus, we have data from 28 tests at 27 organizations. [Update: note that the 3rd edition of intranet research now covers 42 intranets.]

Considering that we only doubled our research, why do we have 5 times more usability findings in the report's second edition? Several reasons:

• In addition to the 28 intranet tests that we can publicly discuss, we've studied many more intranets for our consulting clients. We've also analyzed more than 1,000 intranet designs as part of our series of Intranet Design Annuals. While the specifics of these additional intranets must remain confidential, we draw upon the many general lessons gained from studying them.
• Study 1 was the first project in history to systematically test intranet usability across many intranets with the goal of deriving generalized usability guidelines. Because of the lack of historical precedence, this initial research round was more difficult to plan and perform, and we had to focus on identifying the biggest intranet usability issues. In contrast, we can now build on a much firmer foundation, making it easier for us to identify and document a broader range of issues. Having the conceptual framework in place makes it easier to see things in a research study.
• For Study 2, the only thing we had to do in relation to Study 1's findings was to check whether these issues still constituted usability problems. This left most of the project resources available to identify new issues.
• We added field studies in Study 2. Although less structured than lab testing, field studies give broader insights into users' behavior and contextual needs.
• Intranets now have many more components and features than they did during our first study. With more features to study, there are more usability issues to be found.
• Intranets now have better usability. Users can therefore progress further and attempt more advanced behaviors. All for the good, but as people try to accomplish more, they encounter additional usability issues that were not as prominent in the past when users got stumped by more blatant usability problems.

Improvements in Intranet Usability Metrics

Out of the 18 common intranet tasks we tested, 11 were repeated from Study 1 to Study 2. This fact lets us estimate the change in user productivity across the studies.

The following chart shows the trends for intranets at three usability levels:

• Q1 (the first quartile): the task time that separates the best 25% of intranets from the worst 75%. This number is an estimate of the time needed to perform the tasks on an intranet with good usability.
• Median : the task time at which 50% of intranets are better and 50% are worse. This number is an estimate of the time needed to perform the tasks on an intranet with average usability.
• Q3 (the third quartile): the task time that separates the best 75% of intranets from the worst 25%. This number is an estimate of the time needed to perform the tasks on an intranet with poor usability.

The number of hours an average employee would spend each year performing the 11 common intranet tasks that we tested in both rounds of our user research.

• The upper line (red) shows task times for the worst (slowest) 25% of intranets,
• the middle line (blue) shows task times for an average intranet , and
• the lower line (green) shows task times for the best (fastest) 25% of intranets

Two conclusions are clear from this chart:

• Intranet usability has increased substantially, leading to faster task performance and thus higher employee productivity. The hours people save from faster intranet use is time they have available to do their "real" work. For the median intranet, productivity increased by 44%.
• The most dramatic improvements have come to intranets with poor usability , where productivity increased by 69%. Intranets with good usability have also improved, but not nearly as much as the poor intranets.

Does Intranet Usability Still Pay?

Now that intranets are easier to use, do companies still need to invest in even better intranet usability? One way of considering this question is to look at the cost of the time employees spend using the intranet.

For the 18 common intranet tasks we tested in Study 2, we can compute the cost of the employees' time by multiplying the number of hours per year a user spends doing the tasks with the cost of keeping that user employed. Each company can perform the calculation using its own salary levels and overhead costs, but for the sake of argument, we'll use $30 per hour, which corresponds to the average hourly salary of a white-collar employee in the U.S. loaded with an additional 50% in overhead.

The following table shows the annual costs of the time spent performing our 18 intranet tasks in a company with 10,000 intranet users:

Thus, a company with poor intranet usability would save $3 million per year if it improved its intranet usability to an average level. And a company with average intranet usability would save $2.4 million per year if it improved its intranet to the usability level found in the best 25%.

These numbers only estimate the productivity gains from the common intranet tasks we tested. In addition, most companies have their own special, mission-critical intranet tasks that by definition can't be tested across organizations, but which usually have much more impact on overall employee performance than do the common intranet tasks. Thus, in most companies, the total productivity gains from improving intranet usability will be many times higher than these estimates.

In Study 1, we tested 16 common intranet tasks. Back then, we estimated the following productivity gains from improving intranet usability in a company with 10,000 employees: going from bad to average, $10 M/year; going from average to good, $5 M/year.

Because there was some difference in the tasks tested in Study 1 and Study 2, the productivity savings are not directly comparable, but the overall picture is certainly clear: the productivity gains from improving intranet usability are much smaller now than they used to be.

This should come as no surprise — we've already eradicated the worst mistakes in intranet usability, so we've already collected most of the gains from the proverbial low-hanging fruit.

The downside of the huge intranet usability improvements in recent years is thus that there's not quite the same potential for future gains.

Should we throw in the towel and stabilize the world's intranets at their current level of usability? Not at all.

There are three reasons to advocate continued work on improving intranet usability:

• We estimate productivity gains of $2-3 million per year for common intranet tasks, and several times that amount for company-specific tasks. In comparison, a project to improve intranet usability typically costs less than half a million for the 10,000-employee company we're using as our example. Bigger companies will have bigger expenses, but they'll also realize proportionally bigger productivity gains. A company with 100,000 employees will get 10 times the benefits from intranet improvements, but will probably need only around 4 times as big a budget to realize the gains (that is, $20-30 M return on a $2 M investment).
• Our discussion of "poor intranet usability" looks at the data from the Q3 performance level, representing the cut-off between the best 75% and the worst 25%. But many companies with poor intranet usability are far below this Q3 level, particularly if they've ignored intranet usability until now. The good news is that they still have correspondingly huge potential gains once they start a usability project.
• Our estimate of "good intranet usability" looks purely at current best practices, not at the theoretical optimal design. There's no reason to believe that our Q1 numbers represent the final word in intranet usability. Further improvements can be had if companies continue to push the state of the art in intranet design. Our experiences from the Intranet Design Annuals make us very optimistic on this front, because every year we see that the very best intranets become even better.

Thus, there is every reason to believe that intranet usability projects will continue to show good ROI, even if the improvements won't be as huge as they were for a company's first usability project.

Better, But Not Good

Intranet usability has improved substantially, yes. But is it good enough? No. We started out at an extreme low, with intranets being the impoverished cousins of websites: no company investment in design and usability, pure chaos in navigation and IA. Things are indeed better now — 77% of intranet teams say they get adequate management support. But at most companies, the intranet user experience is still nowhere near what it needs to be to maximize employee productivity.

Full Report With Newer Research

This article reports the findings from our 2nd round of research. Note that the research report on intranet usability has since been updated with newer research findings from additional studies.