WAP Mobile Phones Field Study Findings

In the fall of 2000, we ran a field study of WAP users in London. After a week's experience using WAP, study participants had one resounding conclusion:

• 70% of them said that they would not be using WAP in a year.

For the study, we gave 20 users a WAP phone and asked them to use it for a week and record their impressions in a diary. We also performed traditional usability tests with users at the beginning and end of the field study. We gave half of the users an Ericsson R320s and the other half a Nokia 7110e.

( WAP = Wireless Application Protocol; a way of accessing the Internet through a cellular telephone.)

We ran this study in London because of the advanced state of the United Kingdom's mobile phone market relative to the United States. The U.K.'s WAP services have been under development longer than those in the U.S. and were also more widely deployed at the time of our study.

It is important to notice that users provided their negative review of WAP after a full week of hands-on experience with the technology. It is irrelevant to ask people in a focus group to predict whether they would like something they have not tried, so the only way to get valid data is to let users experience the technology before you ask them for their opinions.

WAP Doesn't Work

Of course, we didn't just collect opinions. We ran timed task-performance studies as well, since observations are the best source of data. We asked users to accomplish simple tasks with their WAP phones, both at the beginning of the week and at the end. Here are some of the findings:

In the above table, the first number indicates the mean number of minutes users needed to perform the task in the beginning of our study and the second number indicates the mean measurement at the end of the study (i.e., after a week's use: people did get better at using these mobile services, as predicted by the power law of learning, but the improved performance was not sufficient to reach a decent level of usability).

As the table shows, our basic conclusion is that WAP usability fails miserably; accomplishing even the simplest of tasks takes much too long to provide any user satisfaction. It simply should not take two minutes to find the current weather forecast or what will be showing on BBC1 at 8 p.m.

I asked a group of Internet experts how long they thought these tasks ought to take (before showing them our data), and most estimated a task time of less than 30 seconds. Considering that WAP users pay for airtime by the minute, one of our users calculated that it would have been cheaper for her to buy a newspaper and throw away everything but the TV listings than to look up that evening's BBC programs on her WAP phone.

Déjà Vu: 1994 All Over Again

Our findings from this WAP usability study in late 2000 bear a striking resemblance to several Web usability studies we conducted in 1994 (the age of Mosaic). It's truly déjà vu: Many of our conclusions are the same as those we reached at the dawn of the Web. Hopefully, mobility's evolution will follow that of the Web: When things got better in subsequent years (especially around 1997), many more users got onto the Web and commercial use exploded.

The usability of current WAP services is severely reduced because of a misguided use of design principles from traditional Web design. This situation is exactly equivalent to Web design problems in 1994, when many sites contained "brochureware" that followed design principles that worked great in print (say, big images) but didn't work in an interactive medium. For example, we came across a WAP design from Excite that used four screens to present two screens' worth of material. They even had a splash screen. Such lavish design may work on the Web if users have a big-screen PC, but on a small-screen device, designers must boil each service down to its essence and show much less information.

Our users often faced unclear labels and menu choices written in special language invented by the WAP designer. NewSpeak was rampant in the Web's infancy, and many sites invented cute vocabulary for their services in a misguided attempt to brand their site with proprietary language. This didn't work. Users want no-brainer design that uses standard terms for standard features. The need for simple language is even stronger in WAP design, because there is no room to explain non-standard terminology with roll-over effects, icons, or captions.

Several WAP services that we tested were unnecessarily hard to use because of a mismatch between their information architecture and the users' tasks. For example, TV listings were organized by television network, meaning that you would have to go to several different parts of the service to find out what was on at 8 p.m. (one screen for BBC1, another screen for BBC2, and so on in an annoyingly slow sequence of screens).

Very precise task analysis will be necessary for WAP services to succeed. Unfortunately, task analysis is a black art as far as most people are concerned and it is the least appreciated part of usability engineering. The traditional Web also suffers from poor task analysis, with many sites structured according to how company management thinks rather than how users typically approach their tasks. Although poor task support is a serious usability problem for a big-screen website, it is a usability catastrophe for a small-screen WAP service. With the big screen, users can see many more alternative options, and thus it is not so critical that designers pick exactly the right ones at each step. For WAP: Be right or be dead.

One WAP usability finding that we have not seen on the Web was a lack of clear differentiation between services. As one of our users noted when comparing the Financial Times and The Guardian: In the real world, you will have trouble finding two more different newspapers. On WAP, however, you can't tell them apart. Websites usually suffer from the opposite problem: They are much too different. With WAP, the service's expressive power is severely reduced because of the need to squeeze everything into extremely short menus and present all content in ultra-short condensed versions. Service providers must cultivate a new appreciation for language and hire copywriters who can develop a distinct voice in a minimum word count. This will be the real way to distinguish WAP services.

Mobile Killer App: Killing Time

Promising mobile Internet services follow a bi-modal distribution with two dramatically contrasting approaches that both work well with users:

• Highly goal-driven services aimed at providing fast answers to specific problems. Examples include: "My flight was canceled; get me a new airline reservation" and "What's the weather?"
• Entertainment-focused services whose sole purpose is killing time. Examples include gossip, games, and sports services. Gossip is particularly suited for WAP because the content can be very brief and still be satisfying.

Mobile services must target users with immediate, context-directed content. General services like shopping are less likely to succeed in the mobile environment. Indeed, in the list of services bookmarked by users, shopping hardly figures at all; sports and entertainment are the two big categories.

Killing time is a perfect application for mobile devices because they are readily available when users are waiting around for something to happen. At the bus stop? Play a short game. In line for something? Read a paragraph of gossip. Stuck in traffic that doesn't move? Check the scores of your favorite teams.

(Similar findings 18 years later: Filling the Silence with Digital Noise.)

Read More: 90-Page Report Available

The full report from the field study can be downloaded (for free).