Usability lab setup for paper prototyping, card sorting, and traditional usability testing.
Paper Prototyping
We used several rounds of usability testing to improve the user interface for Sun's new WWW pages. The initial tests focused on the home page and were done using paper prototyping. We also conducted competitive usability tests where users were observed while browsing WWW pages from other companies.
Usability testing with paper prototypes involves printing out screen designs and having users perform tasks using the printed screens.
For the Initial usability studies done using paper prototyping, we printed out early homepage designs on a color printer. The printouts were magnified to compensate for the poorer quality of the color pixels in the printout and to make it easier for the observers to see what the user was pointing at.
The test was conducted by showing the page to users and asking them to first comment on their general impression of the page and then to point to any element on the page that they thought they could click on and tell us what they expected would happen. This simple method provided us with early feedback indicating the importance of a prominent placement of the month name (since we wanted users to know that the page would change monthly) and the need to make the "What's Happening" bar look very clickable.
The photo shows version B of the homepage "on the operating table," but we ran paper prototype tests of many more versions. It's the beauty of this method that it allows for very rapid iteration and tests of many alternative user interface designs.
A nice trick for paper prototyping (which Meghan Ede suggested to us) is to tape up an area of the desk with masking tape. Marking up an area usually ensures that the user keeps the printed page within the area without a need for the experimenter to try to stop the user from moving the printout around too much. We preferred having the printed page stay within a set area to facilitate video taping as well as observation by other team members in the control room.
Card Sorting to Discover the Users' Model of the Information Space
The category grouping used for the main icons on the home page was derived based on a card sorting study: several users were given a series of index cards with various concepts from the server written on them. The users were then asked to sort cards into piles, placing cards that seemed similar to them into the same pile. The users were also allowed to group piles to indicate looser levels of similarity, and we finally asked the users to name their piles. These names provided us with additional insights into the users' mental model of the information space and served as inspiration for the names we finally chose. Concepts that were placed in the same pile by many users were deemed sufficiently similar that we should place them in the same category in our new design.
See my article about how we designed Sun's intranet, SunWeb, in 1994 for another example of card sorting and a photo of a variant of the method, where users are asked to sort cards onto existing categories. (Often called closed card sorting to differentiate it from the more common open sorting where users build their own categories without any constraints.)
For card sorting, different topics are written on cards to be grouped by participants.
Traditional Usability Test of Running System
Of course, we also used traditional user testing, where users were asked to use a running prototype of the new pages. This is how most usability tests are set up: the user interface that is being tested is running on a computer in the usability lab and the user sits down at the computer and starts working. Normally, we give users a series of set tasks (e.g., find information about the Spring distributed operating system project) and we certainly did so for the web studies. Sometimes, rather primitive HTML mock-ups were used, with many dangling links or links that pointed to pages on the old server that used an obsolete user interface. Even so, we were able to learn a lot about how people use WWW pages. For this particular type of user interfaces we also found it important to let the user spend some time exploring the information space freely: we wanted to see what interface elements the users naturally found interesting without any prompting. We did keep some set tasks since we also wanted to know how usable the design was for people who needed to find specific information.
Most tests were conducted in the usability lab in Mountain View, CA as shown in the photo, though we also conducted three tests in Sun offices in Europe and Asia to assess international usability. The usability lab has a large one-way mirror that allows team members in the control room to observe the test and discuss its outcome without interrupting the test user. Some times the experimenter sits in the control room but my preference is to sit with the user in the lab itself. We have two cameras in the ceiling (not shown) and one camera on a tripod. These cameras are used to record the user's reactions to the interface as well as to film the screen. We do have a scan converter that is used to record a complete screen image, but we often find that there is some extra value from having a camera pointed at the screen because it can record cases where the user points to something on the screen.
It is hard to see on this photo, but to the right of the computer is the lavaliere microphone which we ask users to wear during the test. We have found that the only way to get good sound is to use these clip-on models since table mikes capture too much noise to make it clear for people in the observation room what the user is saying.
A small trick: we have a small table clock to the left of the computer. This clock makes it easy for the experimenter to time the test and to manage the available session time much more unobtrusively than when having to look at a wristwatch.
Traditional usability testing consists of users interacting with the live Web site.
Usability Test of the Old Web Site
In addition to studying our various new design ideas, we also conducted a usability study of Sun's old WWW design. Of course, we were well aware that it contained many usability problems (e.g., inconsistent headerbars and several very strange imagemaps) but we still wanted to learn what worked well and what worked less well in the old design. One interesting finding was that the top button-bar did not look enough as buttons: the design did not have a clickability affordance but was seen as mere decoration by most users. We redesigned this specific aspect of the old design immediately without waiting for the full redesign to come online. The following figure shows before and after versions of the top row of buttons ("What's New", etc.) in the old design:
Changing the buttons to make them look more clickable as illustrated above resulted in 416% increased use over a two-month period (January-March 1995). Considering that the use of the server in general increased by "only" 48% in the same period, there can be no doubt that usability engineering worked and resulted in a significantly improved button design.
Share this article: