Testing Whether Web Page Templates are Helpful

It is easy to test the usability of individual Web pages: simply have users interact with the pages and see whether they understand them or have trouble. Page templates are another story. In a traditional test two issues overlap and are hard to separate:

• usability of the layout specified by the template design
• usability of the specific content that has been poured into this template on the individual pages

Before you unleash a page template on a large number of content creators and direct them to design thousands of pages to its specifications, it would be nice to have more data on the usability of the template itself. By "usability of template" I mean whether a page gets more or less usable because of the layout specified by the template. It would also be interesting to study whether it is easy for page authors to design with the template, but that is a different matter.

Thomas S. Tullis from Fidelity Investments recently conducted a study to assess the usability of five alternative template designs for their intranets. To test whether the layout itself helped users find the page elements, all text on the page was "greeked" — that is, all words were replaced with unintelligible nonsense (sometimes called mumble text). When they can't read the text, users have to rely on the inherent communicative aspects of the layout to perform the test task. If a layout performs well when users can't understand any of the content, then there is hope that the template will survive substantial abuse from authors who fill it with content of varying quality.

Each test user was given printouts of the greeked pages (in random sequence) and asked to draw labeled blocks around the parts of a page that corresponded to each of nine standard elements. Blocks could not overlap and could not be nested. If a user thought that an element was missing from a page, it was to be marked as "not there." The study included five templates of which two are shown here (reproduced with permission).

Page Template 1

Page Template 3

Each page template had the following nine standard elements which you should try to identify in the figures:

• Main content selections for this page
• Page title
• Person responsible for this page
• Intranet-wide navigation (e.g., intranet home, search)
• Last updated date
• Intranet identifier/logo
• Site navigation (e.g, major sections of this section of the intranet)
• Confidentiality/security (e.g, Public, Confidential, etc.)
• Site news items

The correct solutions are shown in a sidebar, but please try to solve the problem yourself before peeking. You only get one chance to approach a problem as a novice user.

As shown in the table, users performed best with Template 3, where they were able to correctly identify 67% of the standard page elements. Users were also asked to subjectively rate how well they liked the templates on a -3 to +3 scale. As shown in the table, Template 1 was the best liked, even though users performed significantly worse with this template. This outcome demonstrates the danger of simply showing designs to people and asking them which one they like the best: users may not know what's good for them.

Since different designs scored best on the two usability metrics (performance and satisfaction), Fidelity decided to combine the best parts of the designs into a new, final design. When this final design was tested with the same methodology, it scored better than any of the original designs, proving that it was indeed an improvement.

The greeking technique is a very fast way of gathering usability data about proposed page templates before a large number of pages have been constructed. Since users can't actually use nonsense pages, there is no need to implement anything: instead, each design can be printed out and marked up on paper.

Reference

Tullis, T.S. "A method for evaluating Web page design concepts." In ACM Conference on Computer-Human Interaction CHI 98 Summary (Los Angeles, CA, 18-23 April 1998), pp. 323-324.
Unless you were at the conference, you will have a hard time getting hold of the conference summary book: I am including the reference for completeness and to acknowledge Tullis' paper, but I cannot supply reprints.