International Usability Testing

In Sweden, the Automatic Teller Machines have very large buttons. I hadn't noticed this particular design element on previous visits, which have usually been in warmer months. In 1996 I was in Stockholm in February and immediately realized why the ATM buttons are so big: you can press them wearing thick gloves.

Clearly, the ATM vendor had manufactured a localized version of the product with Swedish users in mind. Unfortunately, although products are commonly used in countries other than the one they were designed for, designers often forget to consider different usage circumstances. International use of the Web is particularly common since users can access pages from all over the world with a single click.

Levels of Globalization Concerns

International user-interface concerns come at three levels.

1. A computer must be capable of displaying the user's native language, character set, and notations (such as currency symbols).
2. The user interface and documentation must be translated into the user's native language in a way that is understandable and usable.
3. A system must match the user's cultural characteristics. This goes beyond simply avoiding offensive icons; it must accommodate the way business is conducted and the way people communicate in various countries.

Most major computer vendors consider the first level fairly routine, though it has certainly not been fully solved. Most systems either support sets beyond ASCII already or have aggressive plans to do so. Also, most vendors have special manuals that show developers how to code for an international market and the use of, for example, 16-bit character sets.

The figure shows a problem in the culture category from a game called Give the Dog a Bone found on the otherwise excellent JumpStart Toddlers CD-ROM: when the dog asks for its ball, most European kids would probably point to the biscuit rather than the football since they have never experienced balls that are not round. As computers transmute into communication tools, the need to match the users' local language and culture becomes paramount.

Unfortunately, the second- and third-level concerns are much harder to address than the character set issues. There is not yet a standardized way of conforming to language and culture. Indeed, very few people have yet given much thought to systematic ways of addressing these deeper issues in internationalization.

Testing Abroad

Even though some results and guidelines are available, when confronted with a product intended for use abroad your best bet is to conduct international usability testing. As with all user testing, the two fundamentals of international user testing are to involve real users and have them do real tasks without your help. You can usually recruit users through your local branch office in the country in question, but it is important to emphasize that you need users who sit at the keyboard on a day-to-day basis and not necessarily the branch's immediate customer contacts. If you are not explicit about these needs, you may find that unrepresentative test participants have been recruited when you show up for the test. By then it will be too late.

There are four main ways to conduct international user testing:

• go to the foreign country yourself
• run the test remotely
• hire a local usability consultant to run the test for you
• have staff from your local branch office run the test, even though they are not trained in usability

A fifth possibility is open only to the largest companies: build additional usability groups in your major markets. This last option may be the best, but it is usually beyond the available budget. Also, even if you do have local usability groups in major foreign markets, you may still want to do additional testing in some of the smaller markets -- in which case, you face the same problem over again.

In many ways, the ideal approach is to go to the country and conduct the test yourself. Visiting local customers in various countries will give you a much stronger impression than simply reading even the most well-written report. There are always many small but important details you will observe if you go there yourself.

Because observing the customers in their own environment is beneficial, I recommend trying to set up the test at the customer's premises, but this is not always possible. You often need special equipment, hard-to-install software, and access to data that is only available on your internal corporate network (which only works inside your branch office). If you do visit customer locations, try to get permission to bring a still camera and take pictures of their installation and working conditions. Pasting several such photos on a wall in your development lab will often be a good way to remind everybody on the project team about the different needs in different countries.

If you do travel yourself, you should account for jet lag. Conducting users tests is a very intense experience: you have to pay attention to the user, the user interface, the test tasks, your notes, any additional observers, and any video equipment you may be using -- all at the same time. Also, the test may be conducted in a foreign language, which can make it even harder to concentrate. I recommend that you spend the first two days after your arrival visiting your branch office and checking equipment and software to make sure that the tests will run smoothly.

You can eliminate many travel costs by using remote user testing. To do this, you have the user run prototype software (or a screen-by-screen mock-up) on his or her own computer while you observe over the Internet (or some other network connection). There are many utilities that let one user see what is on another user's screen; these utilities are perfect for remote user testing. You can communicate with the user over an audio link via telephone or the Internet. The downsides to remote testing are that you have little visual feedback as to what the user is doing and the user must install and operate possibly quite unfamiliar utilities. You also have to be in your office at horrible hours to accommodate the time difference.

Overcoming the Language Gap

Whether you travel yourself or conduct the test remotely, you will normally have a language problem. One solution is to recruit users who speak your language, even if they don't speak it fluently. This solution is not perfect, but pragmatically, it is often the easiest to implement. The key issue is to make sure that you don't get unrepresentative users who, for example, have spent years at college in your country and thus have been acclimated to the possible linguistic and cultural peculiarities you hope to smoke out during the test. Another option is to conduct the test in the local language. This may work if you speak the language reasonably well, but you will often have to rely on an interpreter. Although you can usually understand what is happening on the screen (since you know the product well), the user's comments will normally lose a lot in translation. You should also meet interpreters beforehand and remind them that they should not help users during the test.

The final option is to either have a local usability consultant or staff from your branch office conduct the tests. You will definitely get the highest quality report from a usability professional, but there are benefits to involving your local staff: not only is it cheaper, but it brings them into the product-development process and they will likely learn a great deal from the test itself. As always, additional information is gained from actually carrying out a usability study as opposed to simply reading the report, and this added information might as well benefit people within your company.

How Many Countries?

You must determine how many foreign countries to cover in international usability engineering. The optimal solution is to cover all countries in which you have non-negligible sales (or where you hope to expand). However, doing so is normally unrealistic unless you are a rather small company that only exports to one or two countries. The typical solution is to evaluate international usability in a few countries with at least one country in each of the main areas of the world. If you only have the resources to cover a single foreign country, then you should do it. There is much to be gained in taking that first step, and the knowledge extends well-beyond the benefits you gain in the test country.

Start Small

A key point is: Don't despair! Don't over plan. Don't give up because you cannot implement the ideal international usability study the first time. You may have to start with a single country. The important thing is to do it. Typically, management is so encouaged by the results of even a small-scale pilot project that more resources are freed up for future projects.

(Obviously, you can hire us to do international testing for you, including having us manage testing through our foreign network of subcontractors. However, a cheaper option is have us teach your team — and your international teams — to do their own testing.)