Many people think that it takes special skill and fancy equipment to run a user-testing session on mobile. Not true. If you’ve ever run user-testing sessions with regular computers, setting up a usability study with mobile devices is a breeze. You need to take care of a few extra details: equipment, the right testing room, and the right users.
Equipment
One of the daunting problems in user testing with mobile devices is: how do you record the screen? In studies run on a computer, you can use a a screen-recording software such as Morae, Camtasia, or Ovo Studios, but what should you use for mobile devices? Screen-recording apps for mobile are still in incipient stages; often they require a jailbroken phone or don’t allow the user to take full advantage of the phone capabilities (for instance, UX Recorder for iOS only records the screen if the user uses an in-app browser). Screen-recording software rarely provides finger footage: the researcher will have to use a separate camera to record the study-participant’s fingers, and then possibly combine the finger and the screen streams.
For all these reasons, recording with an external camera is still the preferred method when doing user testing for mobile. Almost any video camera can be used, provided that it will stay focused on the screen while the study participant is using the device. Although it’s possible to fix a regular camera on a tripod right above the mobile device, most mobile-usability researchers prefer either a webcam or a document camera because they usually come with software that allows real-time projection of the video footage on a computer screen. That allows the facilitator to witness what the study participant does on her phone without intruding in her personal space. In our mobile-usability studies, we have used 2 types of cameras: a document camera and a webcam. There are advantages and disadvantages to each of them.
Document Camera
This solution somewhat restricts the movement of the phone during the session. Since the camera is fixed, study participants must maintain their phones within the camera’s range. They can do that either by keeping the device flat on the table, or by holding the phone in their hands (although in this last case the resulting video quality is suboptimal). They can also easily (and naturally) switch the device orientation from landscape to portrait as needed.
Document cameras are available in a wide range of cost and quality. We have used Elmo (at the higher end of the spectrum) and iPevo (at the lower end). Better document cameras have manual focus, a variety of image controls, and the ability to zoom in without substantially altering the quality of the resulting image. They also have an adjustable, preferably tall neck that allows a wide range of recording and is suited for both tablets and phones.
Webcams and Mobile-Device Cameras
Webcams are another class of devices that can be used for mobile- or tablet-screen recording. These are usually installed on a cradle that can be either self-fabricated or purchased. The user then places her phone on the cradle (sometimes using Velcro to attach it) and holds the entire contraption in their hands, in a position that is more naturalistic than when the phone rests on the table.
How big the cradle is will influence the ability to record tablet screens (small cradles do not work well with tablets). Another essential quality of the cradle is lightness: since the participant is supposed to hold the camera, the cradle, and the mobile device in her hands, the resulting apparatus should be as light as possible.
The combination cradle plus webcam is sometimes called mobile-device camera; the web is full with instructions about building your own cradle. You can also buy either the cradle only or the entire package¬: cradle plus camera.
The webcam needs to allow manual focus and zooming. (With autofocus, the camera tends to focus on the participant’s fingers rather than on the device screen.)
Although at first the mobile-device camera may seem more appealing than the document camera (both in terms of cost and flexibility of device positioning), in practice there are some big disadvantages. First, the mobile-device camera is often too heavy (at least 200 grams), and users tire quickly and place it (and their phone) on the table. Second, the image quality is often less good than with a document camera. Third, often this solution is not suited for tablets: due to the larger surface of the tablet, the cradle needs to be taller than for a mobile phone to support a wider recording angle, and the resulting contraption becomes too heavy. Last but not least, the camera can block the view in landscape mode. There are some solutions to deal with this last issue — for instance, if the phone is not attached with Velcro to the cradle, the user may be able to switch it to landscape without turning the entire cradle and blocking the view with the camera. But that means that she has to use extra force in her fingers to keep the phone stable on the cradle while she’s working with it.
Software
Whatever camera you use, you will need some way to project it on a computer screen to allow the facilitator to follow the participant as he is working on the device. Document cameras and webcams usually come with software that projects the camera capture live on a computer screen. Once you have that live capture on a big monitor, you can take advantage of techniques normally used in traditional usability testing to record and broadcast the computer screen — for instance, you can use your favorite screen-recording software (Morae, Camtasia, Ovo, or even GotoMeeting and Webex).
Up to 2 observers can watch the video on a laptop screen by crowding close to the facilitator; with more observers use the laptop's external-monitor port to project the video feed to an external monitor or projector.
Some cameras come with their own video-recording software; if that is the case, you can skip the screen recording and use the camera software instead. The disadvantage is that often these programs require more computing power than your regular screen-recording software.
Testing Environment
The room in which you set up your testing session needs to satisfy a few extra constraints to be suitable for mobile testing. A regular user-testing lab will be nice, but is not necessary unless you plan to have many live observers.
First and foremost, you need to be able to control the sources of light in the room. That means either a windowless testing room or one with darkening blinds. To avoid glare, there should be no source of light directly above the testing device. In most cases, turning off the lights and using side table lamp(s) works well.
The brightness of the camera, device screen, and the monitor on which the device screen is projected will all influence the quality of the live projection and of the resulting video. You should be able to easily adjust these at the beginning of the testing session and, if necessary, during testing (although it’s best to avoid interrupting the participant during the tasks with a request to adjust device-screen brightness).
You will also need to make sure that you have a good cellular signal in the testing room, as well as a high-speed wireless network available. We usually recommend that you test apps and sites on both cellular and WiFi networks, as these different tests often provide complementary information. Because connectivity is usually worse on cellular signal, you can see how your app fares on a poor connection. In contrast, with a good WiFi, you can focus on interface issues: Is that button salient enough? Can the user understand the content?
Unlike for traditional user testing, for (at least some) mobile studies it may be ok if the testing room has some nearby employee traffic and occasional noise. Often, mobile devices are used in interruptible environments, so sporadic, nondeterministic external sounds may create a slightly more naturalistic situation.
Users
Unless you plan to study the learnability of a new device, we usually recommend that you recruit people who are familiar with their devices and have been using them for at least 3 months. New users often do not exhibit typical behavior — they may not know how to use their device yet, or they may not be familiar enough with conventions specific to the operating system.
Unless you're Apple or Google, it's less important to you how people fare the first few times they use a new phone or tablet. If you have a website or mobile app, almost all your users will already have learned how to use their device by the time they get around to using your site or app.
Some people own several phones or several tablets. When you screen participants, make sure that the questions specify the device that you want to study. For instance, if you want iPad users who shop on their device, you may want to ask “How often do you shop on your iPad?” rather than “How often do you shop on your tablet?”
If you want users to install apps from an app store on their devices, make sure that:
- They know how to do it (this can be an issue especially for tablets, which are often shared devices).
- They either know their App Store password or they will bring it for the testing session.
- They have a credit card associated to their account (if you want them to download paid apps during the study). If they don’t have one, be prepared to give them a gift card that they could use for purchasing the apps —and, of course, reimburse them right away for any such purchase.
Other Parts of Testing
For anything else (screening, task creation, facilitation, and reporting), you can use the skills learned from regular user-testing sessions. The most important point about a mobile usability study is that it's a usability study, not that it's mobile. We already have wide range of articles on user testing, as well as full-day courses with in-depth hands-on practice in study facilitation.
Conclusion
In order to setup a mobile-usability study, invest in a good document or mobile-device camera and make sure that you have a testing room with proper lighting. Connect the camera to a testing computer and use existing screen-recording software to record and broadcast the sessions. Finally, take advantage of prior knowledge about running and analyzing usability studies.
One thing that we've learned from running hundreds of mobile usability studies — whether for clients or to get examples from our courses on mobile and tablet design — is that the usability requirements increase as the platform shrinks. Smaller screens equal bigger needs to test your design with real users, because there are more ways for users to fail. Since it's easy enough to run these usability studies, it's even more important to test your design when you're targeting mobile users.
Share this article: