Ecommerce User Experience Vol. 13: Ecommerce User Research Methodology

This 118-page report describes the methods used for our ecommerce user research and includes the details of the methods used in our user research:

• Diary-based longitudinal study
• Usability testing
• Eyetracking research
• Expert reviews

This report will help you run your own usability studies. The report explains the reasons for the methodologies selected, the tasks and instructions used for each part of the study, and the lessons learned from using these methods.

The recommendations based on any usability study are only as good as the data and information collected. The quality of this information relies on the methods used to collect the data. We work to establish a standard with which to measure the methods used in other studies. We hope to discourage poor-quality imitators while helping our peers (and motivating ourselves) to produce work of even better quality in the future.

Topics

• Setting goals for user research: Goals of each round of research and reasons for selecting the methods chosen: think-aloud usability studies, eyetracking, diary study and expert reviews
   
• Recruiting users: Requirements for testers and participant demographics
   
• Selecting testing locations: Running studies in conference rooms and rented offices
   
• Equipment setup: Setups for each type of study
   
• Selecting representative websites to test: Considerations in selecting diverse websites for research
   
• List of websites evaluated: Comprehensive list of websites involved in all stages of our research
   
• Tasks and scenarios: Full list of tasks used across all rounds of research, provided as a reference for understanding our results as well as for creating your own tasks
   
• Conducting the studies: Details of facilitating the user sessions
   
• Eyetracking studies: Advice and lessons for getting the best results
   
• Diary methodology for longitudinal data: Recruiting users, explaining study instructions and keeping participants motivated in a longitudinal study
   
• Questionnaires and instructions: Pre- and post-task questionnaires and diary study instructions
   
• Lessons learned and tips for running your own study: What we learned from running our studies

Research Method

The information in this report is based on two separate rounds of ecommerce studies, including:

• One-on-one usability testing
• Diary-based longitudinal study
• Eyetracking

Representative users tested 255 ecommerce websites. The studies took place in the United States, the United Kingdom, Denmark, and China (Hong Kong).