The following tables contain links to some of our articles and videos related to quantitative user research. Within each section, the resources are in recommended reading order.
Quantitative vs. Qualitative UX Research
In UX, we often use qualitative research to gather insights or observations about users. This type of research is useful for discovering problems and determining design solutions. (We also have a study guide for qualitative usability testing.)
With quantitative research, our focus is different. We collect UX metrics — numerical representations of different aspects of the experience. Quantitative research is great for determining the scale or priority of design problems, benchmarking the experience, or comparing different design alternatives in an experimental way.
4-minute video: Quantitative vs. Qualitative UX Research
Topics and Methods Covered in This Article
- UX Benchmarking and Return on Investment
- Quantitative Usability Testing
- Analytics and A/B Testing
- Surveys
- Card Sorting and Tree Testing
- Analyzing Quantitative Data
UX Benchmarking and Return on Investment (ROI)
UX benchmarking refers to evaluating a product or service’s user experience by using metrics to gauge its relative performance against a meaningful standard. Teams use benchmarking to track improvements to the user experience over time or to compare against competitors.
Benchmarking metrics are often also used to the calculate return on investment (ROI) of UX work; this type of calculation helps UX professionals prove their value and argue for more resources.
|
Number |
Link |
Format |
Description |
|
1 |
Video |
Track how well your design performs over time |
|
|
2 |
Article |
How benchmarking works at a high level |
|
|
3 |
Article |
Specific steps to follow to get started with benchmarking |
|
|
4 |
Video |
Using metrics to estimate the value of a design change |
|
|
5 |
Article |
||
|
6 |
Article |
Common mistakes people make when they get started with ROI calculations |
|
|
7 |
Article |
An analysis of benchmarking trends since 2006, meant to set expectations for how much your metrics might change over time |
For more in-depth help, check out our report and full-day course. (Unlike the articles and videos in this study guide, these resources are not free.)
Report: UX Metrics and ROI
Full-day course: Measuring UX and ROI
Quantitative Usability Testing
In quantitative usability testing, researchers collect metrics (like time on task, success rates, and satisfaction scores) while participants perform tasks. This version of usability testing requires more participants and a more rigorous study structure than qualitative usability testing.
|
Number |
Link |
Format |
Description |
|
1 |
Video |
How to determine when you need a quantitative study |
|
|
2 |
Article |
Differences between quantitative user testing and (the more-common) qualitative usability testing |
|
|
3 |
How Many Participants for Quantitative Usability Studies: A Summary of Sample-Size Recommendations
|
Article |
The reasoning between the 40-participant guideline for quant user testing and why you may see other recommendations |
|
4 |
Why You Cannot Trust Numbers from Qualitative Usability Studies
|
Article |
Why it’s a mistake to think you can collect quant metrics during qual studies |
|
5 |
Why 5 Participants Are Okay in a Qualitative Study, but Not in a Quantitative One
|
Article |
Why sample sizes differ in quantitative vs. qualitative user testing |
|
6 |
Writing Tasks for Quantitative and Qualitative Usability Studies
|
Article |
The differences between tasks for quant vs. qual user testing and why good quant tasks are specific and concrete |
|
7 |
Article |
How to analyze task completion when you have multiple levels of success |
|
|
8 |
|
Article |
The reason why quantitative usability studies can’t replace qualitative studies, and how qual studies can complement the findings from quant studies |
|
9 |
Article |
How to choose between two alternative study setups in quant usability testing that compare two different designs |
|
|
10 |
Article |
Quantifying the learnability of complex products that take a while for new users to learn by looking at how much time it takes people to learn the interface |
Analytics and A/B Testing
Analytics data describe what people do with your live product — where they go, what they click on, what features they use, where they come from, and on which pages they decide to leave the site or app. This information can support a wide variety of UX activities — it can help you monitor the performance of various content, UIs, or features in your product and identify what doesn’t work.
|
Number |
Link |
Format |
Description |
|
1 |
Video |
Comparing the information obtained from these two sources of quantitative metrics for UX |
|
|
2 |
Article |
How to avoid feeling lost in your analytics data and make it meaningful |
|
|
3 |
Video |
How to use both high-value user actions (macroconversions) and smaller-value, frequent user actions (microconversions) as analytics metrics to track the performance of your site and identify issues |
|
|
4 |
Article |
Advice for choosing the right analytics metrics for your specific UX goals |
|
|
5 |
Video |
Pairing analytics with qualitative research to learn the “why” behind those problems identified through analytics |
|
|
6 |
Video |
How to understand analytics metrics that require interpretation |
|
|
7 |
Video |
When and how to use two key analytics metrics (time spent and page views) to evaluate whether your users are efficient or engaged |
|
|
8 |
Article |
Why metrics that only go up (like total visitors) aren’t very useful and how to avoid these feel-good vanity metrics |
|
|
9 |
5 Information Architecture Warning Signs in Your Analytics Reports |
Article |
How to use analytics to discover potential problems in your product’s information architecture |
|
10 |
Video |
The difference between two metrics that people often confuse |
While you can use analytics metrics to monitor your product’s, you can also create experiments that detect how different UI designs affect those metrics — either through A/B testing or multivariate testing.
|
Number |
Link |
Format |
Description |
|
1 |
Video |
How A/B testing works |
|
|
2 |
Article |
How to ground your A/B testing experiments in research to develop well informed design variations |
|
|
3 |
Video |
Why relying on A/B testing alone is likely to result in design mistakes. |
|
|
4 |
Article |
||
|
5 |
A/B Testing vs. Multivariate Testing for Design Optimization |
Video |
When you need multivariate testing vs. A/B testing and why multivariate testing requires more traffic |
|
6 |
Multivariate vs. A/B Testing: Incremental vs. Radical Changes |
Article |
Full-day course: Analytics and User Experience
Surveys
Quantitative surveys involve asking a large number of users to answer a standardized set of questions. These surveys often involve selecting a response on a rating scales and are used to quantify users’ perceptions.
|
Number |
Link |
Format |
Description |
|
1 |
Article |
Why user satisfaction and performance metrics (like time on task) often correlate, but don’t always |
|
|
2 |
Article |
Biases which might cause problems in your survey data |
|
|
3 |
Article |
Why online surveys must be short to collect many high-quality responses |
|
|
4 |
Article |
An example of how to design and refine your own survey |
|
|
5 |
Rating Scales in UX Research: Likert or Semantic Differential? |
Article |
When to use each of the two most popular types of rating scales |
|
6 |
How Useful is the System Usability Scale (SUS) in UX Projects? |
Video |
Jakob Nielsen’s thoughts on one of the most popular and longest-standing UX questionnaires |
|
7 |
Net Promoter Score: What a Customer-Relations Metric Can Tell You About Your User Experience |
Article |
The Net Promoter Score (NPS) is a popular marketing metric with limited relevance for UX |
|
8 |
Article |
A set of questionnaires to consider as alternatives to the NPS |
Card Sorting and Tree Testing
Card sorting and tree testing are both useful methods for assessing and improving your product’s information architecture.
In a card-sorting study, participants are given content items (sometimes written on index cards) and asked to group and label those items in a way that makes sense to them. This test can either be conducted in person, using physical cards, or remotely using a card-sorting platform. Card sorting can have qualitative and quantitative components.
In a tree test, participants complete tasks using only the category structure of your site. It’s essentially a way to evaluate your information architecture by isolating it away from all other aspects of your UI.
|
Number |
Link |
Format |
Description |
|
1 |
The Difference Between Information Architecture (IA) and Navigation |
Article |
What information architecture is and how it relates to site navigation |
|
2 |
Card Sorting: Uncover Users' Mental Models for Better Information Architecture |
Article |
An introduction to card sorting |
|
3 |
Video |
||
|
4 |
Article |
How many participants to include in your card-sorting study |
|
|
5 |
|
Video |
How to choose between these two variations of card sorting |
|
6 |
Tree Testing: Fast, Iterative Evaluation of Menu Labels and Categories |
Article |
An introduction to tree testing |
|
7 |
Tree Testing to Evaluate Information Architecture Categories |
Video |
|
|
8 |
Article |
How to make decisions based on your tree testing data |
|
|
9 |
Article |
An example of how one team used tree testing when redesigning a B2B site’s information architecture |
Full-day course: Information Architecture
Analyzing Quantitative Data
To draw conclusion and interpret quantitative data, you’ll need to understand some statistics and study-design concepts. The following resources will introduce you to those concepts.
These resources won’t give you step-by-step instructions for calculating things like confidence intervals or statistical significance — these are too complex to be covered in a short article. If you want to learn those analysis procedures, please see our full-day course below.
|
Number |
Link |
Format |
Description |
|
1 |
Article |
Why validity matters in UX studies |
|
|
2 |
Video |
Why you should calculate confidence intervals for your quantitative metrics |
|
|
3 |
Confidence Intervals, Margins of Error, and Confidence Levels in UX |
Article |
Detailed explanations of these three important analysis concepts |
|
4 |
Video |
What statistical significance means, and why you should calculate statistical significance when comparing two designs quantitatively |
|
|
5 |
Article |
||
|
6 |
Video |
What to do when your findings are not statistically significant |
Full day course: How to Interpret UX Numbers
Share this article: