The following tables contain links to some of our articles and videos related to quantitative user research. Within each section, the resources are in recommended reading order. 

Quantitative vs. Qualitative UX Research

In UX, we often use qualitative research to gather insights or observations about users. This type of research is useful for discovering problems and determining design solutions. (We also have a study guide for qualitative usability testing.) 

With quantitative research, our focus is different. We collect UX metrics — numerical representations of different aspects of the experience. Quantitative research is great for determining the scale or priority of design problems, benchmarking the experience, or comparing different design alternatives in an experimental way.

4-minute video: Quantitative vs. Qualitative UX Research

Topics and Methods Covered in This Article

UX Benchmarking and Return on Investment (ROI)

UX benchmarking refers to evaluating a product or service’s user experience by using metrics to gauge its relative performance against a meaningful standard. Teams use benchmarking to track improvements to the user experience over time or to compare  against competitors. 

Benchmarking metrics are often also used to the calculate return on investment (ROI) of UX work; this type of calculation helps UX professionals prove their value and argue for more resources.

Number

Link

Format

Description

1

The Benefits of Benchmarking Your Product’s UX

Video

Track how well your design performs over time

2

Benchmarking UX: Tracking Metrics

Article

How benchmarking works at a high level

3

7 Steps to Benchmark Your Product’s UX

Article

Specific steps to follow to get started with benchmarking

4

Calculating ROI for Design Projects

Video

Using metrics to estimate the value of a design change

5

Calculating ROI for Design Projects in 4 Steps

Article

6

Three Myths About Calculating the ROI of UX

Article

Common mistakes people make when they get started with ROI calculations

7

Average UX Improvements Are Shrinking Over Time

Article

An analysis of benchmarking trends since 2006, meant to set expectations for how much your metrics might change over time

For more in-depth help, check out our report and full-day course. (Unlike the articles and videos in this study guide, these resources are not free.)

Report: UX Metrics and ROI

Full-day course: Measuring UX and ROI

Quantitative Usability Testing

In quantitative usability testing, researchers collect metrics (like time on task, success rates, and satisfaction scores) while participants perform tasks. This version of usability testing requires more participants and a more rigorous study structure than qualitative usability testing.

Number

Link

Format

Description

1

Quantitative vs. Qualitative UX Research

Video

How to determine when you need a quantitative study

2

Quantitative vs. Qualitative Usability Testing

Article

Differences between quantitative user testing and (the more-common) qualitative usability testing 

3

How Many Participants for Quantitative Usability Studies: A Summary of Sample-Size Recommendations

 

Article

The reasoning between the 40-participant guideline for quant user testing and why you may see other recommendations

4

Why You Cannot Trust Numbers from Qualitative Usability Studies

 

Article

Why it’s a mistake to  think you can collect quant metrics during qual studies 

5

Why 5 Participants Are Okay in a Qualitative Study, but Not in a Quantitative One

 

Article

Why sample sizes differ in quantitative vs. qualitative user testing

6

Writing Tasks for Quantitative and Qualitative Usability Studies

 

Article

The differences between tasks for quant vs. qual user testing and why good quant tasks are specific and concrete

7

Success Rate: The Simplest Usability Metric

Article

How to analyze task completion when you have multiple levels of success 

8

Risks of Quantitative Studies

 

Article

The reason why quantitative usability studies can’t replace qualitative studies, and how qual studies can complement the findings from quant studies

9

Between-Subjects vs. Within-Subjects Study Design

Article

How to choose between two alternative study setups in quant usability testing that compare two different designs 

10

How to Measure Learnability of a User Interface

Article

Quantifying the learnability of complex products that take a while for new users to learn by looking at how much time it takes people to learn the interface

Analytics and A/B Testing

Analytics data describe what people do with your live product — where they go, what they click on, what features they use, where they come from, and on which pages they decide to leave the site or app. This information can support a wide variety of UX activities —  it can help you monitor the performance of various content, UIs, or features in your product and identify what doesn’t work.

Number

Link

Format

Description

1

Analytics vs. Quantitative Usability Testing

Video

Comparing the information obtained from these two sources of quantitative metrics for UX

2

Three Uses for Analytics in User-Experience Practice

Article

How to avoid feeling lost in your analytics data and make it meaningful

3

Macro & Microconversions as Metrics in Analytics

Video

How to use both high-value user actions (macroconversions) and smaller-value, frequent user actions (microconversions) as analytics metrics to track the performance of your site and identify issues

4

Translating UX Goals into Analytics Measurement Plans

Article

Advice for choosing the right analytics metrics for your specific UX goals

5

Turning Analytics Findings into Usability Studies

Video

Pairing analytics with qualitative research to learn the “why” behind those problems identified through analytics

6

In Analytics, What do the Numbers Really Mean?

Video

How to understand analytics metrics that require interpretation

7

How to Interpret User Time Spent and Page Views

Video

When and how to use two key analytics metrics (time spent and page views) to evaluate whether your users are efficient or engaged

8

Vanity Metrics: Add Context to Add Meaning

Article

Why metrics that only go up (like total visitors) aren’t very useful and how to avoid these feel-good vanity metrics

9

5 Information Architecture Warning Signs in Your Analytics Reports

Article

How to use analytics to  discover potential problems in your product’s information architecture

10

Bounces vs. Exits in Web Analytics

Video

The difference between  two metrics that  people often confuse

While you can use analytics metrics to monitor your product’s, you can also create experiments that detect how different UI designs affect those metrics — either through A/B testing or multivariate testing.

Number

Link

Format

Description

1

A/B Testing 101

Video

How A/B testing works

2

Define Stronger A/B Test Variations Through UX Research

Article

How to ground your A/B testing experiments in research to develop well informed design variations

3

Don’t A/B Test Yourself Off a Cliff

Video

Why relying on A/B testing alone is likely to result in  design mistakes. 

4

Putting A/B Testing in Its Place

Article

5

A/B Testing vs. Multivariate Testing for Design Optimization

Video

When you need multivariate testing vs. A/B testing and why multivariate testing requires more traffic 

6

Multivariate vs. A/B Testing: Incremental vs. Radical Changes

Article

Full-day course: Analytics and User Experience

Surveys

Quantitative surveys involve asking a large number of users to answer a standardized set of questions. These surveys often involve selecting a response on a rating scales and are used to quantify users’ perceptions. 

Number

Link

Format

Description

1

User Satisfaction vs. Performance Metrics

Article

Why user satisfaction and performance metrics (like time on task) often correlate, but don’t always 

2

Survey Response Biases in User Research

Article

Biases which might cause problems in your survey data

3

Keep Online Surveys Short

Article

Why online surveys must be short to collect many high-quality responses

4

Iterative Design of a Survey Question: A Case Study

Article

An example of how to design and refine your own survey

5

Rating Scales in UX Research: Likert or Semantic Differential?

Article

When to use each of the two most popular types of rating scales

6

How Useful is the System Usability Scale (SUS) in UX Projects?

Video

Jakob Nielsen’s thoughts on one of the most popular and longest-standing UX questionnaires

7

Net Promoter Score: What a Customer-Relations Metric Can Tell You About Your User Experience

Article

The Net Promoter Score (NPS) is a popular marketing metric with limited relevance for UX

8

Beyond the NPS: Measuring Perceived Usability with the SUS, NASA-TLX, and the Single Ease Question After Tasks and Usability Tests

Article

A set of questionnaires to consider as alternatives to the NPS

Card Sorting and Tree Testing

Card sorting and tree testing are both useful methods for assessing and improving your product’s information architecture.

In a card-sorting study, participants are given content items (sometimes written on index cards) and asked to group and label those items in a way that makes sense to them. This test can either be conducted in person, using physical cards, or remotely using a card-sorting platform. Card sorting can have qualitative and quantitative components.

In a tree test, participants complete tasks using only the category structure of your site. It’s essentially a way to evaluate your information architecture by isolating it away from all other aspects of your UI.

 

Number

Link

Format

Description

1

The Difference Between Information Architecture (IA) and Navigation

Article

What information architecture is and how it relates to site navigation

2

Card Sorting: Uncover Users' Mental Models for Better Information Architecture

Article

An introduction to card sorting

3

Card Sorting: How to Best Organize Product Offerings

Video

4

Card Sorting: How Many Users to Test

Article

How many participants to include in your card-sorting study

5

 Open vs. Closed Card Sorting

 

Video

How to choose between these two variations of card sorting

6

Tree Testing: Fast, Iterative Evaluation of Menu Labels and Categories

Article

An introduction to tree testing

7

Tree Testing to Evaluate Information Architecture Categories

Video

8

Tree Testing Part 2: Interpreting the Results

Article

How to make decisions based on your tree testing data

9

Quantifying UX Improvements: A Case Study

Article

An example of how one team used tree testing when redesigning a B2B site’s information architecture

Full-day course: Information Architecture

Analyzing Quantitative Data

To draw conclusion and interpret quantitative data, you’ll need to understand some statistics and study-design concepts. The following resources will introduce you to those concepts.

These resources won’t give you step-by-step instructions for calculating things like confidence intervals or statistical significance — these are too complex to be covered in a short article. If you want to learn those analysis procedures, please see our full-day course below.

Number

Link

Format

Description

1

Internal vs. External Validity of UX Studies

Article

Why validity matters in UX studies

2

Why Confidence Intervals Matter for UX

Video

Why you should calculate confidence intervals for your quantitative metrics

3

Confidence Intervals, Margins of Error, and Confidence Levels in UX

Article

Detailed explanations of these three important analysis concepts

4

Statistical Significance in UX

Video

What statistical significance means, and why you should calculate statistical significance when comparing two designs quantitatively

5

Understanding Statistical Significance

Article

6

Handling Insignificance in UX Data

Video

What to do when your findings are not statistically significant

Full day course: How to Interpret UX Numbers