Quantitative UX metrics allow us to track the quality of experiences over time, and see how they improve. They help UX professionals gauge the quality and impact of their work, and communicate that impact to others.
The process of UX benchmarking involves choosing one or more metrics that represent important aspects of the experience and then tracking those metrics to see how design interventions impact them.
The following case study illustrates how one team used a UX metric to evaluate the impact of its work and to demonstrate it to their client.
In an interview, I spoke with four members of the Marketade team:
- John Nicholson, principal
- Sonya Badigian, UX researcher
- Nora Fiore, UX writer
- Emily Williams, UX researcher
Case Study: Marketade for Baileigh Industrial
Marketade is a small, long-established, and fully remote user-research company with expertise in turning user insights into better experiences. Its clients include GEICO, McDonald’s, Herman Miller, the United Nations, and Stanford University.
One of its clients, Baileigh Industrial, is a Wisconsin-based top manufacturer of industrial metal and woodworking machinery. The company sells through distributors and directly to customers through Baileigh.com. Due to the nature of their products, individual sales on Baileigh Industrial’s site can be huge — often over $15,000 each.
Sales-Representative Interviews Reveal a Major Problem
Marketade began its work with Baileigh by conducting qualitative interviews with sales representatives. Sales and customer-service departments are great places to start looking for potential experience problems. People who regularly interact with customers are likely to know their top issues and concerns.
“At Marketade, we’re dedicated to talking with our stakeholders as much as we can before we launch the research process. We really want to understand the context that they have. In this case, we also spent time learning about the machinery and technical jargon.”
Emily Williams, UX researcher
A major pain point among the sales representatives was that customers were frequently calling because they could not find the information they needed on the website. Because they were flooded with simple inquiries, expert sales reps did not have time to answer complex questions. The website richly documented information about small-value items, whereas some large-value items could only be bought through sales representatives due to the level of specialization. For example, representatives might spend 20 minutes answering questions about a $716 hand bender instead of working to find a unique solution that might result in the purchase of a $130,000 laser table.
When the Marketade team spoke to senior management, they heard the same complaint. The researchers realized that improved self-service on Baileigh’s website would free up sales representatives to focus on people who needed their expertise and would enable them to spend most of their time on high-value sales.
Understanding the Problem Through Qualitative Research
Marketade then moved on to qualitative usability testing to understand why so many customers were reaching out to the sales representatives. They discovered a critical barrier to self-service: customers often struggled to find the product or even the product category that they wanted. They repeatedly wasted time going down the wrong path using the site’s navigation.
To take a baseline measurement of the navigation’s performance and to determine which areas of the navigation structure needed improvement, Marketade conducted a tree test using Optimal Workshop’s Treejack. The team recruited 64 participants for the test, using ads in Baileigh’s email newsletter and social media pages.
The test consisted of eight possible tasks, each one focusing on a different top-selling product category — for example, You need to precisely bend some quarter-inch sheet metal into a V shape on a vertical plane. Where would you look?
After the study, Treejack scored Baileigh’s tasks on various findability-related metrics, resulting in an average overall score of 4 out of 10. Based on this result, the Marketade team decided that the information architecture could be substantially improved, hopefully leading to a decrease in unnecessary sales calls.
Building a Better Information Architecture
For the results of each task, Marketade researchers looked at the most common incorrect paths. Across all tasks, they found some themes:
- Too many top-level product-category choices, leading many users down the wrong path from the start
- Navigation hierarchies that overlapped and were not intuitive to users
- Similar or misleading subcategory names, causing many users to pick the wrong targets
The Marketade team then conducted a card-sort study to understand how Baileigh’s customers think about and categorize their products.
The result was a new information architecture that:
- Broadened categories so that users have an easier first choice and are not funneled away from their target by a single wrong click
- Reevaluated and reworded problematic subcategory names to prevent confusion
- Included context for tools’ functionalities where possible
By narrowing the global navigation, Marketade reduced the number of options that Baileigh customers had to consider initially. Customers could choose a category and then view a product-category page. The product-category pages contained details about the different product options available, helping customers who were unfamiliar with the domain — all without requiring any involvement from the sales team.
Quantifying the Results
The team conducted a second tree test using the new information architecture (IA). To get a benchmark comparison, they reused the original tasks (with minor wording tweaks to account for new navigation labels) and recruited a fresh group of participants.
In the first tree test with the original IA, the overall score was 4.0 out of 10 across eight tasks. In the second tree test with the revised IA, the overall score was 7.4 out of 10 across those same eight tasks — an 85% increase in product findability.
The original IA failed to score above a 5 out of 10 on any of the 8 tasks. The revised IA scored a 7 or higher on 6 of the 8 tasks.
The client team was thrilled. In the following months, Baileigh’s revenue and leads increased substantially.
“Since launching the new site, the company’s web revenue and leads are up by large margins. While it’s not possible to isolate the architecture’s contribution to this growth, follow-up analysis through Google Analytics suggests a significant ROI from our human-centered overhaul of the company’s IA.”
John Nicholson, principal at Marketade
The average improvement in UX metrics from a UX-driven redesign was 75% across our recent research. Obviously, hitting this exact average is rare, and some projects will be better and some worse. Why did this team realize a bigger improvement (85% instead of 75%)? There are many possible factors, and we can’t know for sure. One possible explanation is that this was a complex design problem for a specialized domain, as is common in the B2B space. Paradoxically, the harder the design problem, the larger the opportunities for UX improvements can be, because there are so many possible design dimensions to explore.
For More Information
Marketade’s case study represents an ideal synthesis of qualitative research (for issue discovery and solution development) and quantitative research (for assessing and tracking performance). This case study is just one excerpt from a set of 44 real-life case studies detailed in our 297-page report, UX Metrics and ROI. The case studies include a wide variety of product types, UX improvements, and quantitative metrics. The report also includes more details from the Marketade interview, as well as interviews with 8 other UX teams.
For help developing your own quantitative benchmarking plan, check out our full-day seminar on Measuring UX and ROI.
To learn more about common findability problems, as well as how to identify and solve them, check out our full-day seminar on Information Architecture.
Share this article: