Defining Competitive Evaluations

Definition: Competitive usability evaluations are a method to determine how your site performs in relation to your competitors’ sites. The comparison can be holistic, ranking sites by some overall site-usability metrics, or it can be more focused, comparing features, content, or design elements across sites.

Evaluations can take the form of expert reviews, where an experienced usability practitioner reviews the designs based on her expertise and knowledge of usability, or competitive usability testing, where users complete a set of tasks using 2 or more competing sites.

Rather than simply looking at a competitor’s site to see what they’re doing and what you personally think is interesting or different, doing an evaluation allows the design team to understand what works and what doesn’t from a user’s perspective or an expert’s perspective.

Goals

Competitive evaluations let you assess if your design is better or worse than your competitors as well as discover the relative strengths and weaknesses of competing designs. They allow you to take an in-depth look at how others solve the same design problems. The goal of any competitive evaluation is to see what competitors are doing, how they’re doing it, what’s working and what’s not.

Competitive evaluations are often a good initial research activity for a project. They can help determine the direction of a design or the need for the development of a feature. The goals of the competitive evaluation should be clear before any work is started.  What design challenge are you trying to solve? What features of your competitors seem interesting or appealing? What feature on your site do you want to compare to others? 

Organizations with a substantial investment in strategically-managed user experience can also conduct longitudinal competitive evaluations to track relative status over time. This is expensive and not for companies at lower levels of organizational maturity. (Nielsen Norman Group typically charges clients $45,000 for a competitive usability test, so to do a lot of them requires a beefy budget.)

Benefits

A usability evaluation, whether a test or review, can help the team make decisions based on knowledge about what elements work well for users, rather than based on personal opinions. Competitive evaluations do this, and also have a few other benefits:

  • Risk reduction: Often you'll find that features that are pushed hard at industry tradeshows don't do much for your customers. Get this insight early, before investing in the feature, by testing some competitive sites that are early adopters.
  • Adding value: Other times you'll identify features or design approaches that dramatically benefit users. In these cases, you can proceed to add this value to your site.

Your team can gain these insights based on the usability work of your competitors. Testing others’ designs can show you the strengths and weaknesses of a design that may have already been through one or several rounds of usability testing or reviews. Your team benefits from the work that the competitors’ teams have already done. Competitive evaluations allow you to test several approaches to the same design – and can be done before doing any design work.

In addition, competitive evaluations reflect what your customers do already.  As they look for information on the web, they compare the content, functionality, and overall experience that your site offers to what others offer. People typically browse in parallel, moving between several sites before deciding which company to patronize. It is beneficial for your team to know what your users see and do as they look around at the competition. 

Defining the Competition

A typical review or test focuses on 2 to 4 competitors’ sites. Any more than that can be too expensive and too overwhelming to analyze. If you have a large number of competitors, do an initial review of several of them to determine which sites:

  • Offer similar content and functionality to your site, or to what your site strives to provide
  • Provide the best overall user experience
  • Use innovative designs that set them apart
  • Are your strongest or most important competitors
  • Are the competitors that your customers are most likely to compare you against 

Don't simply pit your site against those hated big competitors that are your worst enemies in the marketplace. Competitive evaluation is supposed to derive insights that can drive your user experience to the clichéd next level. Often the best insights come from comparing yourself to smaller or more innovative companies. You can also often learn much from companies that are only tangential competitors because they may do similar things in different ways based on traditions from different industries or market segments.

Competitive evaluations aren’t about picking competitors who have poorly designed sites and saying your site is the best. While you can learn a lot about what doesn’t work from looking at poorly designed sites, it is crucial to also learn what works by looking at sites with better usability than your own.

Competitive Reviews

In a competitive review, a usability expert takes an in-depth look at a series of related sites. The expert is looking for relative strengths and weaknesses, trends, patterns, and differences. Looking at similar content across several sites can help identify holes in your content or functionality; holes on others’ sites may inspire an addition to your site.

Competitive reviews can be as broad or narrow as desired. They can dive into a particular feature or area, or encompass an entire site.  You may look only at ordering processes, or you may review the overall site experience. In any broad evaluations, it is best if some key areas are identified to help focus the review and its results.

Competitive Testing 

In a competitive test, an expert runs a usability test on your design and on the designs of your competitors.   Each study participant typically completes a series of tasks on 2 or 3 different sites. Testing more than 3 sites with any given participant can be overwhelming to the study participants. Alternate which sites are paired together and which is tested first for each user.

Usability tests are focused on specific user tasks and can give an in-depth look at key areas and functions of the site. You want to see what users do, rather than ask them to look at the site and express their opinions. Focus on representative and key tasks that can be completed across sites, as well as on tasks where sites offer different approaches to information, functionality, or design.

At the end of each session, ask participants to compare the sites they used. People often naturally do this as they start to use the second site, with the experience of the first site still fresh in their minds. Comparing the two designs can help users verbalize what was clear or confusing in each design and can help you gain further insight into strengths and weaknesses.  

Competitive Tests on Design Variations

Competitive evaluations can also be completed with variations on a single design developed in a parallel design process. Not sure which of several designs is best? Evaluate them and find out. 

Rather than sitting around a conference table arguing about the best approach, do some usability testing or an expert review to determine which design works best for users. Competitive evaluations, like any usability evaluations, can be done in the early stages of design. Designs don’t have to be complete – or even interactive – to be tested. Depending on the design, paper-prototype testing can get the results you need before you spend any money on development.

Analyzing Results

By collecting metrics, a “winner” can emerge from competitive testing or competitive evaluations. This can be based on success scores, time on task, users’ subjective ratings, or a scorecard developed to analyze findings. However, the main goal of a competitive evaluation is not to declare a winner. The goal is to improve your design. While metrics can emphasize the point that your site performed, say, 30%  better – or worse – than your competitors’ sites, the important information lies in the details of what worked and what didn’t across designs.

Focus on what’s relevant to your design questions or challenges. What are the biggest strengths of competing designs? What trends arise across sites – and are they good or bad? Did any sites offer unique solutions to a common problem? How did they perform? Is there any opportunity to stand out from the competition? Were there any gaps in the information, content, or features that the sites provided? Don’t forget to look for opportunities beyond what other sites offer. You want to beat the competition, not copy them.

When writing – or reading – a competitive report, keep in mind that what matters is the information that can help you improve your site. Knowing what worked well and why it worked well can help direct design decisions. Knowing what failed and why can help teams avoid others’ mistakes or fix their own.  A good competitive evaluation not only assesses the competition’s designs, but also translates that assessment into recommendations for the site, whether written in the report or brainstormed with the team. The results need to help you design your own solution.

Summary

Competitive evaluations can give your team the benefit of reviewing several sites’ designs, to understand what the competition is doing and what they’re doing well. Learning from others’ designs can help your team create a better site.