We normally assess the impact of a redesign by comparing the before and after values of a certain measure, sometimes called a KPI (key performance indicator). The measure could be task-completion time, number of sales, number of customer-service calls, number of questions on a support-site forum, or any other metric that has some direct or indirect effect onto the site’s business value, because it raises revenues, lowers costs, or improves employee productivity (for intranets and enterprise software).

The improvement score is usually defined as the ratio between the two measurements of the metric (before and after) minus one. Thus, if the before and after measures were identical, then the ratio would be 1.0 and the improvement would be 0%.

Thus, if a redesign for usability reduced the number of questions on a support-site forum by 70%, then it means that the number of questions after the redesign was only 30% (or 0.3) of the number of questions before. For example, the original design might have caused 500 questions per day, whereas the redesign only causes 150 questions per day.

So, for this metric, the ratio before/after would be 1/0.3= 3.33. Now, to compute the improvement score we have to subtract one from this ratio: 3.33-1= 2.33. If we convert this number to percentages, we obtain an improvement score of 233% due to the redesign of the support site.

Note that some metrics are “bad” in that higher scores indicate lower usability, while for “good” metrics a usability improvement means a higher value of the metric. For instance, for task-completion rates or for conversion rates, a higher value is better, whereas for number of questions or task-completion time, a lower value is better.

The improvement score has to be computed differently for “good” and “bad” metrics:

  • For “good” metrics (such as sales), the improvement ratio is the new number divided by the old number. For example, an e-commerce site recorded a conversion rate of 2% of visitors before the redesign and increased this to 5% after the redesign. In this case, the ratio would be 5/2 = 2.5, for an improvement of 150% in the conversion metric.
  • For “bad” metrics (such as time on task), the improvement ratio is the old number divided by the new number. For example, employees used to take 4 minutes to find a phone number on an intranet but only need 3 minutes for that same task after the redesign. Here, the ratio would be 4/3 = 1.33, for an improvement of 33% in the productivity metric.

For more on how to measure design improvements, see our full-day course Measuring User Experience.