Acumbamail UX ROI case study published by Nielsen Norman

5 November 2020

This week, Nielsen Norman Group published the 5th edition of their UX Metrics and ROI report. We are honoured to be featured in it with a case study related to our business unit Acumbamail, alongside brands such as Atlassian, Shopify and Luxottica.

The Acumbamail case

The case study covers a strategic experiment carried out jointly by the central Group Design department and the local Acumbamail Design team, with the final aim to improve usability and ultimately business metrics on the Acumbamail website.

The goals were set as follows:

To improve conversion rates both to paid subscriptions and to free accounts

To enhance the frictionless approach
To solve all the usability problems in the step between the pricing page (or the primary CTA in the website) and the landing on the platform.

The process involved a first step (testing to understand), a second step (redesign), and a third and final step (test to validate).

Problems discovered thanks to usability tests

The password creation instructions were not in contextual proximity of the microtask area (usability problem)
The shape of the pop-up prevented people from finalising the task when the screen resolution was slightly lower than normal (business problem: it prevented people from converting to the product)

The solution

Once the problems had surfaced, we moved on to redesign the affected website areas and to test whether the changes brought the desired results.

The redesign involved two actions:

Forcing the final pop-up to stay above the fold
Reshaping the pop-up modal in order to accomodate side by side input fields and suggestions/feedback for password creation

The design metrics we tracked

During the different phases of the project we tracked some specific metrics with specific tools. Quantitative usability testing and Analytics were the main tools we used. The metrics we tracked and monitored were:

Subscription time on task: -10%
Subscription error rate: -48%
Subscription Conversion rate: +29%
Subscription NPS: +5000% (from 1 to 51)

Why is UX ROI so important

Design is a way for improving business KPIs. Design has a quick ROI and a long term ROI. The first is perceived as soon as you get an easier to use or more effective interface or service. The second is perceived when business KPIs are altered by design actions somewhere else.

The tracking of those ROIs has an effect on many levels:

It demonstrates the value of our design efforts
It shows how design changes impact the bottom line
It can increase the trust on design and designers
It increases the design maturity of our organisation

So no matter the size of the revenue, a good design improves what matters to our users and business.

About the report

The report is the 5th edition of a collection of 44 real-life case studies highlighting how teams have used UX metrics to assess the impact of their design changes.

The 297-page report also includes advice for applying these concepts to own work, as well as detailed quantitative analysis of the 99 total case studies collected for all editions of this report over 14 years.

At the end of the theoretical part, the report lists 44 case studies from different industries (Enterprise & B2B, Nonprofit & Education, Healthcare & Insurance, Entertainment & Tourism, Transportation & Real Estate, Ecommerce, Finance, Intranet/Internal, Miscellaneous)

Each study includes: