
I don’t often talk publicly about what I actually do on the job. Because company privacy and whatnot.
But, I have a project that I’ve been working on for over a year and have started presenting externally at conferences.
While I can’t share my code or even share my company data, I can talk through (at a high level without many specifics) how I approached the project end-to-end.
So I bring you: Measuring the User Experience and the Impact of Effort on Business Outcomes
You can watch a presentation of this that I gave at the PyData Global Virtual Conference or keep scrolling to read the overview.
What’s the problem?
Every project you work on should start with a problem. Otherwise, what’s the point? What do you have to work towards? How do you know when you’re done? And how will you get buy-in if you’re not actually solving any real problems?
I work in tech, specifically in product analytics. Basically, I analyze how people use digital products - every page load and interaction is a data point I’m interested in.
And the problem we were wrestling with was: How much effort does it take our users to complete a task? If we make a change, have we increased or decreased effort? Does it matter?
We had no idea.
There are some common ways to measure digital experiences:
Acquisition: Are new users coming to your product (website, app, etc)?
Activation: Are new users converting?
Retention: Are users coming back?
Referral: Are users helping to attract more users?
Revenue: Are users upgrading or spending more money?
But if one of these metrics changes - do you know why?
Why do we care about effort?
There are multiple apps that can do the same thing. If you have a bad experience with Uber, you can use Lyft. If you don’t like GrubHub, you can use DoorDash. If you don’t like Expedia, you can use Booking.com. If you don’t like the Apple Watch, you can get a Garmin. If you don’t like the iPhone, you can get an Android.
Even our cars and household appliances have digital interfaces that are supposed to improve our experience but if it doesn’t work as expected, it can be very frustrating.
User experience can make or break a tech company. If you don’t think so, consider that at one point, Blackberry dominated market share for smartphones. These days, does anyone have a Blackberry?
In addition to abandonment, effort can also be a leading indicator of contacting customer support, leaving negative feedback (sometimes publicly), lower return rates, fewer new users, and client or customer churn. So it’s in a tech company’s best interest to pay attention and reduce effort.
How do we typically measure effort?
Surveys and other feedback are a great source of information. Users will tell you exactly what they’re thinking, you don’t have to infer anything. Sometimes they will even alert you to problems you didn’t know existed.
But feedback data can be sparse, biased, inconsistent, and costly to collect.
Look at the behavioral data
Every tech company has tons of behavioral data, and there are many things you can look at that can signal higher effort or a bad user experience.
Errors
Repeating a task
Visiting Help information
Using Chatbot
Longer time on task
Going back and forth
Repeated clicks
And other things that are specific to your company’s online experience
How to evaluate effort
Not all friction feels the same to end users, so you need to figure out what can actually signal an issue. Use correlations and predictive models to understand the relationship between friction and key outcomes for your business. Use this to create a calculated score that reflects user effort.
(Yes, I’m intentionally leaving out a lot of details on this step. Gotta keep some stuff proprietary.)
How can this improve the business?
The first benefit of this project was finding gaps in our data collection.
The second was finding pain points in the user experience.
Beyond that, we were able to use this information to inform future projects on the product roadmap related to personalization and automation.
(Sorry, gotta be intentionally vague again.)
What do you think?
How you ever gotten frustrated by a user experience?
How would you measure user effort?
And what would you do with this data?
Tell me in the comments!
Love this summary of measuring user effort from a Product Analytics perspective. Takes me back to one of my favorite projects, where we built a tweaked Sankey diagram to emphasize loops in the user experience. Great catalyst for UX collaboration and product action.