What do you do in Product Analytics?
I’ve been working in product analytics for over 5 years (and web analytics for longer than that). It’s a field I really enjoy and I feel like there is a lot of opportunity - new digital products are introduced all the time, and existing products continue to grow and evolve and collect more data to be used.
But what do you actually do in a product analytics role? What do you need to know? What type of projects do we do - or you can demonstrate into your portfolio?
What do you do in product analytics?
At a very high level, you help the product team use data to make decisions.
What does that usually include?
Data collection: Make sure your company is collecting useful data to answer questions and understand user behavior.
Define key metrics: What will tell us that our product or business is successful?
Data analysis: Answering the many big and small questions from the product team (and other adjacent teams).
Experimentation: Most teams also do A/B testing to make better decisions about product features.
Let’s get into each of those …
Data Collection
You can do all the fancy analysis you want, but if you have bad data, it’s not going to help. Often it is up to the product analytics team to work with engineers to make sure we are collecting the data we need to answer questions. Data doesn’t just magically appear, even for digital products, and it certainly doesn’t magically appear in a clean, consistent format.
Every page load, every interaction, every search, every order, etc etc etc - we want data to tell us in excruciating detail what the user did or saw.
To that end, some teams use a tag management platform to automate analytics tagging to collect data. Other teams work directly with engineers to implement tagging to collect user behavior data. Regardless of approach, you also need to have good documentation (often called tagging specifications or a tagging spec), and it’s often up to the product analytics to write it.
Defining key metrics for product analytics
Once you implement your robust product analytics tagging and start collecting large amount of data, you need to figure out …
What data will tell us that we have a good product?
What are the signals that we’re making or going to make more money?
How do we define the above via a consistent metric? What is the exact definition and calculation we want to report at regular intervals?
Consistency is important! It’s not uncommon for different teams to have different definitions of the same metric. And this is often why your colleagues complain to you that “the data doesn’t match.” They are probably using different definitions or calculations.
When it comes to product analytics, there are a lot of key metrics you could consider. Which ones are important will depend on the purpose of your product.
Conversion: Of all the people who came to our website/app, or all of the sessions that started, or all of the searches that were submitted … how many ended with an order or signed up for a thing that makes us money? Usually, this is calculated as a rate or percent and you want this number to increase.
Revenue per order (or per visit or per user): How much money do we get? You also want this number to increase.
Churn: How many customers did we lose? What percent of total customers does this represent? You want this number of decrease.
Customers: How do we define a customer? Is it all visitors? Anyone who created an account? Anyone who placed an order? This will vary depending on your product. But you usually want it to increase.
Lifetime value: How much money do we get on average per customer? You want this number to increase but it can also inform how much money your company spends trying to attract new customers.
Fallout (or Dropout): How many users (or sessions) exit the funnel before converting? Where are they exiting?
There are other metrics that you can track - let me know if the comments what you’ve looked at.
Once a company agrees on key success metrics, it might be the product analytics team’s responsibility to automate reporting (usually via a dashboard), or they might partner with a business intelligence or data engineering team on automation - it depends on the company structure.
Product Data Analysis
In addition to tracking key metrics, you also do a lot of other analyses.
This could be for small decisions:
Should we include X field in the new version of our search forms? Does anyone use it?
How do users enter the funnel for X task? Do they click this button or navigate via the menu?
Do all users use X feature or just certain user roles?
Etc.
Or for big decisions:
What should be on our product roadmap? What do our users want?
What are the different personas (cohorts) of our users? How do we define and identify them? How are their needs different? Who is more profitable?
Where are users getting stuck/blocked on our product? When are they abandoning (or falling out or dropping out of the funnel/path)? Where should we focus on product improvements?
Or big questions:
Why did one of the key metrics decrease last month?
Why are users abandoning before completing a task/order? How can we stop that?
Why are users churning? How can we stop that?
What tools you use will depend on the amount of data you’re working with, how your company stores its data, and what skills you have. Over the past 8+ years of working with digital analytics, I have used various combinations of Excel, Adobe Analytics, Google Analytics, Adobe Target, SQL (in Snowflake and AWS), Tableau, Power BI, Python, and R.
I don’t recommend learning all of those tools to try to land your first job! Excel, SQL, and Tableau (or Power BI) are a good start. Add Python if you’re comfortable with those three. (What specifically do you need to know in each tool? Read this.)
A/B Testing
Most product analytics do some amount of statistical testing - often via A/B tests - to help product teams make decisions. This hypothesis testing the statistics field and also called Experimentation or Test & Learn. Some teams use automated tools like Adobe Target or Optimizely or something similar. Other teams use a more manual process working directly with engineers to create the different versions, randomize users, and then query their own data and use statistical packages via Python or R to calculate the results.
The end-to-end process for the product analytics teams typically includes some or all of these steps:
Creating a test & learn culture. You need buy-in from the product team to agree to do tests. They “own” the experience, so tests aren’t going to happen without their oversight. They need to understand the benefits of experimentation, and as a data expert (who should know at least some basic statistics), you can get this buy-in via education.
Designing and launching good tests. The product team usually comes up with the test ideas, but often the product analytics team helps to make sure they are coming up with a good hypothesis, they are testing things that can be tested within a reasonable time period, and they are picking the right success metrics. This might also include making sure you are collecting the data necessary to calculate test results, that the test has launched correctly (in the right part of the UX and with the right users, and they are randomly split into control and treatment groups).
Proper statistical analysis. If you use an automated tool, this part is often done for you. If not, then you need to make sure you are querying the correct data, doing the correct cleaning and aggregations, applying the right statistical methods, and interpreting the results accurately.
What skills are important for product analytics?
So you want to work in product analytics, what do you need to demonstrate?
Technical skills:
Excel - even with all the fancy tools, if I’m working with a small amount of data to answer a quick question, I’m going to use Excel.
SQL - you need to know it to get the right data for your analysis or dashboard. Often, you won’t get very far in job interviews if you can’t pass a live SQL assessment.
Tableau or Power BI - this can vary depending on team structure. Some roles will use these all the time, other roles might not. If you’re in more of a “reporting” role, then you will use one of these or something similar.
Python or R - this can vary depending on team structure. Some roles will use these all the time, other roles might not. If you’re in more of an “advanced analysis” role, you might use one of these pretty regularly.
Product Analytics platform - Adobe Analytics, Google Analytics, Amplitude, Mixpanel, Pendo, Heap, etc … most product analytics teams will use one of these to automate their reporting. Some companies prefer to hire candidates who already have experience in their platform of choice, however, it can be hard to find candidates who know a specific platform, so often it is enough to know any product analytics platform. Many of these have a free version you can use on your website or blog (such as Google Analytics) or a demo account you can check out (such as Amplitude).
Math skills:
Descriptive stats: mean (average), median, mode (count), quartiles, min/max, distribution.
Hypothesis testing: sample size, conversion rate, standard deviation, outliers, confidence interval, p-value, t-test.
Predictive analytics: typically this only comes up for more advanced analytics or product data science roles. But for certain types of analysis, it is useful to know regression or tree-based or nearest neighbor models.
Business or “soft” skills:
Good communication: You will be presenting your work. You will need to explain why we define this metric in this exact, specific way or why we can’t just throw around the word “significant” when discussing data or why you are spending your time on this analysis and why it is important to the business or why the product team should pay attention to your analysis.
Business sense: Why is this specific metric important for the company to track? How does it align with bigger goals? What important questions is your analysis answering? What business or product recommendations can you make from your analysis?
Initiative: Your product team is good at product management, not data collection or data analysis. You need to come up with ideas for how to answer their vague questions with valuable insights and recommendations. You also need to be able to work without a lot of hand-holding because they don’t know how to do your job. And if you’re lucky enough to report to someone with an analytics background, they are probably busy. You aren’t building the product - you are overhead. Your “team” is small.
Project management: On any given day, your To Do list includes tasks for data collection, analysis, experimentation, presentation prep, etc. You need to be good at juggling timelines and priorities for different projects and tasks. As I said, your boss is busy and your stakeholders don’t know how to do your job or how long your tasks take.
Want more? Here is a video of me being interviewed by a data science student about my job in product analytics.
Also check out these online courses:
Coursera: Product Analytics and AI
Coursera: Customer Analytics (there is a lot of overlap)
Uplimit: Product Analytics (I’ve connected with Elena via online data communities and she knows her stuff!)
Do you work in product analytics? What advice do you have for folks interested in this field? Let us know in the comments!