How to choose metrics to measure product success

Metrics are the backbone of a product life cycle. It helps in defining success and measure product/feature performance. In more technical terms we also use KPI — Key Performance Indicators, interchangeability with metrics. There is already a lot of content about all kinds of KPIs for products so this article is more about the practical process to choose the KPI that matters. First we will talk about different KPIs

• At different stages of user life-cycle
• For different kinds of products

We will look at these KPIs with the objective associated with their measurement. We will try to define a process to arrive at the important KPI in a more general manner. Simultaneously we will also carry a practical case following the process to verify the effectiveness of our process. The practical case we will be trying to solve will be

“How should we measure the success of the ‘save’ feature on Facebook”

Metrics and User life-cycle

For a product a user can be at different stages of the adoption funnel. The famous Pirate framework AARRR ( Acquisition, Activation, Retention, Revenue, Referral) is one of the most popular frameworks for the user’s life-cycle through the product. Although each of these stages have their own micro-steps, they are defined by certain milestones for the convenience of measurement. Different metrics can be defined to keep track of these stages. Another popular framework with elements of user satisfaction is HEART (Happiness, Engagement, Adoption, Retention, Task Success) Framework. They are a set of user-centric indicators of user experience. Several measurable metrics can be defined to gauge them as well.

AARRR Framework

Acquisition

The first stage in the user life-cycle is acquisition, when the user comes across the product. As said earlier depending on the product, goal and statistics a specific milestone can be defined for quantification. Like the first time an app is downloaded or a website visit or a sign-up depending on the sector and the product. The important metrics at this stage of user life-cycle can be

• Acquisition Rate [New User/Existing User]: Portion of users are composed of new users
• Acquisition by Channel [User through channel A / new users]: Channel contributions to new users
• Traffic by Channel [Traffic on channel A / Total traffic]: Channel contribution to traffic
• Acquisition cost by channel [Per user activation cost through a channel]

Activation

It’s most frequently described as the moment a user has their ‘Aha moment’, the moment they realize the true value of the product and find a coherence between the value proposition and their problem/desire/need. This moment does not always have a clear demarcation and can come at different moments of user journey for different people. A certain measurable landmark is generally set as an indication of activation like for Facebook this moment generally occurred when a user acquired 7 friends in 10 days. The important metrics at this stage of can be

• Activation Rate [New active user / New users]: Activated users per new user
• Time to Activation: The average time an activated user takes to activate
• Actions/Interactions to Activation: The average number of interactions or actions a user takes for activation
• Daily New Active user [Daily New Active user/Total Active users]: Proportion of daily/monthly/yearly active users among total active users.
• First time rate [Actions being performed for the first time by the user/Total number of actions]: The percentage of total actions being performed by the first timers

Retention

Retention refers to the consistent use of the product by already activated users. The churn can come for various reasons and the related metrics helps us to keep track of it. Indicators at retention level is very crucial since it is much cheaper and easier to sell something to an existing user than acquiring a new one.

• Retention Rate [returning user/starting activated user]: The percentage of users retained over a given period of time.
• Average time to churn: Average time a users takes to churn
• Upgrade Rate[Upgraded users/activated users]: For a product with upgrade features or features that clearly demarcate users using higher functionality features.

Referral

It measures how much the current users are willing to recommend the product to someone else. Referral brings organics growth and is extremely critical for rapid growth. Incorporating natural referral channels intrinsic to the product features is one of the major verticals on hyper growth strategies. Most common measurements

• Net promoter score [promoters(%) — detractor(%)]: To the question of “How likely you would recommend the product to your friend or colleague”, Promoters respond 9–10, passive respond 7–8 and detractors respond 0–6.
• Viral Coefficient: It’s the “number of new users a customer refers to the product’’.
• Average Satisfaction Score: The satisfaction score measured from surveys or rating after the use of a feature or important function of the product.

Revenue

Revenue is critical to the success of an enterprise and as described in the article, monetization policies are crucial for the business to survive or thrive. Revenue can have several forms and can be business dependent. Most common metric would be

• Customer Lifetime Value (CLV): It’s the total amount of money a customer has during its lifetime with the product. The exact formula can be different for every business. CLV with customer acquisition cost (CAC) plays a very important role to measure the survivability of the business. A thumb rule for growth is a 3:1 ratio between CLV and CAC.
• Revenue per user [Total revenue/users]: Total revenue averaged per active users.
• Average Revenue per User
• Monthly recurring revenue: Monthly revenue generated through active subscriptions.
• Others

HEART Framework

It’s a framework developed by UX research experts at Google to measure different aspects of user experience and define success for different objectives. The acronym comes from H-Happiness, E-Engagement, A-Adoption, R-Retention and T-Task success. All these hold critical importance in overall user experience and help to structure and quantify goals and define success.

It’s generally used by defining goals, signals and metrics for each of these 5 factors. Goals are the objective, signals give the indications about the user proceeding towards goal and metrics are the measurable quantifiable to track.

Happiness

It represents how users feel about the product and it has similar metrics as to referral like

• Net Promoter score
• Average satisfaction score
• Ratings in stars or points

Engagement

Engagement refers to how engaged the user is while using the product. The signal for this metric is the time spent on the product. The metrics can vary depending on the types of products but generally they are

• Visits per user per week
• Average unique session length
• Average session length per visit.
• Important interactions per user per week
• NPS (Net Promoter score): In practical terms it is used to measure engagement as well since the user is not really thinking about referring to someone while evaluating. Their scores are mostly based on how much they like the product.

Adoption

Adoption is the same measure as Activation as the AARRR framework. It is the user who released the value of the product, who has had that ‘Aha moment’. The associated metrics are also the same activation

• Adoption Rate
• Time to adoption
• Product interactions to adoption

Retention

We have already talked about retention while discussing the AARRR framework and it refers to the longevity of the user with the product.

Task Successful

A successful completion of a task has a huge impact on retention. It has to have a very critical balance between how difficult and easy the task is. In addition to that ease of doing the task for the user matters well. Concepts of flow very precisely describes the interaction between complexity of a task and the skill needed to accomplish it. If the user does not belong to an adequate skill range needed to accomplish the task they either have anxiety or are bored. The associated metrics are

• Action Success Rate [Task completed/total task started]: It measures how easily a user comes to successfully complete a specific action.
• Time to action : The average time a user takes to successfully accomplish an action.
• First timer’s average time to action: It measures the average time a user takes to accomplish a specific task for the first time

Growth Loops

Growth loops a more recent concept where a product is designed in a way to generate new traffic organically. The user itself becomes the biggest advocate for the product and their investment in the product creates values that bring more users to the product. Most of the social network products leverage these features, leading to explosive exponential growth in a short timeframe. A precise product-market fit is an important criteria in these cases which otherwise gives false indications of rapid growth followed by a rapid plunge in active users with minimal retention. It’s generally very different for different products but generally it consists of 3 stages starting with “Entry” of a user, followed by a set of “Actions” for engagement and finally an “Output” that stimulates “Entry” of a new user. Further information can be found at reforge.

Input

It can be sign up, visit or some other point of entry for a new user and can be measured by different metrics like daily signups.

Action

It’s a set of interactions and it can vary with industry, strategy and product.

Output

This is the crucial step which makes it a loop, where an existing user’s natural action brings new users like a photo from an existing user on Pinterest, increasing the content on the platform, finally leading to more new users when they discover the content online. Measures at this stage could be the quantity of output activity per user, new users gained through an existing user’s action and other measures of this kind.

North star metric

Finally, the most important, “North Star metrics” represents the core problem the product is solving and how it brings revenue to the company. It’s the number a company uses to measure its success and focus to improve. It’s a metric which is least impacted by an external factor, revolves around the final value proposition of the product and is the basis for teh teh companies growth strategies. It totally depends on companies for example

• Facebook NSM: Monthly Active users
• Spotify: Time spent listening
• Whatsapp: Messages sent
• AirBNB: Monthly nights booked
• Amazon: Number of purchases per month
• Uber: Rides per week

Metrics and Industry

The metrics at different stages could vary a lot based on industry and type of product. A few important industry specific metrics for some industries have been listed to have some idea about what kind of metrics could matter and help align our thinking. A detailed description of these metrics can be found here.

E-Commerce

• Average order value
• Abandoned cart
• Average product shipping cost
• product searches
• Shipping time

API products

• API calls
• Tickets raised
• Downtime
• Customer count

Content Generators

• Content view rate
• Email subscribers
• Bounce rate
• Content share rate

SAAS Products

• Leads
• Visits to lead percentage
• Monthly Recurring revenue

Mobile applications

• Session interval
• Download
• In-app purchases
• Uninstalls

Process to Choose

There is of course an enormous choice for metrics and with growth analytics we can literally measure everything. This reinforces the importance of choosing the right metrics which quantifies the parameter intended. The process involves understanding

• The core value propositions of the product and its objective
• User life cycle and user journey
• Possible metrics for measurement
• Evaluate these metrics

Product

Facebook is a social network product with 2.89 billion Monthly active users and 1.9 billion daily active users. The largest demographic group ages between 25–35. Some other statistical facts about Facebook:

• 81% of user only access the platform via mobile device
• On an average use spends 34 min on Facebook daily
• There are more than 4 billion video watches on Facebook every day.
• 85% of videos displayed on Facebook are consumed while being mute.

Goal

Global Objective: The north star metrics for Facebook is monthly active user (MAU). Facebook is a social network, so active users are the most important asset for Facebook and focal point of the growth strategy.

Save feature: Save feature lets users save a post or a video to watch for later. They can make a collection of saves connected in different folders and share the collection with other users as well. The goal is definitely user engagement since it’s unavailability cannot be the reason for churn and the user is already an active user. Rather it can lead to an organic engagement since it can be shared among friends. In addition, having a collection of posts or videos would also increase the user’s online asset with Facebook like shared posts, videos and other content which leads to better engagement as well. It also gives a strong feedback about users preference on connect and posts.

User journey and life-cycle

The save feature responds to the need to access content again in the future. The possible cases when this feature could be useful

1. When a user likes a content and wants to see it again later.
2. When a user watches a video in mute but wants to watch it later with volume.

The user will go through the life-cycle from acquisition to referral. Metrics for each stage can be defined to evaluate the performance of the feature in addition to the overall impact of the feature on Facebook.

Metrics

• Daily new user for the feature [new users using the feature / total number of active Facebook users] (Acquisition): This measurement can show how many users start using the feature after it’s release. It would measure the acquisition ease of the feature.
• Average time to first use : This will show the organic need of the feature in the customer’s life. How often the user felt the need for this feature when it was not there.
• Average number of content in the collection per active user (Retention): This would be a good measure of retention since a higher number would show a regular usage by the user.
• Number of interactions with the saved content per user (Adoption, Activation): After a few saves once a user sees their saved content again a couple of times, this could be the ‘aha moment’ for them, when they realise the value of saving the content.
• Number of users with collections shared with someone else as well (Referral, happiness): This would be a great measure and means to bring new users to “save” feature. It organically encourages a new user to make their own collection and add content to the collection of theri friends as well. referral, happiness
• Daily active user: This can be the metric to measure overall success of the feature.

A few precision would be needed like defining active users, interaction within the feature and others based on the more precise details of the application.

Conclusion

Measuring success is extremely important in defining success and alignment of a product. With increasing digital tools and functionalities it’s becoming easier and easier to track and measure parameters and even easier to get lost in figures. A systematic and structured approach gives us tools to use these tools to its fullest. Clear understanding of our customers journey and life-cycle can definitely simplify the road to organic growth and success using the right figures and numbers.