A metric is only as useful as your ability to measure it

The most important attribute of a metric is that it tracks well with your goal. The second most important attribute is that you can effectively measure it. I want to share lessons from picking a metric that aligned well with our long term strategy, but was challenging to measure, track, and move. To summarize: Metrics need to be not only strategically valuable, but also feasible to use operationally. Consider metrics that:

1. Measure impact over a holdout group, rather than goaling on a topline number

2. Measure absolute numbers, rather than proportions

3. Measure cohort effects or conversion, rather than acquisition

What were the issues with our metric?

Our goal was to drive adoption of a product. Specifically, this meant that we tried to inspire each of our users to adopt a feature once in their lifetime. We derived a lot of value to our platform when this happened, so this adoption represented something very meaningful. And yet:

Even if a metric is strategically valuable, if you can't operationalize it, it's a poor metric.

This was exactly our problem. Our metric was hard to measure, track, and move. Here were the issues:

1. It was vulnerable to factors outside our control: Our metric measured the number of users who adopted our product divided by the total number of users on our platform. This exposed us to changes in the user mix. New users are least likely to have adopted a specific feature, so as growth accelerated for our platform, our metric sagged. This was good in some ways (it encouraged us to focus on experiences for new users), but the effects were so large that our team's efforts were overshadowed.

   1. Recommendation: Goal on the difference between a test and holdout group. While it's standard practice to measure the impact of tests using a holdout group, teams choose whether their goal will be the topline metric or the delta between test and holdout groups. Topline goals force teams to confront external factors, whereas goals focused on impact over a holdout isolate the team's impact. Had we goaled on the difference between a test and control, the changes in user mix would have been irrelevant given they'd impact both groups equally and we could've understood our team's contribution more clearly.

   2. Recommendation: Measure using absolute numbers rather than proportions. Rather than proposing we increase adoption +10%, we could have proposed increasing adoption +X, where X is equal to +10%. These two numbers represent the same thing, but the latter wouldn't have been muddled by changes in user growth.

2. It was a lagging indicator of progress: We needed to convince users to do something once. Adoption goals are often lagging indicators; in other words, adoption metrics move after a trend change has occurred. Our changes could bend the adoption curve, but net very small benefit immediately. Consider a change made on the last day of a measurement period (e.g. a quarter or year) that makes new users 2x likelier to add your product on their first day: it would have no immediate impact on adoption even though it's very valuable in the long term. We can try to forecast long term impact, but extrapolating trends is imprecise and time bounding “long term” is arbitrary. There is no perfect replacement for adoption, but there are relevant alternatives depending on the team's strategy.

   1. Recommendation: Measure cohort behavior. When trying to change new user behavior, you can measure cohort differences. If our goal was to ensure this week's new users were adopting our product at greater rates than last week's new users, we could use cohort analysis to track just this.

   2. Recommendation: Measure conversion: If the goal is to fix a leaky funnel, you can measure conversion. For example, we know that some number of people an adoption flow each month where we prompt them to use our product. We could measure what proportion of the people who start the flow take the desired action. Notably, this lacks incentive to drive top-of-funnel traffic (which is sometimes the most effective lever). For that, adoption is the only option.