Metrics are divine. Metrics are worthless. What’s the truth?
A dominant aspect of product culture is our obsession with and divisiveness around metrics. Metrics can be a super-power or a major vulnerability. I’ve observed that individuals, and even full teams, tend to fall into one of two camps:
Metrics are worth everything
Metrics are worth nothing
These views lack nuance. Metrics are at once blunt instruments that must be contextualized – and powerful tactics to achieve a goal.
I wanted to share a note warning of the risks of these absolutist views, and highlight ways to avoid them.
Risks of over-fitting to metrics
Some teams seem ready to hire experimentation tools as their PM. But to paraphrase wisdom from a Facebook product executive:
“Do not outsource your product intuition to experiment results.”
Before we place our unadulterated faith in metrics, we should reflect on why they exist in the first place: Metrics are proxies for a goal.
Products exist to solve problems for or add value to people, or simply to achieve business objectives. In either case, products ladder to an aspirational mission, like helping people entertain each other or making meaningful connections between people and businesses.
Since missions are intentionally abstract, we create metrics to measure our success achieving them. The metrics get ingrained in the way we prioritize what work gets done, and tell us what’s working and what’s not. This makes metrics operationally relevant to our day-to-day in a way that missions often are not. Still, we can’t lose sight that the hierarchy between a mission and a metric is irreversible: the metric should evolve to fit the mission, the mission shouldn’t change to fit the metric.
The phenomenon of prioritizing metrics at the expense of goals is understood in academia. A UX researcher who also has a PhD in cognitive psychology, once surfaced two social science laws to me, and they stuck:
Goodhart’s law: When a measure becomes a target, it ceases to be a good measure.
Campbell’s law: The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.
These resonate with my product experience. I once worked on a product team whose goal was to increase brand equity. For a metric, we chose a sentiment measurement instrumented by surveying users each day.
We developed a playbook for moving the metric. Among the most effective tactics was using the exact language of the survey question within in product copy. That tagline became prolific. Every product cycle, copywriters would suggest alternative framing, and every time PMs like me would shoot them down.
It’s no wonder: If you continually make a statement to people, a small proportion of people will respond differently to a survey that uses its language verbatim.
Now, the fact that people responded differently to the survey may indicate we were changing their hearts and minds. You could also argue this was the reasonable manifestation of a strategy to create the brand we desired, not unlike any company plastering their tagline across all their marketing. But on its face, it’s likely we were overfitting our strategy to a narrow sentiment metric rather than achieving our goal to build a sustainable brand foundation.
Risks of writing-off the data
Yet, it’s as bad a mistake – or worse – to ignore data we don’t like or fail to collect sufficient signal.
We’re all understandably passionate about our intuition. Companies hire talented designers, creatives, and product thinkers for their minds. It feels good to go with your gut, and even righteous to claim that we’re doing the “right thing” in spite of metrics.
What we forget is that numbers on a dashboard are not abstract concepts, but aggregated views of real people’s behavior. The ability to see how thousands or even millions of people are interacting with something in a single chart is an almost inconceivable innovation.
When a number regresses, we shouldn’t think of it as a chart inflecting, but as a shift in behavior of a lot of people at once.
Building and shipping things blind to metrics is a form of narcissism. It means you’re prioritizing what you like rather than what your users empirically value.
Embracing nuance
The remedy for these extreme perspectives is embracing nuance.
First, take time to pick the right metric. In my opinion, this is the single most important thing a PM does because it is the biggest influencer in determining what work gets done and what work survives.
In the context of avoiding metrics that incentivize the wrong thing, pressure test a metric by enumerating what you would do to game it. If required, flag counter metrics that will protect you from hitting your metric at the expense of your goal.
And while every team member should own this, I challenge product executives to consistently emphasize accountability to underlying goals; we don’t want people to perceive a choice between doing the right thing and a good performance rating.
Second, always contextualize data. Your obligation is to contextualize information, and pursue a product strategy that marries data with principles. Take a quantitative insight and state in sentence form what it means. Uncover nuance behind numbers through qualitative research. If you’re advocating to proceed in spite of a regression, explain why. If you’re advocating for doing what the data say in spite of misalignment with intuition, explain why. Don’t move on until you (and ideally, your team) feel convicted. Always be intentional about the relationship between data and your decisions.
Finally, and most importantly, use common sense. It’s often obvious, if unspoken, when we’re going down a path that’s good for numbers and bad for the goal. I find people with fresh eyes are particularly insightful at recognizing this because they lack institutional baggage and disillusionment. And here’s a litmus test for PMs: The harder you have to work to justify your strategy, the worse it probably is.