Insight from Cohort Analysis
In this article we’ll peel back the onion on vanity metrics to understand the health of the business and make informed strategies that ensure long-term success and data-driven decisions.
“Look at this photograph
Every time I do, it makes me laugh”— Nickelback
(Yes, I realize quoting Nickelback may have ostracized half my audience, but bear with me as I explain) Chad Kroeger may have been on to something when he wrote the lyrics to “Photograph,” fondly reminiscing an event that, at the time, had dark undertones. As is with life, businesses tend to paint the rosiest picture to win customers, recruit the best people, and generate funding using industry-standard KPIs — revenue, ad impressions, new customers, customer growth etc — and encouraging practices that improve those numbers. And while KPIs may be factual they may not be truthful, thereby becoming vanity metrics.
Vanity metrics are metrics that make you look good to others but do not help you understand your own performance in a way that informs future strategies.
To illustrate my point, take a look at the following hypothetical set of data highlighting three KPIs: product impressions, sales, and click-through rate. Revenue and impressions are structured in the triple-triple-double-double-double pattern we like to see in ideal early-stage company growth models. On the surface everything looks like a slam dunk, which is why we’re going to use cohort analysis on the same set of data to understand if our business is actually healthy and draw insights on improving it.
Introduction to Cohort Analysis
Cohort analysis is a kind of behavioral analytics that breaks the data in a data set into related groups before analysis.
A cohort analysis is used to visually display cohort data in order to help analysts compare different groups of users at the same stage in their lifecycle, and to see the long-term relationship between the characteristics of a given user group. In our example, let’s categorize the vanity Impressions into cohorts:
Not all product impressions are the same. When someone signs up for a free trial, that carries less weight than someone who signed up and spent five days evaluating the product. Therefor they should be represented by different cohort groupings. In the first period (P1) we have 100 impressions of which 35 signed up and never started the trial, another 35 that started the trial but never used, and so on. This seems like a common funnel chart and you may be tempted to investigate abandonment rates (e.g. “Trial Use %”) but these can lead to another layer of vanity metrics and behaviors that distract from problems in the system. Further, cohort analysis is not always funnel-shaped (see example in Appendix). This thinking is too tactical, too specific. Step back and look at patterns, first by converting the table into a cohort visual.
Visualizing Behaviors
“Pay no attention to that man behind the curtain” — Oz
This visualization shows cohorts as the percent of total impressions in a given period. An increase from 2% to 3% in the P1 and P2 Paid cohort does not necessarily mean more customers signed on in P2, nor does it necessarily mean that less customers signed on in P4 when the cohort shrunk from 4% to 3%. But it does demonstrate that a greater percent of the total impressions in P2 paid compared to those in P1, even if P1 had one millions impressions and P2 just 100. This scenario is modeled with constant growth so you need only focus on changes to cohort size to interpret business patterns. In reality, your mileage may vary.
Here we see three interesting patterns:
- The percentage of Trial usage has been increasing each period (shrinking Signed Up and Trial Started cohorts, and growing Use cohorts)
- A growing percentage of long-duration trials, beginning in P4 (growing Use >1day cohort)
- Despite efforts, only approx. 2% of all impressions convert to paying customers in any given period (flat Paid cohort)
It doesn’t warrant imagining or investigating events that drove these behaviors, being a hypothetical scenario, just assume a marketing campaign in P3 drove signups and encouraged discounted purchases; followed by extensive feature release in P4, P5, and P6 led to more features to explore during trials. Cohort analysis explains that while a greater percentage of trials were being used, there was not proportional growth in paid customers. This was hidden from the vanity metrics because both Impressions and Paid customers were growing exponentially but significant revenue was being left on the table. Meanwhile, in fact, features developed to encourage trial use may have increased cost to host, support, and maintain across the free segment without receiving justifiable revenues in the paid. This becomes poignant once Cost of Goods Sold is factored in, proportional to the growth in number of trial accounts, to calculate gross profit. What was a comfortable $505k margin in P4 has become a ($1.1m) deficit in P5. (This is obviously a simple espousal of cohort analysis identifying a business challenge hidden behind vanity metrics.)
Rewind the clock and use cohort analysis to, instead, strategically invest appropriate features that grow the Paid cohort, just 1% per period, and the business has changed significantly (to $16.8M sales in P6 from $4.8m). (Our underlying growth model dramatizes this increase but demonstrates that small changes in the right area can yield great results.)
Appendix
Look at KPIs on which you report and discuss if greater insight can be gained through cohort analysis. Dissect Number of Marketing Event Registrations by source (ex: marketing campaign, marketing email, AE-outreach, website, founder outreach etc); Number of New Accounts by consumption (ex: no activity, single user, up to 5 users, greater than 5 etc); or Number of Successful Demos by story as shown below:
