dear Advisor: What metrics should i be thinking about?

What are the top challenges startups experience related to metrics?

First, let's talk about why we need metrics:

  • Metrics are 1-number summaries that tell us what's happening in our complex business

  • They may tell us ASAP if we have product market fit (e.g. hockey stick growth) or a "hairy back"

  • Show us where we're doing well and where we need to improve

Now let's answer your questions! I hosted a "data and metrics" office hour on the 805 Startups Discord server on July 20th. I'll summarize my advice around metrics here.

What are the top challenges you've seen startups experience related to metrics in the first year?

  • Not tracking any metrics :)

    • "If you cannot measure it, you cannot improve it" [Peter Drucker]

  • Not knowing what the One Metric That Matters (OMTM) for your business is.

  • Not tracking metrics around customer journeys [Phil Libin's talk] -- to see what can be improved and what can be scaled.

    • Note that if you have a 2-sided marketplace, you have two sets of customers: buyers and sellers. I recommend tracking the journeys of each.

    • If your product is HaaS, I recommend tracking your hardware journey, from idea to prototype to manufacturing to delivery.

  • If tracking metrics, focusing on vanity metrics or positive metrics (such as sales) only -- and not tracking (what I call) negative metrics, that captures quality/efficiency -- and shows you where you need to improve.

    • Some examples of negative metrics:

      • on-boarding time from app download to first action

      • time from purchase to delivery of physical good

      • time between purchases

  • Not tracking metrics VCs are interested in.

    • Some examples:

      • Top-line growth -- though I would recommend profit

      • Payback period: time to LTC > CAC

      • Efficiency

      • Retention/churn

  • (If offering metrics as part of your service) Not tracking metrics your customers are interested in, or their customers (in B2B space) are interested in

    • Are you tracking how you're helping your customers solve their pain points? And their customers' pain points?

    • What metrics do your customers (and their customers) care about?

  • Focusing on leading metrics before lagging metrics; e.g. developing predictive models before understanding what happened historically [more on this here].


Which metrics are most commonly misunderstood and utilized incorrectly?

  • Focusing on growth vs profit

  • Forgetting that a metric is a 1-number summary of what's happening

  • Goodhart’s Law: when metric becomes the target

  • Metrics given without needed context, especially if it depends on another metric(s)

  • Confusing correlation with causation

  • Churn vs cohort analysis:

    • To find churn, you need to specify how much time between purchases is too long -- and the customer churned

    • To do a cohort analysis, you need to specify a time window that treats all incoming customers as most similar to each other, e.g. a time window of one month assumes that each monthly cohort is most similar to each other


Parting advice:

  • Tips for designing and evaluating metrics here

  • When using metrics in the "build-measure-learn" context, make sure that the metric in the "measure" step is something that can be experimented with -- and have direct impact on

    • Example of metric that you may have less control over: average number of items purchased by a customer last year, because it reflects customer preferences, product availability, product pricing, etc.

  • "What people are measuring today as 'success' can change rapidly overnight" -- Brad Feld in fireside chat [39 min] with Founder Institute

  • Start tracking metrics with whatever tool(s) you're most comfortable with -- Excel is OK

  • Understand what's happening first and automate later

Don't forget that metrics are a way to help you get a bird's-eye view of what's happening in your business, and help you identify where you're doing great and what needs to improve -- to refine your product market fit. Are your metrics reflecting your business? If not, I'd like to help.


You may also like:

References: