dear Advisor: What metrics should i be thinking about?

First, let's talk about why we need metrics:

• Metrics are 1-number summaries that tell us what's happening in our complex business

• Show us where we're doing well and where we need to improve

• When metrics are tracked over time, they can tell us ASAP if we have product market fit

Now let's answer your questions! I hosted a "data and metrics" office hour on the 805 Startups Discord server on July 20th. I'll summarize my advice around metrics here.

What are the top challenges you've seen startups experience related to metrics in the first year?

• Not tracking any metrics :)

  • • "If you cannot measure it, you cannot improve it" [Peter Drucker]

• Not knowing what the One Metric That Matters (OMTM) for your business is -- and how that's different from your North Star metric.

• Not tracking metrics around customer journeys to see what can be improved and what can be scaled.

  • • Note that if you have a 2-sided marketplace, you have two sets of customers: buyers and sellers. I recommend tracking the journeys of each.

    • If your product is HaaS, I recommend tracking your hardware journey, from idea to prototype to manufacturing to delivery.

    • Recommended resources to help you identify and quantify customer journeys: Phil Libin's talk and Wardley map

• If tracking metrics, focusing on vanity metrics or positive metrics (such as sales) only -- and not tracking (what I call) negative metrics, that captures quality/efficiency -- and shows you where you need to improve.

  • • Some examples of negative metrics:

      • • on-boarding time from app download to first action

        • time from purchase to delivery of physical good

        • time between purchases

• Not tracking metrics VCs are interested in.

  • • Some examples:

      • • Top-line growth -- though I would recommend profit

        • Payback period: time to LTC > CAC

        • Efficiency

        • Retention/churn

• (If offering metrics as part of your service) Not tracking metrics your customers are interested in, or their customers (in B2B space) are interested in

  • • Are you tracking how you're helping your customers solve their pain points? And their customers' pain points?

    • What metrics do your customers (and their customers) care about?

• Focusing on leading metrics before lagging metrics; e.g. developing predictive models before understanding what happened historically [more on this here].

Which metrics are most commonly misunderstood and utilized incorrectly?

• Focusing on growth vs profit

  • • Examples: DoorDash and pizza arbitrage, ScaleFactor

• Forgetting that a metric is a 1-number summary of what's happening

• Goodhart’s Law: when metric becomes the target

  • • Examples: American Airlines Flight 3411, Wells Fargo

• Metrics given without needed context, especially if it depends on another metric(s)

• Confusing correlation with causation

• Churn vs cohort analysis:

  • • To find churn, you need to specify how much time between purchases is too long -- and the customer churned

    • To do a cohort analysis, you need to specify a time window that treats all incoming customers as most similar to each other, e.g. a time window of one month assumes that each monthly cohort is most similar to each other

Parting advice:

• "We've all heard some variants of 'if you can’t measure it, you can’t manage it.' However, we're often so focused on measuring that we neglect to clarify what the true 'it' is. If you don't define it, you certainly ain’t measuring right." [Michiko Wolcott]

• Tips from Sean J. Taylor for designing and evaluating metrics here

• When using metrics in the "build-measure-learn" context, make sure that the metric in the "measure" step is something that can be experimented with -- and have direct impact on

  • • Example of metric that you may have less control over: average number of items purchased by a customer last year, because it reflects customer preferences, product availability, product pricing, etc.

• 40 examples of North Star" metrics of growth-stage companies here

• "What people are measuring today as 'success' can change rapidly overnight" -- Brad Feld in fireside chat [39 min] with Founder Institute

• Start tracking metrics with whatever tool(s) you're most comfortable with -- Excel is OK

• Understand what's happening first and automate later

Do you need an expert to help you improve your product market fit? I'd like to help.

This blog post was originally based on an office hour I hosted on the on the 805 Startups Discord server on July 20th.

You may also like:

• Dear Advisor: What metrics should I include in my pitch deck?

• Data you need to calculate metrics

• Dear Advisor: Can I just survey my customers to help me decide what to build next?

• Talk: Your Product Market Fit Roadmap: How to get Started Understanding your Customers and Product, Refine your PMF and Grow

• 16 Key Metrics for the Passion Economy, by Li Jin

• Lenny Rachitsky's advice on "Choosing Your North Star Metric", which includes examples of North Star metrics of 48 growth-stage companies

• Deciding what your product should do next: How to "Build Measure Learn" in 8 Steps

• Book: Lean Analytics: Use Data to Build a Better Startup Faster

• More metrics: SaaS metrics, GitLab KPIs, "Search to Fill" Marketplace Metric

References:

• Cartoon: https://marketoonist.com/2016/02/marketing-vanity-metrics.html