dear Advisor: What metrics should i be thinking about?
What are the top challenges startups experience related to metrics?
First, let's talk about why we need metrics:
Metrics are 1-number summaries that tell us what's happening in our complex business
They may tell us ASAP if we have product market fit (e.g. hockey stick growth) or a "hairy back"
Show us where we're doing well and where we need to improve
Now let's answer your questions! I hosted a "data and metrics" office hour on the 805 Startups Discord server on July 20th. I'll summarize my advice around metrics here.
What are the top challenges you've seen startups experience related to metrics in the first year?
Not tracking any metrics :)
"If you cannot measure it, you cannot improve it" [Peter Drucker]
Not knowing what the One Metric That Matters (OMTM) for your business is.
Not tracking metrics around customer journeys [Phil Libin's talk] -- to see what can be improved and what can be scaled.
Note that if you have a 2-sided marketplace, you have two sets of customers: buyers and sellers. I recommend tracking the journeys of each.
If your product is HaaS, I recommend tracking your hardware journey, from idea to prototype to manufacturing to delivery.
If tracking metrics, focusing on vanity metrics or positive metrics (such as sales) only -- and not tracking (what I call) negative metrics, that captures quality/efficiency -- and shows you where you need to improve.
Some examples of negative metrics:
on-boarding time from app download to first action
time from purchase to delivery of physical good
time between purchases
Not tracking metrics VCs are interested in.
Some examples:
Top-line growth -- though I would recommend profit
Payback period: time to LTC > CAC
Efficiency
Retention/churn
(If offering metrics as part of your service) Not tracking metrics your customers are interested in, or their customers (in B2B space) are interested in
Are you tracking how you're helping your customers solve their pain points? And their customers' pain points?
What metrics do your customers (and their customers) care about?
Focusing on leading metrics before lagging metrics; e.g. developing predictive models before understanding what happened historically [more on this here].
Which metrics are most commonly misunderstood and utilized incorrectly?
Focusing on growth vs profit
Examples: DoorDash and pizza arbitrage, ScaleFactor
Forgetting that a metric is a 1-number summary of what's happening
Goodhart’s Law: when metric becomes the target
Metrics given without needed context, especially if it depends on another metric(s)
Confusing correlation with causation
Churn vs cohort analysis:
To find churn, you need to specify how much time between purchases is too long -- and the customer churned
To do a cohort analysis, you need to specify a time window that treats all incoming customers as most similar to each other, e.g. a time window of one month assumes that each monthly cohort is most similar to each other
Parting advice:
Tips for designing and evaluating metrics here
When using metrics in the "build-measure-learn" context, make sure that the metric in the "measure" step is something that can be experimented with -- and have direct impact on
Example of metric that you may have less control over: average number of items purchased by a customer last year, because it reflects customer preferences, product availability, product pricing, etc.
"What people are measuring today as 'success' can change rapidly overnight" -- Brad Feld in fireside chat [39 min] with Founder Institute
Start tracking metrics with whatever tool(s) you're most comfortable with -- Excel is OK
Understand what's happening first and automate later
Don't forget that metrics are a way to help you get a bird's-eye view of what's happening in your business, and help you identify where you're doing great and what needs to improve -- to refine your product market fit. Are your metrics reflecting your business? If not, I'd like to help.
You may also like:
Deciding what your product should do next: How to "Build Measure Learn" in 8 Steps
Book: Lean Analytics: Use Data to Build a Better Startup Faster
More metrics: SaaS metrics, GitLab KPIs, "Search to Fill" Marketplace Metric
References: