AI AlgorithM illusions: 8 ways it looks superior but falls short

What to Watch out for in AI Startups: Advice for AI Due Diligence (Part 2)

March 2024

Also in this series:

Over the past two weeks, I’ve reviewed over 65 AI pitches and pitch decks (!), with more to come. There’s AI for everything now, but that’s not surprising. :) What’s still puzzling is how to tell what’s hype and what’s not. Here’s advice to help you detect the difference while evaluating a start-up. 

At this stage in the diligence process, I'll assume that we already know that:

Here’s what else to consider.

Potential Risks when ``AI” is part of the MVP 

Due diligence focus 1: As I’ve shared in the past, this is a potential orange flag, which depends on many things, including if it’s core to the product, the product vertical, the availability of customer data to learn from, and how much the ``AI” solution needs to be state-of-the-art (or hopefully not!) to pull off the start-ups’ claims.

How much value is AI bringing to customers at this stage in the product? Is it core to the experience – or a nice-to-have?

Due diligence focus 2: Demos always look great! Let’s evaluate whether a demo differs from the GTM product. 

Based on the pitch, I’ll suggest a walk-through of how the product solves a common challenge your customers have end-to-end. 

Due diligence focus 3: Is there a possibility of bias to invalidate the product/service/solution claims?

Most commonly, that bias can result from data leakage or incorrect application of an algorithm:

Please note that this can be true for "AI” developed at later product stages; we’ll need to evaluate it then as well.

Potential Risks for AI at any Product Stage: Data Leakage Coming from Clinical Workflows

Epic (an electronic healthcare records system) had to remove its sepsis-predicting algorithm after it included information on antibiotic prescriptions to predict infections because physicians would have already identified sepsis when they ordered an antibiotic. 

Due diligence focus 4: How does/doesn’t “AI” build on and integrate with clinical workflows to improve decision support and/or efficiency?

Potential Risks for AI at any Product Stage: Data Leakage Coming from the Collection Process

Start-ups developing HealthTech solutions for diagnoses (such as heart or cancer-related) from images (such as MRI, CT, ultrasound, and echo) may step onto this landmine. 

Algorithms designed to predict the existence of a health condition from these images will seem to perform well at detecting the condition because to get those images, a patient must first get a referral to a specialist and then be seen by a radiologist/technician/similar who will capture any “weird-looking” images. This way, a naive model that predicts everyone has the specific condition – will do very well (!). 

This is the most common mistake HealthTech start-ups make!

Due diligence focus 5: How was the data collected to power the “AI” algorithm? How does it account for the diversity in the conditions predicting and patients with those conditions?

Potential Risks for AI at any Product Stage: Data Leakage Coming from the Algorithm’s Testing Process

Unfortunately, where patients get their care also plays a role, from their clinicians’ expertise to the hospital’s national ranking to geographic locations to the volume of patients the care center sees in specific expertise. An algorithm’s performance at a top-ranked hospital, for example, relative to a baseline of a rural one, will typically look better artificially due to that hospital's patient ecosystem and the volume of patients it cares for. 

Due diligence focus 6: How was the algorithm's performance evaluated? Are the metrics we see based on the business use case and the testing data set?

Potential Risks for AI at any Product Stage: Using the Wrong Algorithm

There are algorithms aimed at forecasting the efficacy of a treatment for better managing patient care. We know that the demographics of each patient and the characteristics of where and from whom they’re getting care may affect outcomes; by controlling for this variability, we may be able to explain why something happened the way it did. If we don’t remove the effects of this individualized experience to treatment, we won’t understand the true efficacy of the treatment effect – if there’s even any left. Many founders forget to do the last step!

Due diligence focus 7: How well does the team understand, even at a high level, what the algorithm is actually doing under the hood? Or is it treated as a black box?

Potential Risks for AI at any Product Stage: Setting and Forgetting About It

AI demos look slick! But demos and real-time product usage differ!  

Due diligence focus 8: How well does the team understand—and budget for in their financial projections—what it will take to host, support, maintain, debug, and ensure that the algorithm doesn’t hallucinate when the stakes are high?

As you know, gauging how the founders approach and answer these questions will tell you a lot! :) 

Next Steps for More Support

VC Frequently Asked Question: How would we work together? 

I'm here to help you evaluate risks, challenges, and opportunities. Here's how I typically support funds:

You may also like (advice for investors):

You may also like (advice for founders):