Iterative Development of Machine Learning Models

I often get asked how long it takes to develop a predictive model; the answer is it depends... It depends on what the business question is, what the deliverable looks like, what data is available -- and usable, what the scope is, what the timeline to deliverable is, etc. Having said that, I've found that ML development falls into the following 2-step cycle.

Part I: Proof-of-concept (POC, v0) -- offline data (90% cleaning + 10% analysis)

1. Understand (in general) use cases behind different modeling approaches

2. Understand business use case (including ethics), production environment and outcome type, to narrow down scope of approaches to use

3. Get an overview of what it will take to get a model into production (at the end of Part II)

4. Collect/get access to data/database(s)

5. Determine appropriate data split for (offline + online) training + validation

6. Feature engineering + business logic

7. Estimate a baseline (offline) model

8. Interpret results

9. Evaluate (offline) performance

10. Determine what Steps (from above) can potentially improve (offline) model performance, subject to business constraints of Step 2 -- and any technical constraints

Part II: (If applicable) Productionalize promising POC (Step 7) -- near-real time data (20% pre-production + 80% post-production)

1. Iron-out model input and output spec, architecture diagram

2. Determine model roll-out strategy and success metric for A/B testing model performance

3. Code refactor and bug fixing

4. Develop an API, if applicable

5. Add testing suite

6. Dockerize code

7. Set-up continuous integration

8. Push to staging, production

9. Roll out the model per Part II, Step 2 and evaluate (online) model performance

10. Ongoing model and customer support

11. Once model is in prod, next iteration starts at Part I, Step 9

TIP: Check-in with stakeholders often to educate and get buy-in

Keywords: Data products, Machine Learning in Production, Business impact

You may also like:

• How to Deliver Business Impact with Data (blog)

• Talk: Data Projects vs Products: Top 5 Reasons Why your Data Science Project Didn't Make it and How to Get It Right the First Time

• UCLA Graduate Course: Stats 404 - Statistical Computing and Programming

References:

• Cartoon: https://xkcd.com/1319/