Iterative Development of Machine Learning Models

Post was originally published in January 2020 and has been updated in June 2020 for clarity.

I often get asked how long it takes to develop a predictive model; the answer is it depends... It depends on what the business question is, what the deliverable looks like, what data is available -- and usable, what the scope is, what the timeline to deliverable is, etc. Having said that, I've found that ML development falls into the following 2-step cycle.

Part I: Proof-of-concept (POC, v0) -- offline data (90% cleaning + 10% analysis)

  1. Understand (in general) use cases behind different modeling approaches

  2. Understand business use case (including ethics), production environment and outcome type, to narrow down scope of approaches to use

  3. Get an overview of what it will take to get a model into production (at the end of Part II)

  4. Collect/get access to data/database(s)

  5. Determine appropriate data split for (offline + online) training + validation

  6. Feature engineering + business logic

  7. Estimate a baseline (offline) model

  8. Interpret results

  9. Evaluate (offline) performance

  10. Determine what Steps (from above) can potentially improve (offline) model performance, subject to business constraints of Step 2 -- and any technical constraints

Part II: (If applicable) Productionalize promising POC (Step 7) -- near-real time data (20% pre-production + 80% post-production)

  1. Iron-out model input and output spec, architecture diagram

  2. Determine model roll-out strategy and success metric for A/B testing model performance

  3. Code refactor and bug fixing

  4. Develop an API, if applicable

  5. Add testing suite

  6. Dockerize code

  7. Set-up continuous integration

  8. Push to staging, production

  9. Roll out the model per Part II, Step 2 and evaluate (online) model performance

  10. Ongoing model and customer support

  11. Once model is in prod, next iteration starts at Part I, Step 9

TIP: Check-in with stakeholders often to educate and get buy-in

Keywords: Data products, Machine Learning in Production, Business impact

You may also like:

References: