Iterative Development of Machine Learning Models

Post was originally published in January 2020 and has been updated in June 2020 for clarity.

Part I: Proof-of-concept (POC, v0) -- offline data (90% cleaning + 10% analysis)

  1. Understand (in general) use cases behind different modeling approaches

  2. Understand business use case (including ethics), production environment and outcome type, to narrow down scope of approaches to use (Weeks 1-2)

  3. Get an overview of what it will take to get a model into production (at the end of Part II)

  4. Collect/get access to data/database(s)

  5. Determine appropriate data split for (offline + online) training + validation

  6. Feature engineering + business logic

  7. Estimate a baseline (offline) model

  8. Interpret results

  9. Evaluate (offline) performance

  10. Determine what Steps (from above) can potentially improve (offline) model performance, subject to business constraints of Step 2 -- and any technical constraints

Part II: (If applicable) Productionalize promising POC (Step 7) -- near-real time data (20% pre-prod + 80% post-prod)

  1. Iron-out model input and output spec, architecture diagram

  2. Determine model roll-out strategy and success metric for A/B testing model performance

  3. Code refactor and bug fixing

  4. Develop an API, if applicable

  5. Add testing suite

  6. Dockerize code

  7. Set-up continuous integration

  8. Push to staging, production

  9. Roll out the model per Part II, Step 2 and evaluate (online) model performance

  10. Ongoing model and customer support

  11. Once model is in prod, next iteration starts at Part I, Step 9

TIP: Check-in with stakeholders often to educate and get buy-in

Keywords: Data products, Machine Learning in Production, Business impact

You may also like: