EffectiveML 01: Delivering successful ML projects
Table: Comparison of feedback mechanisms and time-to-feedback in high- and low-effectiveness environments
| Task | High-effectiveness environment | Low-effectiveness environment |
|---|---|---|
| Testing if code changes worked as expected | Automated testing (~ seconds to minutes) | Manual testing (~ minutes to hours) |
| Testing if training pipeline works end to end | Training smoke test (~ 1 minute) | Full model training (~ minutes to hours, depending on model architecture) |
| Getting feedback on code changes | Pair programming (~ seconds to minutes) | Pull request reivews (~ hours to days) |
| Understanding if application is working as expected in production | Monitoring in production (~ seconds - as it happens) | Customer complaints (~ days, or longer if not directly reported) |
How teams can move from a low-effectiveness environment to a high-effectiveness environment? MLOps is not enough (MLOps and ML platforms aren’t going to write comprehensive tests for you, talk to users for you, or reduce the negative impacts of team silos for you.)
Delivering successful ML projects requires a multi‐disciplinary approach across these five disciplines: product, software engineering, data, ML, and delivery:
software engineering:
- code design
- automated testing
- refactoring
- dependency management
- code editor efficency
data:
- data quality
- data engineering
- data security and privacy
Machine learning:
- ML techniques
- model evaluation
- ML governance
- MLOps
- CI/CD
Product:
- product thinking
- responsible AI
- human-centered AI
Delivery:
- lean delivery
- team topologies