Lesson 1 of 7 · 8 min
Why ML systems fail differently
What makes a machine learning system harder to operate than ordinary software, and the maturity levels teams pass through.
Three ways ML differs
Behaviour comes from data, not only code. Two runs of identical code on different data give different systems, so reproducibility requires versioning the data and the configuration, not just the source.
Correctness is statistical. There is no test that passes or fails — there is a metric on a held-out set, and whether it is good enough is a judgement about the business, not about the code.
It decays without changing. Deployed software behaves the same next year unless someone edits it. A model faces a world that moves, so its accuracy falls while every deployment stays green.
Maturity levels
| Level | Looks like | Retraining |
|---|---|---|
| 0 — Manual | Notebooks, hand-off to engineering, manual deploy | Rare and manual, weeks of lead time |
| 1 — Pipeline automation | Training is a versioned pipeline; deploys are scripted | Scheduled or triggered, model registered automatically |
| 2 — CI/CD for ML | Pipeline itself is tested and deployed by CI; monitoring triggers retraining | Continuous, with automated validation gates |
Hidden technical debt
The well-known observation from Google's 2015 paper is that model code is a small box in a large diagram — configuration, data collection, feature extraction, serving infrastructure and monitoring surround it.
The specific debts worth naming: entanglement, where changing any input changes everything downstream; correction cascades, where a model patches another model's output; undeclared consumers, where teams depend on your predictions without your knowledge; and pipeline jungles, where the preparation code has grown past comprehension.
Worth remembering
- The behaviour is determined by data, so the data pipeline is part of the system.
- ML systems degrade silently — no error is raised when predictions become wrong.
- Maturity is about how much of the retrain-to-deploy path is automated.
Test it now
Lifecycle and terminology questions open the easy bank.