Lesson 6 of 7 · 10 min
Monitoring, drift and retraining
Detect degradation before users report it, distinguish the kinds of drift, and decide when to retrain.
The kinds of drift
| Type | What moved | Example | Fix |
|---|---|---|---|
| Data drift | P(X) — input distribution | A campaign brings a new user demographic | Retrain on recent data |
| Concept drift | P(y|X) — the relationship | Fraud tactics change, so old signals stop working | Retrain, possibly re-engineer features |
| Label drift | P(y) — outcome base rate | Churn doubles after a price change | Recalibrate and retrain |
| Upstream change | The pipeline, not the world | A unit changes from dollars to cents | Fix the pipeline — retraining hides it |
What to monitor, in layers
Service level: latency, error rate, throughput, saturation. Standard, and still the first thing to break.
Input level: feature distributions against a reference window, null rates, cardinality, out-of-range values. This is your early warning, because it needs no labels.
Prediction level: the distribution of scores and of decisions after thresholding. A sudden shift in the positive rate is visible immediately.
Outcome level: the actual metric once ground truth arrives. Most accurate, most delayed — sometimes by months.
Detecting drift statistically
Population Stability Index is the common industry choice: under 0.1 is stable, 0.1 to 0.25 warrants attention, above 0.25 is significant. Kolmogorov–Smirnov works for continuous features, chi-squared for categorical.
The practical difficulty is alert volume. With hundreds of features, a p-value test fires constantly at production sample sizes. Use effect-size thresholds rather than significance, monitor only the features the model actually relies on, and require a shift to persist across several windows before it pages anyone.
Retraining triggers
- Scheduled — simple and predictable, but retrains when nothing changed and waits when something did.
- Performance-based — retrain when a monitored metric crosses a threshold. Best, but needs timely labels.
- Drift-based — retrain when input drift is sustained. The usual proxy when labels are slow.
- Volume-based — retrain after enough new labelled data accumulates to be worth it.
Worth remembering
- Data drift is the inputs moving; concept drift is the relationship moving.
- Ground truth arrives late, so input monitoring is your early signal.
- Retraining triggers should be threshold-based, and retrained models still need validation gates.
Test it now
Drift diagnosis and retraining policy scenarios concentrate in the hard bank.