Skip to content
GKgkml.dev
← Change mode

AUC fell 14 points and every pipeline is green

Design review
MLOpsbrutal

A fraud model retrains weekly and is monitored on data drift (PSI per feature), prediction distribution, and service metrics. Ground-truth labels arrive with a 30-day lag, so offline AUC is computed monthly.

This month's AUC came in at 0.79, down from 0.93. Every data-validation check passed. Every PSI value is below 0.1. Prediction-score distribution is within its usual band. Service latency and error rate are normal. No alert fired at any point.

The last four weekly retrains all completed successfully and each new model beat the incumbent on its own validation split before promotion.

Every guardrail passed and the model got substantially worse. Tell me what class of failure is invisible to all four of those monitors, and what you would compute first to distinguish between the candidates.

Enter sends · Shift+Enter for a new line