AUC fell 14 points and every pipeline is green
A fraud model retrains weekly and is monitored on data drift (PSI per feature), prediction distribution, and service metrics. Ground-truth labels arrive with a 30-day lag, so offline AUC is computed monthly.
This month's AUC came in at 0.79, down from 0.93. Every data-validation check passed. Every PSI value is below 0.1. Prediction-score distribution is within its usual band. Service latency and error rate are normal. No alert fired at any point.
The last four weekly retrains all completed successfully and each new model beat the incumbent on its own validation split before promotion.
Every guardrail passed and the model got substantially worse. Tell me what class of failure is invisible to all four of those monitors, and what you would compute first to distinguish between the candidates.
Choose how you want to be interrogated
The same case under two modes is two different exercises. If you are unsure, RCA drill is the one that trains root-cause analysis most directly.
RCA drill
A symptom, and nothing else. Find the cause and prove it.
Trains: Hypothesis generation, ranking by prior, and naming the test that discriminates.
Red team
You propose. It attacks. It is trying to make you wrong.
Trains: Defending a position under pressure, and noticing when you should concede.
Socratic
Questions only. It will never tell you the answer.
Trains: Deriving from first principles instead of retrieving from memory.
Design review
Present an architecture. Justify every trade-off you made.
Trains: Stating what you optimised and what you gave up, on purpose.
Viva
Rapid escalating questions. Ends in a graded verdict.
Trains: Answering under time pressure without hedging.
Integration
Problems that span several domains at once. The hardest mode.
Trains: Reasoning across boundaries, where the cause is in a different system to the symptom.
Before you start: you begin locked. The Mentor will give you nothing — not a nudge, not a category — until you state a specific position and the mechanism you think produces it. Four genuine attempts unlock the resolution. There is no shortcut.