0.997 AUC on the first attempt

You inherit a model predicting whether a support ticket will breach its SLA. First training run: 0.997 AUC on a random 20% held-out split, using 340 features assembled by joining the ticket table to an activity log, an agent table and a customer table.

Cross-validation with five folds gives 0.996 ± 0.001. The features were standardised inside a scikit-learn Pipeline. The team is delighted and wants to ship on Monday.

You are suspicious. Tell me precisely why, what the candidate mechanisms are, and the order in which you would test them — cheapest and most diagnostic first.