The retraining job that only fails on Fridays
A nightly retraining job runs on an EC2 Spot instance, reads Parquet from S3 via a Glue catalogue, trains a gradient-boosted model, and writes the artefact back to S3. It has run for eight months.
For the last three weeks it has failed with an out-of-memory kill, but only on Friday nights. Monday through Thursday it completes in about 50 minutes using roughly 60% of the instance's 64 GB. Saturday and Sunday runs succeed.
The job code has not changed in four months. The instance type has not changed. The upstream data pipeline that writes the Parquet was modified five weeks ago to add a new categorical column.
What is the mechanism that makes this fail on exactly one day of the week? Rank your hypotheses and tell me the single cheapest observation that would let you discard most of them at once.
Choose how you want to be interrogated
The same case under two modes is two different exercises. If you are unsure, RCA drill is the one that trains root-cause analysis most directly.
RCA drill
A symptom, and nothing else. Find the cause and prove it.
Trains: Hypothesis generation, ranking by prior, and naming the test that discriminates.
Red team
You propose. It attacks. It is trying to make you wrong.
Trains: Defending a position under pressure, and noticing when you should concede.
Socratic
Questions only. It will never tell you the answer.
Trains: Deriving from first principles instead of retrieving from memory.
Design review
Present an architecture. Justify every trade-off you made.
Trains: Stating what you optimised and what you gave up, on purpose.
Viva
Rapid escalating questions. Ends in a graded verdict.
Trains: Answering under time pressure without hedging.
Integration
Problems that span several domains at once. The hardest mode.
Trains: Reasoning across boundaries, where the cause is in a different system to the symptom.
Before you start: you begin locked. The Mentor will give you nothing — not a nudge, not a category — until you state a specific position and the mechanism you think produces it. Four genuine attempts unlock the resolution. There is no shortcut.