The Mentor 🔨
Explanations are everywhere. What is hard to find is something that refuses to help you until you have committed to a falsifiable position, then attacks that position, then tells you precisely where your reasoning failed.
This is not the in-quiz tutor. It does not explain, encourage, or soften. It withholds by design: hints are released one rung at a time, and each rung costs you a real attempt.
How the hint ladder works
You start locked. Every falsifiable position you state — a named cause with a mechanism, a concrete design decision, a test with its expected outcomes — buys one rung. Questions, guesses without reasoning and requests for help buy nothing.
0
Locked
No content at all
1
Reframe
Which assumption is load-bearing
2
Narrow
Which layer, one direction eliminated
3
Mechanism
How this class of fault works
4
Resolution
The answer, and what you missed
Enforced server-side against a signed counter. There is no override, and phrasing the request differently will not move it.
Your dossier
Empty. Once you have run a session, the Mentor records what you failed to establish and hands those gaps into the next session's briefing — so it opens by attacking your worst recorded weakness rather than starting fresh.
The six modes
The scenario supplies the material; the mode supplies the style of attack. The same case under two modes is two different exercises — pick the mode after you pick the scenario.
RCA drill
A symptom, and nothing else. Find the cause and prove it.
Trains: Hypothesis generation, ranking by prior, and naming the test that discriminates.
Red team
You propose. It attacks. It is trying to make you wrong.
Trains: Defending a position under pressure, and noticing when you should concede.
Socratic
Questions only. It will never tell you the answer.
Trains: Deriving from first principles instead of retrieving from memory.
Design review
Present an architecture. Justify every trade-off you made.
Trains: Stating what you optimised and what you gave up, on purpose.
Viva
Rapid escalating questions. Ends in a graded verdict.
Trains: Answering under time pressure without hedging.
Integration
Problems that span several domains at once. The hardest mode.
Trains: Reasoning across boundaries, where the cause is in a different system to the symptom.
Scenarios
Every case has a real, findable cause and at least two plausible wrong turns a competent person takes first. The symptom is deliberately in a different layer to the cause.
The canary that only hurt the slowest requests
A recommendation service runs on ECS Fargate behind an Application Load Balancer, twelve tasks across three Availability Zones. Each request reads a user's feature vector from DynamoDB, calls an in-process model, and returns ranked items.
rca · ~25 min
The retraining job that only fails on Fridays
A nightly retraining job runs on an EC2 Spot instance, reads Parquet from S3 via a Glue catalogue, trains a gradient-boosted model, and writes the artefact back to S3. It has run for eight months.
rca · ~20 min
Inference cost tripled while traffic stayed flat
A batch scoring service reads feature files from S3, scores them, and writes results back to S3. It runs on EC2 instances in private subnets, egressing through a NAT gateway. Instances are covered by a Compute Savings Plan.
rca · ~20 min
AUC fell 14 points and every pipeline is green
A fraud model retrains weekly and is monitored on data drift (PSI per feature), prediction distribution, and service metrics. Ground-truth labels arrive with a 30-day lag, so offline AUC is computed monthly.
rca · ~30 min
Excellent offline, mediocre live, no error anywhere
A churn model scores 0.91 AUC on a held-out set built with a point-in-time-correct feature store query. Deployed behind an API, its live decisions perform at roughly the level of the previous heuristic — a large, consistent gap with the offline number.
rca · ~25 min
0.997 AUC on the first attempt
You inherit a model predicting whether a support ticket will breach its SLA. First training run: 0.997 AUC on a random 20% held-out split, using 340 features assembled by joining the ticket table to an activity log, an agent table and a customer table.
debug · ~20 min
The nightly job that went from four minutes to six hours
A Python job de-duplicates and enriches event records before loading them into a warehouse. It has run nightly in about four minutes for two years.
rca · ~20 min
The queue that stops draining above 40,000 items
A worker consumes messages from SQS, and for each message performs a breadth-first traversal of a dependency graph held in memory to decide which downstream tasks to enqueue. Graphs are typically a few thousand nodes.
debug · ~18 min
Bisect blames a merge commit that changed nothing
A model-training repository produces subtly worse models than it did three weeks ago — evaluation AUC down about two points, reproducibly, on a fixed dataset and a fixed seed.
rca · ~20 min
The credential that was deleted in the very next commit
A developer committed a file containing a live AWS access key pair to a repository, noticed within four minutes, deleted the file, and committed the deletion. Both commits were pushed to a private GitHub repository with eight collaborators.
tradeoff · ~18 min
Three days of a colleague's work vanished from main
A colleague reports that their commits from the last three days are gone from `main`. `git log` on `main` shows your commits but not theirs. They have not deleted anything and their local branch still has their work.
rca · ~15 min
The DR plan that claims fifteen minutes
A documented disaster-recovery plan states an RTO of 15 minutes and an RPO of 5 minutes for a transactional web application. The architecture: an Auto Scaling group behind an ALB in eu-west-1, an RDS PostgreSQL instance with Multi-AZ enabled, an S3 bucket with cross-region replication to eu-central-1, and AMIs copied nightly to eu-central-1.
tradeoff · ~22 min
They enabled Multi-AZ to fix read load
An RDS PostgreSQL instance is CPU-saturated. Roughly 85% of the query volume is reads from a reporting dashboard; the remaining 15% is transactional writes from the application.
tradeoff · ~15 min
The lifecycle rule that increased S3 cost
A bucket holds 40 TB of model training artefacts: roughly 9 million objects, most between 20 KB and 200 KB, with a few large checkpoint files. Objects are read heavily in the week after creation and almost never afterwards.
rca · ~18 min
The fine-tune that learned the voice and lost the knowledge
You fine-tune a 7B base model on 4,000 internal support transcripts so it answers in your company's house style. Three epochs, learning rate 1e-4, full fine-tune, batch size 8, no scheduler.
rca · ~22 min
RAG that retrieves the right document and answers wrongly
A retrieval-augmented system answers questions over 60,000 internal documents. Retrieval is measured at recall@5 of 0.94 — the correct document is in the top five results 94% of the time. Answer accuracy, graded by humans, is 0.61.
rca · ~28 min
The pipeline that produces fluent nonsense
A pipeline caches tokenized text to avoid re-tokenizing a large corpus. Stage one tokenizes with a tokenizer loaded from a local path and writes integer arrays to Parquet. Stage two loads those arrays and runs them through a model.
debug · ~15 min
The summariser that leaked its own instructions
An internal tool summarises documents that users upload, and can call two functions: `search_documents(query)` and `email_summary(recipient, body)`. The system prompt sets the persona, forbids revealing itself, and forbids emailing anyone outside the company domain.
design · ~25 min
You ablated a head and the model got worse at everything
You are investigating which attention head implements a name-copying behaviour in GPT-2 medium. You zero-ablate head 9.6 and the behaviour degrades substantially: the logit difference between the correct and incorrect name falls by 2.1.
tradeoff · ~25 min
The negation neuron with p < 0.001 that vanished
You scan all 4,096 MLP units in each of GPT-2 medium's 24 layers for units whose activation differs between 40 negated sentences and 40 matched affirmative ones. Unit 1,423 in layer 8 gives t = 4.9, p = 0.00001. You report a negation neuron.
rca · ~22 min
Design real-time fraud scoring
Payments platform. 4,000 transactions per second at peak, 800 average. Every transaction must receive a fraud score before authorisation, and the end-to-end budget for scoring is 50ms at p99.
design · ~35 min
Design an internal LLM serving platform
Twelve product teams want to use LLMs. Some need a hosted API model, some need a fine-tuned open-weights model, one needs a model running where no data leaves the VPC. Usage is bursty and unpredictable: one team may generate 200 requests per second during a batch job and nothing for a week.
design · ~35 min
Design an ML platform for twenty teams
Twenty product teams currently train and deploy models however they like: some in notebooks on laptops, some in ad-hoc SageMaker jobs, two with a real CI pipeline. There are around 60 models in production and nobody can produce a list of them.
design · ~35 min