The canary that only hurt the slowest requests
A recommendation service runs on ECS Fargate behind an Application Load Balancer, twelve tasks across three Availability Zones. Each request reads a user's feature vector from DynamoDB, calls an in-process model, and returns ranked items.
You deploy a new model version to 10% of tasks as a canary. The new model is architecturally identical to the old one — same framework, same input shape — but was trained on six months more data and has roughly 30% more parameters.
After the canary goes live: median latency is unchanged at 42ms. p99 latency rises from 180ms to 1,400ms. The p99 rise is visible on ALL tasks, not only the canary tasks. Error rate is unchanged. CPU utilisation across the fleet averages 40%. The DynamoDB table reports no throttling. Rolling back the canary returns p99 to 180ms within two minutes.
The p99 degradation appears on tasks running the OLD model as well as the new one. Give me your ranked hypotheses for how a canary on 10% of tasks degrades the other 90%, and for each one the specific measurement that would confirm or eliminate it.
Choose how you want to be interrogated
The same case under two modes is two different exercises. If you are unsure, RCA drill is the one that trains root-cause analysis most directly.
RCA drill
A symptom, and nothing else. Find the cause and prove it.
Trains: Hypothesis generation, ranking by prior, and naming the test that discriminates.
Red team
You propose. It attacks. It is trying to make you wrong.
Trains: Defending a position under pressure, and noticing when you should concede.
Socratic
Questions only. It will never tell you the answer.
Trains: Deriving from first principles instead of retrieving from memory.
Design review
Present an architecture. Justify every trade-off you made.
Trains: Stating what you optimised and what you gave up, on purpose.
Viva
Rapid escalating questions. Ends in a graded verdict.
Trains: Answering under time pressure without hedging.
Integration
Problems that span several domains at once. The hardest mode.
Trains: Reasoning across boundaries, where the cause is in a different system to the symptom.
Before you start: you begin locked. The Mentor will give you nothing — not a nudge, not a category — until you state a specific position and the mechanism you think produces it. Four genuine attempts unlock the resolution. There is no shortcut.