Inference cost tripled while traffic stayed flat
A batch scoring service reads feature files from S3, scores them, and writes results back to S3. It runs on EC2 instances in private subnets, egressing through a NAT gateway. Instances are covered by a Compute Savings Plan.
Month over month: requests scored is flat within 3%. Total bytes read from S3 is up 4%. The AWS bill for this workload has gone from about $2,100 to $6,400. EC2 instance-hours are unchanged.
Two changes landed last month. The team enabled a new feature group, which added six columns to the feature files. And they moved the scoring fleet from one Availability Zone to three, for resilience.
Instance-hours are flat and bytes read are up only 4%, yet the bill tripled. Where is the money going? Name the line items in order of likely contribution and how you would confirm each from billing data alone.
Choose how you want to be interrogated
The same case under two modes is two different exercises. If you are unsure, RCA drill is the one that trains root-cause analysis most directly.
RCA drill
A symptom, and nothing else. Find the cause and prove it.
Trains: Hypothesis generation, ranking by prior, and naming the test that discriminates.
Red team
You propose. It attacks. It is trying to make you wrong.
Trains: Defending a position under pressure, and noticing when you should concede.
Socratic
Questions only. It will never tell you the answer.
Trains: Deriving from first principles instead of retrieving from memory.
Design review
Present an architecture. Justify every trade-off you made.
Trains: Stating what you optimised and what you gave up, on purpose.
Viva
Rapid escalating questions. Ends in a graded verdict.
Trains: Answering under time pressure without hedging.
Integration
Problems that span several domains at once. The hardest mode.
Trains: Reasoning across boundaries, where the cause is in a different system to the symptom.
Before you start: you begin locked. The Mentor will give you nothing — not a nudge, not a category — until you state a specific position and the mechanism you think produces it. Four genuine attempts unlock the resolution. There is no shortcut.