The canary was green and the rollout still failed
5% canary for 24 hours, all metrics flat. At 100% the model degraded within an hour.
The system
- A churn-prediction model driving a retention-offer campaign for a subscription business.
- Deployment is a canary: 5% of users get the new model for 24 hours, then a full ramp if metrics hold.
- Canary routing is by a hash of user id.
- Success metric is offer-acceptance rate, measured over the canary window.
What you observe
- Canary metrics were statistically indistinguishable from control across the full 24 hours.
- At 100%, offer-acceptance rate dropped 18% within an hour and stayed down.
- Rolling back restored the previous rate immediately.
- The model artifact and features were identical in both phases.
Stop and answer before reading on
What do you check first, and why that before anything else? Write down three checks in order. The ordering is what is being graded — anyone can list plausible causes.
Diagnostic order
- 1
Check whether the canary population is representative.
Hash-based routing is uniform over user ids, which is not the same as uniform over the behaviour that matters. Compare the canary cohort against the full population on the features the model actually uses.
- 2
Check whether the metric had the statistical power to detect the effect.
5% of traffic for 24 hours may simply not produce enough offer acceptances to detect an 18% relative change. A flat canary is not evidence of no effect if the confidence interval spans the effect size you care about.
- 3
Look for interference between the canary and control arms.
If both arms compete for a shared, finite resource — an offer budget, an inventory pool, a rate-limited downstream service — the canary's behaviour at 5% does not extrapolate to 100%.
- 4
Compare the canary window against the ramp window on time-of-day and day-of-week.
A canary that ran over a weekend and a ramp that landed on Monday morning are two different populations, whatever the routing says.
Root cause
A shared-resource interference effect. The new model was more aggressive about issuing retention offers. At 5% traffic that extra volume was absorbed by the daily offer budget without anyone noticing. At 100% it exhausted the budget by mid-morning, after which the offer service silently returned a no-offer response — so the majority of users the model targeted received nothing, and acceptance rate collapsed. The canary could not have surfaced this because the failure mode only exists at scale.
The fix
- Roll back, then re-ramp in stages (5% → 25% → 50%) with a hold at each step long enough to reach statistical power.
- Add the offer-budget consumption rate to the canary dashboard as a leading indicator.
- Make the offer service fail loudly when the budget is exhausted rather than returning an empty response.
- Calibrate the new model's offer threshold against the budget before ramping.
What should have caught it
- Do a power calculation before choosing canary size and duration; a canary too small to detect the effect is theatre.
- Enumerate shared resources between arms and monitor their consumption rate, not just the outcome metric.
- Prefer staged ramps over a single 5% → 100% jump for anything touching a finite budget.
- Never let a downstream dependency degrade silently — an empty response is a failure, not a result.
Follow-ups you should expect
- How would you size the canary given a baseline acceptance rate and a minimum detectable effect?
- When is a shadow deployment better than a canary here?
- What are the ethics and mechanics of an interleaved experiment for this use case?
- How would you detect this class of failure from monitoring alone, without a staged ramp?