Skip to content
GKgkml.dev
← Change mode

The nightly job that went from four minutes to six hours

Socratic
DSAMLOpshard

A Python job de-duplicates and enriches event records before loading them into a warehouse. It has run nightly in about four minutes for two years.

Over the past two months the runtime has crept up: 4 minutes, then 11, then 40, then 2 hours, and last night 6 hours. Input volume over the same period went from 90,000 records to 240,000 — under 3× growth.

The instance is not memory-constrained, CPU sits at 100% on one core throughout, and there is no network or database wait in the profile. The code has not been modified in five months.

Input grew under 3× and runtime grew 90×. What does that ratio tell you before you look at any code, and what would you look for once you do?

Enter sends · Shift+Enter for a new line