Keep a toil ledger for your on-call rotation
Most on-call handovers are a list of incidents. That list is useful, but it misses the thing that burns people out: the dozens of small, repetitive interventions that never become incidents at all.
The ledger
We keep a single table that the on-call engineer fills in during the shift. One row per manual action, however small:
| What | Trigger | Minutes | Could a script do it? |
|---|---|---|---|
| Restarted stuck queue consumer | Lag alert | 10 | Yes |
| Cleaned up /var on worker-7 | Disk 90% | 15 | Yes |
| Rotated expiring internal cert | Calendar reminder | 25 | Mostly |
From ledger to backlog
At the end of each rotation, anything that appears three or more times becomes a backlog item, with the total minutes as its justification. Nothing fancy — the numbers do the arguing.
If a human did it twice this week, a computer should do it next week.
After two months our pager volume dropped by about a third, mostly from boring fixes: a log rotation policy, an auto-restart with a backoff, and a certificate renewal job that should have existed years ago.
← All posts