Chapter 38
Drills and postmortems
Circuit breakers, degradation, switching data centres: none of these ever runs on a normal day. Will they work on the day something breaks? And if the alerts go off dozens of times a night, will anyone still look at the one that matters?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
Chapters 33 to 37 added a lot to v3: circuit breakers, degradation, retry queues, draining gateways, a third data centre, switching cities. Chapter 33’s probes and sampling run all the time, and draining runs at every deploy; but most of the other fallbacks have one thing in common: they never run on a normal day, only on the day something breaks.
There is also something that runs all the time and that nobody trusts: the alerts. They go off at every error, and after enough of them, people stop looking.
This chapter’s question: how do you know these bad-day-only things really work, and how do you wake people only when something has really broken?
1. What has never run will most likely not work
Slack’s report on its outage of 4 January 2021 contains two things (Slack Engineering):
- The service that sets up new machines had been load-tested before, but that day it had to add more machines at once than it had ever been tested with; it hit the limit on open files and the cloud provider’s quotas, and the scale-up failed.
- The dashboards and alerting service sat in a different network from their own database, with the network equipment that failed that day in between. So just when monitoring was needed most, monitoring went down first.
Both were fallbacks: one was scaling up, the other monitoring. Both were there every day, but at that scale and in that situation, neither had ever really run.
v3’s fallbacks are the same. Whether the circuit-breaker thresholds (chapter 36) are set right is known only when a bad shard turns up; the cross-city switch (chapter 37) needs a person to make the call, and for someone who has never practised it, the first time is at 3 a.m.
2. Alerts: ringing at every error
The most direct alert: ring if more than 0.01% of messages were late in the last 5 minutes (chapter 33’s target).
That looks reasonable, but it uses chapter 33’s sampling: 1 delivery in every 1,024 is measured. At night traffic is light, and 5 minutes sample only about 1,000 deliveries; a single late one makes 0.1%, ten times over the line.
Simulate a month: normal lateness is only 30% of the target, yet this alert goes off 888 times, about 30 a day, 319 of them between midnight and 7 a.m.
Two real things happened that month: one gateway suddenly lost power at 8:30 p.m.; and a load balancer cut idle connections early, so 0.5% of pushes on one gateway went nowhere, adding about 0.025% lateness across the whole service for two days (chapter 33’s example). They were mixed in with the 888, looking no different from the rest. After an alert has gone off 888 times, nobody gets out of bed for the 889th.
3. Alert on how fast the budget burns
Chapter 33 set the target at 99.99%, meaning 0.01% of messages may be late in a month. As a whole-service outage, over 30 days that is 4.32 minutes. This is the error budget.
The alert asks not “is anything wrong right now?” but how fast the budget is burning: at the current rate of lateness, in how many days the budget would be gone. Using it up in exactly a month is a burn rate of 1. Google’s SRE Workbook recommends three tiers (Alerting on SLOs):
| Burn rate | Window | Equal to this much of a month’s budget | What to do |
|---|---|---|---|
| 14.4 | 1 hour (and the last 5 minutes too) | 2% | Wake someone |
| 6 | 6 hours (and the last 30 minutes too) | 5% | Wake someone |
| 1 | 3 days (and the last 6 hours too) | 10% | Open a ticket, look at it during working hours |
The third column comes from this: a month is 720 hours, so a burn rate of 1 uses 1/720 of the budget an hour; 14.4 for 1 hour is 14.4 × 1 ÷ 720 = 2%. The other two work the same way: 6 × 6 ÷ 720 = 5%, 1 × 72 ÷ 720 = 10%.
The long window spreads out occasional late messages, so one or two sampled at night no longer trigger it; the short window lets the alert stop as soon as the trouble ends. It is not slow in a real outage either: the 14.4 line means 0.144% of the hour’s deliveries late (0.01% × 14.4); in a whole-service outage every delivery is late, so 5.2 seconds of it (3,600 seconds × 0.144%) puts the hour’s average over the line.
The same month with these three tiers: the two waking tiers went off for one thing only, the gateway that lost power, ringing right at 8:30. Two tickets were opened: one for the same event, the other for the load balancer, opened the next afternoon. That one burned only a little each day, 14.7% of the budget over the two days: not worth waking anyone at night, but not to be ignored.
4. Drills: spend a small slice of the budget
If the budget is not used up, part of it can be spent breaking things on purpose, during working hours, with people watching.
First in the simulator, then in the test environment, finally in production, from small to large:
- Stop a gateway. Stop the process normally; the system closes the connections for it, the 50,000 phones know at once, and they spread over the other 19, about 2,600 each, back within seconds, at almost no cost to the budget.
- Pull the power on a gateway. The connections die silently, and a phone only notices when its next heartbeat gets no answer: the heartbeat goes out after 4 minutes of silence, so about 2 minutes away on average, then 10 seconds without an answer, about 130 seconds in all (chapter 23). At average traffic that costs about 2.4% of a month’s budget; four times a month, about a tenth. The budget counts deliveries, and at the evening peak there are 3 times as many a second as on average, so the same pull at the peak costs about 7.5%: that was the real power loss above.
- Slow down one storage shard, and watch the circuit breaker and retry queues (chapter 36).
- Shut down a data centre, and watch the majorities and leases (chapter 37).
- Switch cities, step by step from the runbook; this one trains people.
Netflix’s Chaos Monkey automates the first kind: it randomly terminates instances in production so that engineers build services that can withstand instance failures (Chaos Monkey).
Every drill measures one thing: how much budget it actually burned, and whether that matches what was worked out beforehand. If it doesn’t, the design has something nobody thought of, and that is exactly what drills are for.
5. Postmortems: change the system, not the people
What drills find, and what real incidents expose, must both turn into changes to the system.
When a postmortem is required is decided in advance. Google’s sample error budget policy has a rule: if one incident burns more than 20% of four weeks’ budget, there must be a postmortem, with at least one top-priority fix (Error Budget Policy). With our numbers, a fifth is 0.86 minutes of whole-service outage, for example 1.7 minutes for half the users. Lost data, manual intervention, and trouble that monitoring itself did not catch should also get a postmortem (Postmortem Culture).
A postmortem blames no individual. In the SRE book’s words: assume everyone acted with good intentions and did the right thing with the information they had, and look only for what contributed to the incident; a culture of blame gets problems “swept under the rug”. Blame people, and next time everyone hides the details, and the details are exactly what changing the system needs.
So a postmortem does not end with “so-and-so must be more careful”, but with a change: add a rate limit, change a timeout, make the alerts not depend on what they monitor, put a fallback that has never run into next month’s drills.
6. The cost
- A drill is a small failure you choose yourself. About a tenth of the budget a month, and users really do feel a little of it.
- On-call is people’s time. There are fewer alerts, but each one needs someone to act at once, with a runbook that says what to do first.
- Sometimes a month’s budget is not enough. If a big incident comes first, that month’s drills stop and deploys slow down; this has to be agreed with the product side in advance.
- A postmortem takes time to write, and more to act on. If nobody does the fixes, it is just a record.
7. Other answers
- Alert on error counts or thresholds (section 2). The simplest; fine at low volume with steady traffic. Once volume is high and traffic swings between day and night, it gets noisy.
- Drill, but not in production. Less risk, but the test environment doesn’t have production’s scale, configuration and traffic.
- Game days: not random; a day is agreed on and everyone acts out one big failure together, such as a whole data centre. AWS’s Well-Architected Framework recommends holding them regularly (REL12-BP05).
- No budget, just “nothing may break”. If nothing may break, nothing may change; any change breaks something, and in the end nobody dares deploy.
8. This chapter’s decision
v3 can now grow, and keep running. From here, the same system serves different products, starting with chat used inside companies. Next: workplace chat.
Comments
Comments live in GitHub Discussions and need a GitHub account.