Chapter 36
When a dependency slows down
Storage is split into 20 shards, and one shard’s disk goes bad: every read and write goes from 5 milliseconds to 500. Nothing is down. Why is almost everyone staring at a spinner when they open the app?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
8 p.m., the peak. v3 keeps conversation records in storage split into 20 shards, by user. One shard’s disk has a problem: every read and write goes from 5 milliseconds to 500. It is not down; on the dashboards it is still “alive”.
A few minutes later, support gets a flood of complaints: the app spins forever on open, the chat list doesn’t move, unread counts are wrong. Yet only one of the 20 shards is bad; by rights it should affect one person in twenty.
This chapter’s question: when a dependency slows down, how do you keep the slowness on the part that really uses it, instead of letting it spread to everyone?
1. Sending is not affected
Good news first. Since chapter 29, the ACKACKACK(确认)The receiver’s “got it” to the sender. In this series, the server’s ACK means the message is in the message log, not that anyone has received it.See the glossary waits only for the message logmessage log消息日志A durable, append-only queue: the hand-off between receiving and delivering. A message counts as received once it is in the log, and the server ACKs then. It usually runs on a message queue, such as a topic partitioned by conversation.See the glossary, not for storage. If the ACK still waited for storage (as in v2), then at the peak of 13,900 messages a second, each waiting 500 milliseconds, 6,950 messages would be stuck halfway at any moment. Now not one is.
Two things still wait on storage:
- Sync: each time the app opens, the sync servicesync service同步服务Answers a device that comes back: what changed in its conversation records, each conversation’s latest seq, and then the messages it is behind on, page by page (from the timeline, or from the inbox with write fan-out).See the glossary reads the person’s changed conversation records (chapter 11). About 6,944 a second at the v3 peak.
- Conversation records (one row per person per conversation, holding the latest seq and how far they have read): a consumer behind the log updates one for each recipient of each message (chapters 11 and 29), about 92,667 a second.
2. Slow means taking up room
There is a simple relation called Little’s law: requests waiting at once = requests arriving per second × how long each waits.
Syncs that read the bad shard, one in 20 of the 6,944, come in at 347 a second. With the shard healthy, fewer than 2 are waiting at any moment; at 500 milliseconds each, it is 174.
In fact it is worse. Normally this shard writes 20,000 rows a second at 5 milliseconds each, so by the same law it is working on 20,000 × 0.005 = 100 rows at once, and that is as many as it can. When the disk slows, the limit is still 100 rows, but each now takes 500 milliseconds, so it finishes only 100 ÷ 0.5 = 200 rows a second, while about 9,600 rows of work are sent to it: about 4,600 conversation-record updates a second land on this shard, two rows each, plus the syncs’ reads.
Requests at the back of the queue get no answer and can only wait for the timeout. Say the storage client’s timeout is 30 seconds and nobody set a separate one for these calls: then each one waits the full 30 seconds, and 347 × 30 ≈ 10,417 are waiting at once.
Each waiting request holds something of the caller’s: a connection, a piece of memory, a slot.
3. One pool, and everyone waits together
To avoid being overwhelmed, the sync service caps how many requests may be waiting on storage at once. Say the whole service allows 2,000, and normally uses 35: plenty of room.
But those 2,000 slots are shared by all 20 shards. Requests for the bad shard come in at 347 a second and hold a slot for 30 seconds each, so in under 6 seconds they take every slot. After that only about 67 slots free up each second (2,000 slots ÷ 30 seconds); about 63 of them go to users on healthy shards, who need 6,597 a second: enough for 1%.
In the simulation it is worse still: a phone gives up after 10 seconds without an answer (section 6), yet when a slot frees up it goes to the request at the front of the queue, the one that has waited longest, whose phone has long since given up. About 9 seconds after the shard went bad, users on healthy shards open the app and fewer than 1 in 100 succeed. The server is still dutifully answering requests nobody is waiting for: over that minute of trouble, it gives about 290,000 such answers.
5% of the data is bad, and almost 100% of people see a spinner. Slack’s outage of 4 January 2021 looked like this: packet loss made the web tier’s calls to the back end slow, which “saturated system resources in our web tier”, and Slack went down (Slack Engineering).
4. An in-order consumer, stuck behind the slow few
The conversation-record consumer, one per log partition, moves forward in order: it can write several records at once, but its “done up to here” can only move past records already written (chapter 29). Say records 1 to 50 are being written, record 3 is stuck, and 4 to 50 are done: “done up to here” still stays at 2.
Say the log has 100 partitions, and at peak each must update 927 records a second. Chapter 34 assumed one consumer can update 10,000 a second, that is, 50 writes in flight at once.
Recipients are spread over 20 shards, so 1 record in 20 is on the bad shard and waits 30 seconds. Such records soon take all 50 write slots: a write now takes about 1.5 seconds on average (30 seconds divided by 20), so the 50 slots get through only about 33 a second, 4% of what it must.
So every partition falls behind, and everyone’s chat list and unread counts freeze, not only those on the bad shard. A minute of trouble leaves each partition about 58 seconds behind. In chapter 34 one spamming group stuck one partition; here one bad piece of storage stalls them all.
Chapter 17’s OS push lives in this consumer too: after updating the record, it calls APNs or a phone maker’s channel. If any channel stalls, it stalls the partitions the same way.
5. Retries make it worse
Every layer thinks “retry a few times on failure” is good practice. The sync service retries storage 3 times, so the bad shard gets up to 4 times the load; the phone retries the sync service 3 times as well, up to 4 × 4 = 16 times. Google’s SRE book gives the example of three layers each retrying 3 times, turning one user action into 64 attempts on the database (Addressing Cascading Failures).
The last thing a shard already down to a hundredth of its capacity needs is more requests.
6. The fix: keep the slowness where it is used
Every call has a deadline, shorter than its caller’s. The phone waits 10 seconds; the sync service gives its whole answer 5 seconds; one storage call within it gets 1 second. The deadline travels down with the request, and each layer first checks how much time is left and skips the work if it is not enough (the SRE book calls this deadline propagation). With a 1-second deadline, the requests waiting on the bad shard drop from 10,417 to 347.
Each dependency and each shard gets its own slots. The bad shard can take at most its own 100 (it can only work on that many at once anyway); when they are full, new requests are refused at once instead of queueing. The other 19 shards don’t lose a single slot. This is a bulkhead: one flooded compartment does not sink the ship.
When failures pile up, trip the circuit breaker, then degrade. If most calls to the bad shard fail over a period, stop calling it and return a failure straight away, letting a few test calls through now and then. Sync need not be all or nothing: the phone already has the chat list locally, and each conversation’s latest seq and latest message sit in chapter 29’s cache, so those are returned as usual, and the changed conversation records are marked “to follow”. The 5% of people on the bad shard open the app and see new messages; only new conversations, pins and mutes arrive a bit later.
Slow work goes into a queue of its own. When the conversation-record consumer reaches a record on the bad shard, it does not wait: it puts the record into that shard’s own retry queue (also a log) and moves on. Deadlines alone are not enough: at up to 1 second per record, the consumer keeps up with only 99% and still slips behind little by little; once the breaker trips and records go straight into the queue, it is back to full speed. OS push works the same way: one queue per channel, so a stalled channel stalls only itself.
A minute of trouble leaves about 278,000 records in the bad shard’s queue (4,633 a second × 60 seconds); once it recovers, its spare capacity (10,000 a second, less the usual 4,633) clears them in 52 seconds. For the other 95% of people, the chat list stays current throughout.
Retries have a budget. Retries may be at most 10% of calls; beyond that, fail straight away. The SRE book’s numbers: with only a per-request limit of 3 attempts, the worst case is close to 3 times the load; with this 10% budget, usually only 1.1 times (Handling Overload). And retry at one layer only, preferably the one closest to the failure.
7. The cost
- Users will see errors. Failing fast and degrading means some people see “try again later”, or part of the data missing. That is on purpose: a bit less for 5% of people beats a spinner for 100%.
- Every timeout, slot count and breaker threshold is a number that can be set wrong. Too tight, and everyday blips trip it; too loose, and it does nothing. They only act on bad days, so they have to be tried in drills (chapter 38).
- The side queues need watching too. How much has piled up and how long it has gone uncleared are numbers to monitor (chapter 33).
- What is caught up arrives out of order. A record from the retry queue is written after the records for later messages; but values in a conversation record only ever go up (chapter 11): if the latest seq is already 4,830 and a late “set it to 4,812” arrives from the retry queue, it is smaller, so nothing changes, and a late old value cannot overwrite a newer one.
8. Other answers
- Netflix’s Hystrix. A Java library that wraps each dependency in its own small thread pool (or semaphore), refuses when it is full, and trips a breaker when the error rate passes a threshold. Its documentation says that under high traffic one slow backend “can cause all resources to become saturated in seconds on all servers” (Hystrix wiki). It is now in maintenance mode and no longer developed (README), but its two ideas, bulkheads and circuit breakers, have stayed.
- Timeouts alone. The simplest; they stop “waiting forever”. But with shared slots, a 1-second timeout still lets the bad shard take 347 slots; how much that crowds out depends on the size of the pool.
- Make way by priority. Instead of separating dependencies, rank requests by importance and drop the unimportant ones first when full (chapter 34). The two are often used together.
- Adaptive concurrency limits. Instead of fixed slot counts, shrink and grow the limit automatically from observed latency. Less to configure, one more algorithm to understand.
9. This chapter’s decision
When one piece of storage is slow, the slowness stays on its own 5%. But what if the whole data centre is gone? Next: losing a data centre.
Comments
Comments live in GitHub Discussions and need a GitHub account.