Chapter 34
Overload: limits and degradation
Someone writes a script that posts 100 messages a second into the 500-member hiking group. The gateways cope, but other groups’ unread counts and notifications start running minutes late. When more comes in than the system can take, where should it say “no”, and what goes first?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
On Saturday evening, someone in the hiking group has their account hijacked, and a script starts posting 100 ads a second into the group.
The phones of all 500 members go off together. A few minutes later, strange things happen in other groups too: unread counts stop moving, and people who are offline get no notifications, in groups that have nothing to do with the hiking group.
This chapter’s question: when more comes in than the system can take, where does it say “no”, and what goes first?
1. First, work out what can’t keep up
When a message comes in, the work it causes is multiplied by the size of the group:
100 messages a second × 500 people × 1.5 devices each = 75,000 deliveries a second, 54% of v3’s whole-site peak of 139,000
That sounds alarming, but these devices are spread over 20 gatewaysgateway网关A connection-layer server. Each holds many long-lived connections and handles auth, heartbeats and traffic without reading messages. One of the few stateful parts: a restart drops all its connections.See the glossary, so each gets only 3,750 more. Added to its usual 6,950, that is 21% of its push capacity (50,000 a second). The gateways are not the bottleneck.
The bottleneck is conversation records. Chapter 11 settled that each message changes one conversation record per member (stamping the version, updating the latest seq). At 100 messages a second, that is 50,000 small transactions a second; each changes two rows (the person’s version and the conversation’s row, chapter 11), so 100,000 rows a second, the write capacity of 5 storage nodes (one writes 20,000 rows a second).
In chapter 29, this work is done by the “conversation records” consumer behind the message logmessage log消息日志A durable, append-only queue: the hand-off between receiving and delivering. A message counts as received once it is in the log, and the server ACKs then. It usually runs on a message queue, such as a topic partitioned by conversation.See the glossary. All of a group’s messages are in the same section (partition) of the log, read in order by one consumer. Suppose it can do 10,000 of these updates a second. Then it keeps up with only a fifth of the incoming rate, and falls 40,000 updates further behind every second:
After the script runs for 10 minutes, 24 million updates are waiting; 10 minutes have done only 2 minutes’ worth, so this partition is 8 minutes behind
The same partition holds thousands of other conversations. Their conversation records, unread counts (chapter 14), and the offline notifications that hang off the conversation records (chapter 17) all queue behind these 24 million unfinished updates. Those are the unrelated groups from the opening.
Then there are the people being spammed: 750 devices, each receiving 100 messages a second, 60,000 in 10 minutes.
2. Limits at the entrance, where the state already lives
The message has to be stopped before it is multiplied by 500. There are two checks:
Per device, at the gateway. Each device may send at most 5 messages a second. The gateway knows from the login token (chapter 25) whose device this is, and the count lives on the connection, so it has nothing else to ask.
The script drops from 100 messages a second to 5: deliveries from 75,000 to 3,750, record updates to 2,500.
The limit is a token bucket: the bucket gains 5 tokens a second and holds at most 20, and each message takes one. When Ana forwards 9 photos at once, the bucket has tokens saved up, and all of them go through; a rule of “at most 5 in any one second” would refuse 4 of them. People are sometimes fast in short bursts; scripts are fast all the time. A token bucket tells the two apart.
Per conversation, at its message server. A per-device limit cannot stop 20 accounts working together: 20 × 5 is 100 a second again. But each conversation already belongs to just one message server (chapter 28), which can see at a glance how many messages a second the conversation is getting.
What is limited is not the number of messages but messages × members: each conversation may reach at most 5,000 recipients a second. Each recipient is one conversation-record update, which chapter 11 writes as two rows: 10,000 rows a second, exactly the “half a storage node” worked out in chapter 11 (a node writes 20,000 rows a second). So:
| Conversation | Most per second |
|---|---|
| A one-to-one chat | set by the per-device limit of 5 |
| A 50-member group | 100 messages |
| The 500-member hiking group | 10 messages |
A refused message is not lost. The message server answers “slow down, retry in N milliseconds”, the message stays in the phone’s outbox (chapter 3), and it is sent again when the time is up. The sender sees “sending”, rather than “sent” followed by nobody receiving it.
3. Behind the entrance: every queue has a limit
Limits stop the load that can be foreseen. What they cannot stop: say an availability zone goes down, and each of the remaining 10 gateways has to push 13,900 times a second; 13,900 is the evening peak, and the day’s average is a third of it; a burst like chapter 29’s is 10 × the average, which makes 46,300 each, just short of the 50,000 limit. At such a time, what goes first has to be decided in advance.
The rule: every queue has a limit, and when it is full it drops by priority.
Take each connection’s push queue: at most 64 KB per connection, 300-odd messages. With the usual 50,000 connections on a gateway, that is 3.2 GB at worst; with a zone down and 100,000 on each, 6.4 GB. Either fits (chapter 22 set aside 16 GB for connections). Without a limit, one phone that reads slowly could make the gateway’s memory grow and grow.
When it is full, things are dropped in this order:
- Hints like “typing…”: useless a second later.
- Presence (chapter 21): about 3,500 a second across the site at peak. If one change is dropped, the server answers with the latest presence the next time the phone reports who is on its screen.
- Read receipts (chapter 15): the read position is kept on the server and comes with the next sync.
- Message pushes, but not messages: the message is already in the log, and the push is only a hint (chapter 2). The gateway drops the pushes queued for this connection and sends one frame instead, “some pushes didn’t reach you, go sync”; the phone syncs once as in chapter 11 and pulls them itself.
Nobody notices the first three being dropped; the fourth only makes things a little late. None of them makes a message disappear for good.
4. The numbers
- No limits: 75,000 deliveries a second, only 3,750 more per gateway; 50,000 record updates a second, the partition 8 minutes behind after 10 minutes.
- 5 a second per device: 3,750 deliveries a second, 2,500 record updates.
- 5,000 recipients a second per conversation: the hiking group at most 10 messages a second, 5,000 record updates, half of one consumer.
- Push queues: 64 KB per connection, 3.2 GB per gateway at worst (6.4 GB with a zone down).
5. The cost
- A wrong limit stops real people. In a 500-member group, 10 messages a second may really not be enough during a red-packet grab (everyone racing to claim gift money posted to the group) or a New Year countdown; limits have to be adjustable per conversation, and someone has to watch the adjustments.
- One group’s allowance can be used up. With 20 accounts flooding together, the allowance runs out and the real people in the group can’t send either. Limits can only keep the damage inside this group; banning the accounts takes reports and moderation.
- Refusals need the client’s help. Old app versions don’t understand “retry in N milliseconds” and only resend with chapter 3’s backoff (chapter 31).
- Degradation code doesn’t run on normal days. Dropping presence and switching to “go sync” only happen on bad days. The SRE book says that code paths you never use are often the ones that don’t work, and suggests regularly running a small share of servers near overload so that degradation is always being exercised (Addressing Cascading Failures). Drills are chapter 38.
- Limits are on volume, not content. An ad account posting just 1 message a second is no concern of the limits. Spam detection and content safety have to be done separately on the send path, for example with a before-send callbackbefore-send callback发送前回调Before storing a message, the IM calls the business’s service to ask “may this go through?” and lets it through, refuses it or rewrites it accordingly. The rules are always current, but every send waits an extra round trip, and when the business is down the IM can only let everything through or refuse everything (branch: IM as a platform).See the glossary (chapter 44).
6. Other answers
- A fixed number per group, with the excess silently dropped. Tencent Cloud IM defaults to 40 messages a second per group; over that, high-priority messages (red packets, gifts) go first, and messages stopped by frequency control are “not delivered or stored in the message history”, yet “a success response will be returned to the sender” (Tencent Cloud IM message types documentation). Right for live rooms, where nobody minds one missing like (chapter 41). Wrong for the hiking group: Ana thinks her message went out.
- Limit per caller, and say when to come back. Slack’s API says apps should post no more than one message per second per channel, with short bursts allowed; over the limit it returns HTTP 429 and
Retry-After(Rate limits). Telegram’s API does the same: call one method with the same parameters too often and it returns error 420FLOOD_WAIT_X, where X is the number of seconds to wait (chapter 24, Error handling). - Give requests levels, and refuse the low ones first under overload. Google sorts requests into four levels, from
CRITICAL_PLUStoSHEDDABLE; a client that sees many of its requests refused starts refusing some itself, locally, without sending them (Handling Overload). The same idea as dropping by priority in section 3, placed at both ends of the call. - Switch a busy group entirely to “notify + pull”. Push no messages, only “there are new messages”, and the phone pulls them itself. Useful for groups of tens of thousands and live rooms (chapters 40, 41); a 500-member group at 10 messages a second can still be pushed and doesn’t need it.
7. This chapter’s decision
Limits deal with too much coming in. But the gateways themselves need upgrading too: restart one, and 50,000 devices disconnect together. Next: deploys without disconnects.
Comments
Comments live in GitHub Discussions and need a GitHub account.