Chapter 24
Reconnect storms
Every gateway in one availability zone stops, and 500,000 phones drop and come back at once. The remaining gateways can do 20,000 handshakes a second, so everyone should be in within 25 seconds. Why do half of them wait a minute or two?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
Chapter 22 split v3’s 20 gatewaysgateway网关A connection-layer server. Each holds many long-lived connections and handles auth, heartbeats and traffic without reading messages. One of the few stateful parts: a restart drops all its connections.See the glossary across two availability zones (two groups of data-centre rooms with separate power and networks), 10 in each, so that if one zone stops entirely, the other can still hold everyone.
One afternoon, a zone stops. The 500,000 phones on its 10 gateways drop at once, come back at once, and pile onto the other 10. This chapter answers: if there is room for them, why can’t they get in, and how do we bring them back quickly and in order?
1. One gateway first: not a storm
By the series’ assumptions, a gateway can do 2,000 full TLS handshakes a second (chapter 2). Normally it holds 50,000 devices (chapter 22); if they all came back to it, that would take 25 seconds.
But v3 has a locatorlocator接入调度The service that tells a device which gateway to connect to. The device asks it first, then connects.See the glossary (chapter 20): the phone first asks it which gateway to connect to, and it spreads these 50,000 over the other 19 gateways, about 2,600 each. They are all connected in 1.3 seconds. A gateway restart needs nothing more.
If they all went back to the same gateway (no locator, or a locator that sees it as the emptiest and sends everyone to it), 50,000 would squeeze into 1. That is exactly the zone case below, only ten times smaller.
2. A zone: the storm
500,000 phones, 10 gateways left, 20,000 handshakes a second between them: 25 seconds at the very least.
Today’s client is chapter 3’s: reconnect at once when the connection drops; on failure, retry with a backoff that doubles, 1, 2, 4, 8 seconds…, with random jitter added to each wait. Two more assumptions here: a phone counts a handshake as failed if it is not done in 10 seconds; and the backoff waits at most 32 seconds.
On the gateway side, every connection is accepted as it comes (accepting is cheap), and the handshakes all queue for the CPU with no limit. That is how most servers behave by default, for example Go with one goroutine per connection. The model does them one at a time in arrival order; a real server lets all the handshakes take turns on the CPU, so each one is slower.
The results:
- Backoff without jitter: half the phones are connected only after 87 seconds, all of them after 132; on average each phone handshakes 4.4 times.
- Backoff with jitter (chapter 3’s client): slower instead. Half take 121 seconds, all 150; 5.3 handshakes per phone.
Both are several times slower than 25 seconds, and the gateways are busy at full load the whole time. Busy with what?
3. Working for people who have left
In the first 10 seconds, the 10 gateways finish 200,000 handshakes. The other 300,000 phones wait the full 10 seconds, give up, and get ready to retry.
But their connection requests are still in the gateways’ queues. A gateway does not know the other side has left: the phone’s first handshake packet has been sitting in the buffer, so the gateway reads it as usual, does the most expensive step, the signature (the gateway signs with its certificate’s private key to prove it is the real server; chapter 2), and only on replying finds the connection already closed. For the next 15 seconds, the gateways do nothing but this wasted work.
The retrying phones queue behind this wasted work; they cannot get through within 10 seconds either, give up again, and leave another batch of wasted work behind. In one zone’s storm, each phone takes 4.4 to 5.3 handshakes on average and only the last one counts: the gateways waste about 1.7 to 2.2 million handshakes this way.
This is also why jitter made things slower: the jitter simulated here (chapter 3’s) picks a moment anywhere between 0 and this round’s backoff, so some phones come back earlier than without it, right into a queue still full of wasted work. The jitter itself is not wrong, since it spreads the retries out; what is wrong is that the gateways are still doing wasted work.
4. Admission control: say no at once to what can’t be done
The fix is on the gateway: accept only what it can get done in time.
- Cap the handshake queue: each gateway queues at most 5 seconds’ worth (at 2,000 a second, 10,000). That is half the phone’s 10-second timeout, so everything that gets in can be done before the phone gives up.
- Close the extra connections at once, before the TLS handshake. This step does no cryptography and costs almost nothing.
- Rejected phones come back with chapter 3’s backoff and jitter. The client does not change.
The same zone stops: half the phones are connected in 12.4 seconds, 99% in 38, all in 43 seconds; each phone handshakes only once, and no work is wasted. The gateways reject about 1.6 million connections, 3 per phone on average, each just picked up and hung up at once.
Should the first reconnect be spread out too, say at a random moment in the first 10 seconds? The figure tries it: when a zone stops, the result is about the same; when one gateway restarts, what took 1.3 seconds now takes 10. It makes every disconnect slower, so we leave it out.
5. There is more behind the gateway
Connecting is only the first step. Every phone that connects syncs at once (chapters 6 and 11): it pulls the changes to its conversation records, then asks again for the latest seq (the contiguous number each message gets within its conversation, chapter 5) of its 200 conversations.
At v3’s normal peak there are about 6,900 syncs a second (200 million app opens a day, with the peak at 3 times the average). When one gateway’s devices are spread over 19, 50,000 phones connect in 1.3 seconds, about 38,000 a second: 5.5 times the normal peak. When a zone stops, 10 gateways connect 20,000 a second: 2.9 times. Once the handshakes hold, the pressure moves to the services behind them.
So the admission cap is not set by the handshakes alone, but by the slowest part behind them; and it is the total that counts, not each gateway’s own number. The caps of all the gateways added up are what the services behind must take: add gateways, and the total grows with them. Say each gateway lets in 2,000 phones a second: with 10 gateways left after a zone is lost, the services behind take 20,000 syncs a second; grow that zone to 20 gateways without changing the per-gateway cap, and they must take 40,000, though the sync service has not grown.
That is what happened to Discord in an outage in March 2026: a configuration change stopped 17% of sessions and the clients reconnected together. The rate limits had not yet been retuned for the new architecture after a migration; there were more machines than before, so “the per-host rate limit allowed more capacity through”, and memory on all the gateway nodes in that availability zone shot up and began to run out (Discord’s write-up).
6. The numbers
- One gateway: 50,000 devices ÷ 2,000 a second = 25 seconds; spread over the other 19, about 2,600 each, 1.3 seconds.
- One zone: 500,000 devices ÷ 20,000 a second = 25 seconds at least; phones give up after 10 seconds, so only 200,000 make it in the first round.
- With admission: everyone in 43 seconds, 1 handshake per phone; without it, 4 to 5 per phone.
- Behind the gateways: during a reconnect, syncs per second run 2.9 to 5.5 times the normal peak.
7. Cost
- Some wait longer than they need to. The capacity is enough for 25 seconds, yet the last phone takes 43: rejected phones sleep in their backoff, and the gateways are sometimes idle.
- One more number to tune. The per-gateway cap and the total must be recomputed whenever the number of machines changes; Discord’s outage came from not having recomputed after a migration.
- Rejecting costs something too: 1.6 million picked up and hung up is a lot, but each is cheap (an assumption: a rejection without TLS is far cheaper than a handshake).
A planned restart needs no storm: move the devices off in batches first, then restart. That is chapter 35.
8. Other answers
- Limit how many sessions start in a period: Discord’s gateway tells each application its
max_concurrency, the number of new sessions allowed to start every 5 seconds (gateway docs). Same idea as capping the queue by time in this chapter. - Only the client’s backoff and jitter: Marc Brooker’s article compares several backoffs: without jitter is worst, both slow and doing more work; among those with jitter, “full jitter” does the least work (AWS Architecture Blog, 2015). It spreads the retries out, but as section 3 showed, while the server is still doing wasted work it is not enough on its own. Ably also recommends randomized exponential backoff for reconnects, with the server rejecting some new connections when it is close to overload (Ably).
- Make handshakes cheaper: TLS session resumption skips the most expensive signature (chapter 2); each gateway can take more a second, and the mechanism stays the same.
- The server tells the client “come back in N seconds”: a wait time sent with the rejection can pull 43 seconds closer to 25; but the rejection comes before TLS, when there is no encrypted channel yet to carry that number, so the protocol needs another way to say it. Telegram’s API has both: once connected, a call made too often gets
FLOOD_WAIT_X, where X is the number of seconds to wait; but at the MTProto transport layer, when too many connections come from the same IP in a short time, the server sends back only a 4-byte error code, -429 (transport flood), with no wait time in it. - A cold start behind the gateways: Slack’s write-up of its 4 January 2021 outage says that on the first day after the holidays “client caches are cold and clients pull down more data than usual on their first connection” (Slack). The sync after reconnecting is the same bill.
9. This chapter’s decision
Now the gateway decides who gets through the door first. But who is it letting in? How does a gateway know this phone really is Ana, without asking the app’s login service every time? Next: accounts.
Comments
Comments live in GitHub Discussions and need a GitHub account.