IM Systems in Depth

Chapter 35

Deploys without disconnects

The gateways need new versions too. When a gateway restarts, the 50,000 devices on it all drop at once. How do you replace the program on 20 gateways so that users notice nothing, and a bad version never reaches everyone?

This chapter is a first draft. The author is still checking it, and its text and numbers may change.

Chapter 20 moved the logic out of the gatewaysgateway网关A connection-layer server. Each holds many long-lived connections and handles auth, heartbeats and traffic without reading messages. One of the few stateful parts: a restart drops all its connections.See the glossary. The logic ships twice a week and nobody is disconnected. But the gateways need new versions of their own too: to fix their own bugs, or to upgrade the TLS library. This series assumes they ship once a month.

v3 has 20 gateways, each normally holding 50,000 devices (chapter 22). When a gateway’s process exits, all its connections drop.

This chapter’s question: how do you replace the gateway’s program so that users notice nothing, and if the new version has a problem, only a few people run into it?

1. The first idea: restart them one at a time

The most common way is a rolling restart: restart one, wait for it to come up, then the next, 30 seconds apart.

First, what it gets right. The locatorlocator接入调度The service that tells a device which gateway to connect to. The device asks it first, then connects.See the glossary spreads one gateway’s 50,000 devices over the other 19, about 2,600 each; a gateway can do 2,000 handshakes a second, so they are done in 1.3 seconds. Chapter 24 worked this out: it is not a storm.

But two problems remain.

Everyone drops once. Every device briefly shows “Connecting…”: 0.6–0.8 seconds to reconnect, then a sync. At the moment of the drop, the hints being pushed to those devices are lost and the sync makes up for them; messages being sent are resent from the outbox (chapter 3). One deploy means 1 million forced reconnects across the whole service.

A bad version reaches everyone in 10 minutes. 20 gateways, 30 seconds apart: in 10 minutes they have all been replaced. If the new version has a bug that makes 0.05% of pushes arrive late, nobody can see it in 10 minutes. By the time anyone does, it is on every gateway.

2. One gateway first: the canary

Replace just 1 gateway first, let it run for a while, and compare it with the other 19. This is a canary. Google’s SRE workbook defines it as “a partial and time-limited deployment of a change in a service and its evaluation”, and recommends evaluating it with SLIs (Canarying Releases).

The evaluation uses chapter 33’s two instruments:

  • Probes: one a second goes through this gateway. If it cannot push a single one, that shows within seconds.
  • Sampling: this gateway’s 50,000 devices are 5% of everyone online. Chapter 33 samples 1 delivery in every 1,024, so about 7 sampled deliveries a second pass through it, about 24,000 an hour.

How fast a problem shows depends on how big it is. Count it as seen once 10 sampled deliveries are late. Say a new bug makes 0.5% of pushes late: of the 7 sampled deliveries a second, 0.035 are late, so 10 pile up in 5 minutes; a healthy version should have only 0.2 late in those 5 minutes, so the difference is plain at a glance:

New bug makes this share of pushes late Time until it shows Late deliveries from a healthy version in the same time
0.5% 5 minutes 0.2
0.05% 49 minutes 2

So watch a canary for at least an hour, ideally across the evening peak: the SRE workbook also warns that performance problems often appear only under high load. Compare it with the other 19 gateways, not with yesterday.

3. Draining: move them out, then restart

The canary keeps a bad version from reaching everyone. “Everyone drops once” needs something else: before the restart, move the devices off one by one. This is called draining:

  1. Take no new ones. The locator stops giving new devices to this gateway.
  2. Ask the devices to move. The gateway sends its devices a “please switch” frame, 1,000 a second.
  3. Connect to the new one, then close the old. On that frame the phone asks the locator where to connect, connects, has its token checked and syncs, and only then closes the old connection. At no moment is it disconnected.
  4. Restart once empty. 50,000 devices at 1,000 a second drain in 50 seconds. Once the new version is up, the gateway starts taking devices again.

During step 3 the phone is connected to two gateways for a while. Could it get a message twice, or miss one?

  • Online routesonline routes在线路由表Which gateway each online device is connected to. Gateways register devices as they connect, and dispatch looks gateways up here.See the glossary: the new connection overwrites the old row when it registers; when the old connection closes, chapter 20’s rule deletes the row only if the connection ID matches, so it cannot delete the new row.
  • The same push may come on both connections: the phone’s local database keeps only one copy per “sender + message ID” (chapter 7).
  • Anything missed between the two: sync fills it in by record version (chapter 11).

Why 1,000 a second? Each device that moves is one sync. At peak the whole service does about 6,900 syncs a second; draining adds 1,000, which is 14%, and the back end can take it. Move faster and the syncs rise with it. So how fast devices can move is set not by how fast the gateway can handshake but by the slowest step behind it, here the syncs: the same reasoning chapter 24 used for its admission limit.

Old versions of the app don’t know the “please switch” frame: they did not report it as a capability when they connected, so by chapter 31’s rules the gateway does not send it to them. At the end of the drain, the gateway closes the connections that are left, and they reconnect as after any ordinary drop.

The gateways have the same problem: the “please switch” frame is sent by the old version, the one being drained. So the frame must already be in the previous version; the first release that adds it can only be rolled out the old way of section 1, restarting one gateway after another.

4. Where they move to

The 20 gateways are drained one after another. Where the moved devices go matters.

If the locator spreads them evenly over the other 19, many land on gateways not yet drained, and move again when that gateway’s turn comes. In the model, each device moves 1.7 times on average; the later in the order, the fuller the remaining gateways, and the fullest reaches 130,000, over chapter 22’s limit of 100,000 a gateway.

The fix is simple: send them to the gateway with the fewest connections. The one just upgraded is empty, ready to take the devices moving off the next one; 1,000 a second is half of what it can handshake. Then only the first gateway’s 50,000 devices move twice, each device moves 1.05 times on average, and the fullest gateway holds only 53,000.

The canary starts the same way: the first gateway is empty after its upgrade; draining the second fills it with that gateway’s 50,000 devices, and only then does the hour of watching begin.

5. Two versions at once

From the start of the canary until the last gateway is replaced, for over an hour, gateways of the old and new versions run side by side. So:

  • The protocol between the gateways and the logic behind them must be understood by both versions, the same reasoning as old and new apps in chapter 31.
  • If the canary shows a problem, you must be able to roll back: a rollback is also a drain, in the other direction.

The message servers hold state too (chapter 28). They deploy one at a time, each handing its conversations over by chapter 28’s “planned handover”: the old owner stops taking new messages for a conversation and finishes what it holds before letting go, and only then does the new owner take over. Hand over first, then restart: the same idea as draining a gateway.

6. The numbers

  • Restarting a gateway outright: 50,000 devices drop, spread over 19 at about 2,600 each, handshakes done in 1.3 seconds; one deploy, 1 million reconnects.
  • Canary: 1 gateway, 5% of online devices; a 0.5% problem shows in 5 minutes, a 0.05% one takes 49 minutes.
  • Draining: 1,000 devices a second, 50 seconds a gateway, about 17 minutes for all 20; 14% more syncs.
  • Where they move: spread evenly, each device moves 1.7 times and the fullest gateway holds 130,000; to the emptiest, 1.05 times and 53,000.

7. The cost

  • Slow. An hour of canary plus 17 minutes of draining; a straight restart takes only 10 minutes.
  • Two versions online at once: the protocol has to be understood by both, and a rollback is another full drain.
  • The phone has one more thing to know: on “please switch”, connect to the new gateway before closing the old. Old apps cannot, and still drop once.
  • The drain path runs only once a month, and paths rarely walked break easily (drills, chapter 38).

8. Other answers

  • Hot upgrade: hand the open connections to the new process. Not one connection drops, but the TLS keys, sequence numbers and protocol state all have to be handed over too, which is the hardest to get right. Envoy’s hot restart hands over only the listening sockets: the new process takes the ports and accepts new connections, and “the hot restart functionality does not transfer existing connections to the new process”; the old process closes them gradually during a drain period (Envoy hot restart).
  • Let the old process wait it out. When Slack used HAProxy, after a reload the old process was left to keep running “for many hours to allow long-lived websocket connections to drain”; after moving to Envoy, they wrote that hot restart works “without dropping any connections” (Slack): the old process keeps serving existing connections through the drain period, so nobody drops at the moment of the restart.
  • The server tells the client to reconnect, and resumes the session. Discord’s gateway has a Reconnect event, meaning “Server is going away, client should reconnect to gateway and resume”; resuming sends the session ID and the last sequence number received, and the gateway replays the missed events in order (Gateway, Gateway Events). This chapter’s “please switch” plus sync is the same idea.
  • QUIC connection migration. It lets a connection survive the client changing networks; but the spec says “only clients are able to migrate in this version of QUIC”, and moving to another server address in the middle of a connection is not supported (RFC 9000, sections 1 and 9.6). It does not help with changing gateways.

9. This chapter’s decision

Deploys no longer disturb anyone. But what if the gateways are fine, and the storage behind them slows from 5 milliseconds to 500? Nothing is down; everything is getting slower. Next: when a dependency slows down.

Comments

Comments live in GitHub Discussions and need a GitHub account.