# Chapter 20: Split the gateway

> A million devices online at once will not fit on one server; copy the server a few times, and every deploy still disconnects everyone. How do you separate the connections from the logic?

IM Systems in Depth · https://im.liko.page/en/split-the-gateway/

The app has grown to 10 million daily users, with 1 million devices online at once at peak. This is v3.

Up to v2, the server was one program: it held every long-lived connection and also ran all the logic: dedup, taking the seq (a message’s number within its conversation), writing conversation records, pushing. A million connections do not fit on one machine. The obvious fix is to run more copies, but then every deploy still disconnects everyone at once.

This chapter decides: the part that holds connections is separated from the logic.

## 1. The first idea: run a few copies

Copy v2’s server 20 times and put a load balancer in front, so each copy holds a twentieth of the connections. This is a reasonable first step, and many systems start this way.

It does fit: 50,000 connections a machine, at 30 KB each, is 1.5 GB of memory (how many a machine should hold is assumed here and worked out in chapter 22). The new problem it creates is easy to solve too: Ana is connected to machine 3 and Ben to machine 7; once machine 3 has accepted Ana’s message, it needs to know which machine Ben is on. That table is section 4 below, and it is needed whether or not you split.

What copying cannot fix is deploys.

## 2. A deploy disconnects everyone

The logic changes often: bug fixes, new message types, new rules. The series assumes two deploys a week. Each deploy restarts the 20 servers one by one, and when a process exits, every connection it holds is dropped.

> 2 a week × 1 million connections = **2 million reconnects a week**

As a count, that is not much: every app open makes a new connection anyway, 200 million a day (chapter 26). The cost lies in two things:

- **It drops people who are using the app.** Ana is in the middle of a chat, and “Connecting…” flashes on her screen. It has nothing to do with her network; the server is being deployed. Reconnecting takes 0.6–0.8 seconds (chapter 2), and pushes missed in those seconds come back through the sync after reconnecting.
- **They all come at once.** Restart one machine and its 50,000 devices connect back together. At 2,000 handshakes a second per machine (chapter 2), the handshakes alone take 25 seconds, followed by 50,000 syncs.

Bugs in the logic do the same. A new message type crashes the parsing code, the process exits, and that machine’s 50,000 connections go with it. A bug in the logic becomes a connection incident.

One more thing: the two halves grow differently. Connections eat memory; even an idle one holds 30 KB. Logic eats CPU and follows message volume, 13,900 a second at peak. Copying a whole server buys both together.

## 3. Split it: the gateway only holds connections

Cut the program in two:

- **The gateway** only holds connections. When a frame comes in, it hands it to the right logic service according to the frame’s type; when the logic sends a frame down, it writes it to the right connection. It does not look at message content.
- **The logic**: the message layer, dispatch layer and business layer, all behind the gateway, deployed on their own. They hold no connections. Restart one machine and the gateway sends requests to the others; the phone notices nothing.

With several gateways, which one does a phone connect to? Before connecting it asks the locator, which answers with an address based on how loaded each gateway is.

What goes into the gateway and what stays out follows one test: **does it change often?**

| In the gateway | Not in it |
|---|---|
| TLS, framing (chapter 2, side trip “Framing”) | Dedup, taking the seq (chapters 4, 5) |
| Heartbeats, to find dead connections (chapter 23) | Group members, who may message whom (business layer) |
| Checking the login token (chapter 25) | Message types, message content |
| Queueing when connections come too fast (chapter 24) | Conversation records, unread counts |
| Registering in and removing from the routes table; telling the logic when a connection closes | |

The left column changes only a few times in years; the right one changes twice a week. As long as nothing from the right goes into the gateway, adding a message type does not mean deploying the gateway. The gateway still has to be deployed now and then (to fix its own bugs, to upgrade the TLS library), just much less often. Say once a month: reconnects caused by deploys drop from about 8.6 million a month (about 4.3 weeks of 2 million) to 1 million (one gateway deploy, each of the 1 million connections dropped once).

## 4. A push has to find the person: the online routes table

With many gateways, two kinds of things go down to devices, and they find their connection differently:

- **Replies**: Ana sends a message, and the ACK has to get back to her. The gateway remembers which connection the request came in on, and the reply goes back the same way, with no table to look up.
- **Pushes**: Ben asked for nothing; the server comes looking for him. For that, the server needs to know which gateway each of his devices is connected to.

So there is one more table, the online routes:

> person, device → gateway, connection ID, time connected

When a device connects, the gateway writes a row (overwriting any old one); when it disconnects, the row is deleted. Before pushing, the dispatch layer looks the person up once, gets each of their online devices and its gateway, and hands the frame to those gateways. If nothing is found, the person is treated as offline, and the message goes by OS push (chapter 17).

The table is small but busy:

- **Size**: 1 million rows at 64 bytes each, about 64 MB.
- **Writes**: one when the app is opened and connects, one when it closes and disconnects; about 13,900 a second at peak.
- **Reads**: one lookup per person a message reaches. At peak, 13,900 messages a second each reach 6.67 people on average (10 devices a message, 1.5 devices a person, chapter 6): about **92,700 lookups a second**, around 9% of one cache node assuming a cache node answers 1 million lookups a second (pipelined).

## 5. One thing to get right: check the connection ID before deleting

Deleting the row on disconnect sounds simple. Watch Ben’s phone:

| Time | What happens | Ben’s phone in the table |
|---|---|---|
| 19:00 | On home Wi-Fi, connects to gateway 3, connection c1 | gateway 3, c1 |
| 19:05 | He goes out; the Wi-Fi silently drops; the phone switches to 4G and connects to gateway 7, connection c2 | gateway 7, c2 |
| 19:17 | Gateway 3 has heard nothing on c1 for 12 minutes (3 heartbeat intervals, chapter 23), closes c1 and deletes the row by device | **gone** |

The phone is plainly connected to gateway 7, yet the table has no row for it. From then on, messages to Ben are not pushed; he sees them only at his next reconnect and sync. The old connection died silently, the gateway took over ten minutes to notice, and that late delete wiped out the new row.

The fix is to include the connection ID in the delete: **delete only if the row is still c1**.

```sql
DELETE FROM online_routes WHERE device = 'ben-phone' AND conn = 'c1'
```

The delete at 19:17 now changes nothing, and gateway 7’s row stays. The same holds when the table lives in a cache: the compare and the delete must happen in one step. In two steps, read “still c1”, then delete, gateway 7 may write c2 in between, and the delete removes c2.

When a whole gateway dies, its 50,000 rows go stale together. Pushes to it fail; those devices soon connect to other gateways and overwrite their own rows (chapter 24); for those that do not come back, one failed push is enough to treat them as offline.

## 6. The numbers

- **One machine is not enough**: 1 million connections × 30 KB = 30 GB, while a gateway leaves 16 GB for connections; at peak there are 139,000 pushes a second and one machine pushes 50,000, so pushes alone need 2.8 machines. Assuming 50,000 connections each normally, that is 20 gateways (chapter 22 works this number out: 10 in each of two availability zones, so that if one zone fails each remaining gateway holds 100,000).
- **Deploys**: before the split, 2 million reconnects a week; after it, deploying the logic drops nothing, and the gateway deploys once a month, 1 million a month.
- **One more hop**: gateway to logic is inside the data centre, about 0.5 ms a round trip; phone to gateway is 200 ms, so only 0.25% more.

## 7. The cost

- **One more hop, one more service.** A frame crosses the network once more between gateway and logic, and that leg can time out too; the routes table is read on every push, so when it is slow, pushes are slow.
- **The table goes stale.** When a gateway dies or a connection drops silently, its rows stay wrong for a while; deletes must check the connection ID.
- **Deploying the gateway still disconnects.** Less often, but not never. How a gateway moves its connections away gradually before restarting is for chapter 35.

## 8. Other answers

- **One connection layer shared by many products, knowing none of them.** Baidu has written publicly about such a unified long-connection service (百度Geek说 (Baidu Geek Talk), glstr, “千万级高性能长连接Go服务架构实践” (Building a high-performance Go long-connection service for tens of millions), [reposted by 52im](http://www.52im.net/thread-4623-1-1.html)): an access layer holds the connections; a routing layer keeps “device ID → connection ID”, and the products push by device; downstream it supports unicast, batched unicast and broadcast. Among the lessons the article sums up: “stick to the principle that the long-connection service does not get deeply involved in business logic”.
- **Gateways subscribe by channel.** Slack’s gateway servers track which channels each connection subscribes to; a message first goes to a channel server, chosen by consistent hashing, and then to every gateway subscribed to that channel ([Real-time Messaging](https://slack.engineering/real-time-messaging/)). The gateway has to know about channels, not just “device → connection”.
- **No routes table: send to every machine.** Socket.IO’s Redis adapter: a packet for a group of people goes first to the local connections, then is published to Redis; every other server receives it and sends it to its own matching connections ([Redis adapter](https://socket.io/docs/v4/redis-adapter/)). With few machines this is the least work; with chapter 22’s 20 gateways (10 in each of two availability zones), every message goes to all 20, 278,000 frames a second, and every gateway has to know who is in every group.
- **Hot upgrade.** When the gateway is deployed, the open connections are handed to the new process, and not one drops. It saves the reconnects, at the price of code that is very hard to get right.

## 9. This chapter’s decision

**Decision card**

- Problem: 1 million connections do not fit on one machine; copying the whole server fits them, but the logic is deployed twice a week and each deploy disconnects everyone (2 million reconnects a week), and a bug in the logic drops connections too.
- Choice: The gateway only holds connections and forwards frames, knows nothing of the business, and is rarely deployed; the logic behind it is deployed on its own. Pushes find the gateway through the online routes table (person, device → gateway, connection ID); replies go back the way they came; deleting an entry checks the connection ID.
- Cost: One more hop, and one more service read on every push (about 92,700 a second at peak); the table goes stale; deploying the gateway itself still disconnects.
- Revisit when: How many connections per gateway (chapter 22); reconnect storms when a gateway restarts (chapter 24); checking the login token at the gateway (chapter 25); splitting the message layer across machines (chapter 28).
- Other answers: A connection layer shared by many products and knowing none of them (Baidu); gateways subscribing by channel (Slack); no routes table, send to every machine (Socket.IO Redis adapter); hot upgrade.

The online routes table knows where each device is connected. So should it also tell Ana that Ben is online right now? Whom to tell, and how many people: next, presence.
