Chapter 22
A million connections
How much memory does a connection that carries nothing take? For 1 million connections, how many gateways?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
Chapter 20 moved the connections into the gatewaygateway网关A connection-layer server. Each holds many long-lived connections and handles auth, heartbeats and traffic without reading messages. One of the few stateful parts: a restart drops all its connections.See the glossary. At v3’s peak there are 1 million connections. Most of the time they carry not a single byte, yet each one takes memory.
This chapter answers one question: how many connections one gateway should hold, and so how many gateways are needed.
1. What an idle connection costs
The series’ assumptions for a connection that carries nothing:
| Item | Size | What it is |
|---|---|---|
| Stack | 8 KB | Each connection has a goroutine (Go’s lightweight thread) waiting to read. It starts at 2 KB and grows once it runs deeper calls such as the TLS handshake; here we count 8 KB |
| Read buffer | 4 KB | For reading bytes from the connection |
| Write buffer | 4 KB | For writing bytes to the connection |
| TLS | 10 KB | The encryption state and its own send and receive buffers |
| Kernel | 4 KB | The operating system’s record and buffers for this TCP connection |
| Total | 30 KB |
1 million of them are 30 GB. These are round numbers chosen for the example; the real ones depend on the language and libraries. Go 1.4 cut a goroutine’s initial stack from 8 KB to 2 KB (Go 1.4 release notes); OpenSSL says one of its switches saves about 34 KB per idle connection (SSL_CTX_set_mode), so the TLS item can be large or small.
Note which items are needed even when idle: only the kernel’s share and the TLS keys. The stack and the buffers are there for when data arrives, and sit unused most of the time.
2. The first idea: as many as possible on one machine
Fewer machines are easier to run. Say a gateway sets aside 16 GB for connections: 16 GB ÷ 30 KB is 530,000. Two machines would take all 1 million.
This is not just theory. In 2012 WhatsApp held 2 million TCP connections on one server with 24 cores and 103 GB of memory (WhatsApp blog); in 2015 the Phoenix framework connected 2 million WebSocket clients to one machine with 40 cores and 128 GB (Phoenix blog).
But a gateway does more than hold connections. It also pushes, and it also restarts.
3. Which fills first
Add connections to one gateway from 0 upward and watch three things. The series assumes one gateway can push 50,000 times a second.
- Memory: 30 KB each, full at 530,000.
- Peak pushes: at v3’s peak there are 13,900 messages a second, each pushed to 10 devices on average, so 139,000 pushes a second; spread over 1 million connections, 0.139 a second per connection. At 360,000 the machine cannot keep up.
- Burst pushes: chapter 29’s burst is 10 times the all-day average of 4,630 messages a second and lasts a minute; at 10 pushes per message that is 463,000 pushes a second, 0.463 a second per connection. Full at 110,000.
Memory is the last to fill.
One more thing is not about filling up but about how many people drop at once. Gateways restart, and gateways die. With 100,000 on a machine, losing it disconnects 10% of those online; with 530,000, more than half. Everyone disconnected has to handshake again; at 2,000 handshakes a second per machine, 100,000 take 50 seconds and 530,000 take over 4 minutes (how other gateways share the load is chapter 24).
4. The decision: 100,000 per machine
Each gateway holds at most 100,000 connections, just under the burst line:
- 1 million connections need 10 machines.
- Assume further that the region has two availability zones (two groups of data centres with independent power and networks), and that when one fails entirely, the other has to take everyone: 10 machines on each side, 20 in all.
Normally each machine holds 50,000; if one zone fails, each remaining machine holds 100,000:
| Connections per machine | Memory | Peak pushes | Burst pushes |
|---|---|---|---|
| 50,000 (normal) | 1.5 GB | 6,950/s | 23,150/s |
| 100,000 (one zone down) | 3 GB | 13,900/s | 46,300/s |
A gateway pushes 50,000 a second and leaves 16 GB for connections. At the worst moment, pushes are at over 90% and memory under a fifth.
All of these come from the machine this chapter assumes. How many connections to put on a gateway depends on the actual machine: on different hardware, first measure how many pushes a second it can make, how many handshakes, and how much memory it can leave for connections; then work it out the same way, take whichever line comes first, and keep half free for a lost zone. When later chapters say “a gateway”, they mean this assumed machine: 50,000 connections normally, 100,000 at most.
So at this scale, 30 KB per connection is not a problem, and there is no need to change code to save memory. Published practice lands in the same range: an article from Baidu says its unified long-connection service keeps each instance to 100,000–200,000 connections, “to avoid a single instance holding too many connections, so that when an instance has a service hiccup, it does not have a large impact on the whole cluster” (百度Geek说 (Baidu Geek Talk), glstr, “千万级高性能长连接Go服务架构实践” (Building a high-performance Go long-connection service for tens of millions), reposted by 52im). The documentation of Centrifugo, an open-source real-time messaging server, is more cautious: “we suggest not putting more than 50-100k clients on one node - but you should measure for your use case” (Centrifugo FAQ); at the other end are WhatsApp and Phoenix in section 2, 2 million on one machine.
5. To make gateways bigger, what to save first
If push capacity goes up (more cores, batched pushes) and you want 500,000 or 1 million on one machine, memory becomes the next wall. What can be saved are the items not needed while idle, from the smallest change to the largest:
| Step | Saves | Left per connection | 1 million | Cost |
|---|---|---|---|---|
| Take read and write buffers from a pool only when needed, and return them after | 8 KB | 22 KB | 22 GB | Small: one more pool; in a burst every connection wants a buffer at once, and the pool suddenly grows |
| Let TLS release its send and receive buffers while idle | 8 KB | 14 KB | 14 GB | Depends on the TLS library: OpenSSL has a ready switch; some libraries do not, and the only way is to change library |
| Keep no goroutine for idle connections: wait with epoll (Linux’s way of watching many connections at once) until there is data to read, then take one from a pool to handle it | 8 KB | 6 KB | 6 GB | Large: the code changes from “one straight-line piece of code per connection” to an event loop, hard to write and hard to debug |
(The TLS step assumes 2 KB of state is left once the buffers are released.) The remaining 6 KB, the kernel’s share and the TLS keys, cannot be saved even when idle.
Among public examples, an article from Mail.Ru (Sergey Kamardin, 2017) worked it out for 3 million connections: two goroutine stacks per connection, 4 KB read and write buffers each, and the buffers left by the standard library’s HTTP server came to 72 GB “for the application that does nothing yet”, about 24 KB per connection. They waited for readability with epoll and took buffers and goroutines from pools, saving most of it (A Million WebSockets and Go).
Many goroutines cost one more thing: Go’s garbage collector has to scan them. The Baidu article says that with the common model of two goroutines per connection, 1 million connections on an instance are 2 million goroutines, and since connections are created often, this “puts enormous pressure on the service’s GC”; it changed the writing half into one shared “downstream task group”.
6. One line of configuration
Each connection counts as an open file in the system, so 100,000 in one process need more than 100,000 file descriptors. Linux’s default soft limit is often 1,024 and has to be raised in systemd; since Go 1.19, Go raises the soft limit itself (chapter 2). Today it is a line of configuration; the real limits are the ones above.
7. The cost
- More machines. 20 gateways, each using only a tenth of its memory on a normal day. The extra machines buy this: when one fails, only 5% to 10% of people drop.
- The headroom is for bad days. On a normal day, pushes use just over a tenth; they reach over 90% only when a zone is down and a burst comes at the same time.
- Measure these numbers yourself. 30 KB, 50,000 pushes a second, 2,000 handshakes a second are all the series’ assumptions; with another language or TLS library, what fills first may change.
8. Other answers
- A few big machines. WhatsApp, 2 million connections on one; Phoenix, 2 million clients on one (section 2). Fewer machines and easier to run, at the price of more people dropping when one dies, and every memory-saving step has to be taken.
- Many small instances, 100,000–200,000 each (Baidu, section 4). This is also this chapter’s choice.
- A goroutine per connection, or an event loop. The first is easy to write but holds a stack even when idle; the second saves memory but is hard to write (section 5).
9. This chapter’s decision
Most of the 1 million connections are silent most of the time. But a silent connection may have died long ago without the gateway knowing. Next: heartbeats.
Comments
Comments live in GitHub Discussions and need a GitHub account.