# Chapter 32: v3: a million online

> With a million devices online at once, how many machines does each layer need? And when a gateway suddenly dies, what happens?

IM Systems in Depth · https://im.liko.page/en/v3-million-online/

When v3 begins, the app has 10 million daily users, 1 million online at peak and 13,900 messages a second. v2 put every connection and all the logic on one server: it cannot hold 1 million connections, and every deploy disconnects everyone (chapter 20).

This chapter makes no new decisions. It does four things: draws the whole of v3 and follows one message through it; puts the twelve decisions into one table; redoes the numbers layer by layer from v2 to v3; and finally kills a gateway to see how the whole system catches it.

## 1. The whole of v3

Compared with v2, the one server has split into five main kinds:

- **Gateways**: only connections, heartbeats, auth and forwarding frames; they do not understand messages (chapters 20, 22–25).
- **Online routes**: which gateway, and which connection on it, each online device is on (chapter 20).
- **Message servers**: each conversation belongs to one; they spot duplicates and assign seqs (chapter 28).
- **The message log**: once a message is written to it, it is ACKed; the consumers behind it each go at their own pace (chapter 29).
- **Storage**: messages, conversation records, read positions.

Ana sends “What time do we meet on Saturday?” in the hiking group. Its path:

1. Ana’s phone is connected to gateway 7; the message comes in there.
2. The gateway finds the hiking group’s message server by conversation ID and forwards the frame.
3. The message server checks for a duplicate, assigns seq 4,823, appends it to the log and ACKs.
4. Behind the log: the recent-message cache stores it first and updates the latest seq; push looks up in the online routes which gateway each member’s devices are on and pushes there; conversation records get one row changed for each of the 500 members; storage writes it a little later.
5. Ben’s phone, on gateway 12, receives 4,823.

## 2. v3’s twelve decisions

| Chapter | Decision | Cost | Revisit when |
|---|---|---|---|
| [20 Split the gateway](/en/split-the-gateway/) | Gateways only hold connections and forward frames, know no business logic, and are rarely deployed; the logic behind them deploys on its own; push finds the gateway through the online routes (device → gateway, connection number), and deleting an entry checks the connection number | One more hop; the routes are read on every push (about 92,700 a second at peak); they go stale; deploying a gateway itself still disconnects | Gateway restarts (chapter 24); deploys without disconnects (chapter 35) |
| [21 Presence](/en/presence/) | Tell only those who are looking: a device reports the whole list of people on its screen, and changes are pushed only when they happen; subscriptions follow the connection and are resent on reconnect; going offline waits a little before it counts | A list sent on every screen change (about 27,800 a second at peak); going offline shows late | Presence for a whole contact list in workplace chat (chapter 39) |
| [22 A million connections](/en/million-connections/) | On the machine assumed in chapter 22, at most 100,000 connections per gateway (set by pushes in a burst and how many one restart disconnects, not by memory), 50,000 normally: 20 gateways across two availability zones; no code changes to save memory for now | Many machines, mostly idle; the limit has to be measured yourself | When push capacity grows and bigger gateways are wanted |
| [23 Heartbeats](/en/heartbeats/) | The client sends a heartbeat after 4 minutes of sending nothing, under the NAT’s 5 minutes; no reply in 10 seconds means reconnect; a gateway that hears nothing for 12 minutes closes the connection and deletes the route | A dead connection can take up to 12 minutes for the gateway to notice; the NAT timeout is a guess | Different networks, different intervals |
| [24 Reconnect storms](/en/reconnect-storms/) | Gateway admission: queue at most 5 seconds’ worth of handshakes and close the rest before TLS; refused phones back off with jitter and try again; the first reconnect is not spread out | When a zone goes down, the last device takes 43 seconds; about 1.6 million refusals | Moving devices off before a planned restart (chapter 35); losing a data centre (chapter 37) |
| [25 Accounts](/en/accounts/) | The app’s server signs a 1-day token, which gateways check with a public key, calling no one; the IM has its own user IDs plus a mapping; revocation is one timestamp per person; guests get temporary identities | The private key matters most; tokens need renewing; revocation is only checked on connect; deleting an account has to reach the IM’s history | One IM service for several apps (chapter 44) |
| [26 App lifecycle](/en/app-lifecycle/) | A connection state machine driven by OS events: going to the background first tells the server to switch to OS push, then closes the connection; back in the foreground, connect and sync; a network change reconnects at once | A connection and a sync on every open; the rules change by platform and phone maker | When a new OS version changes the background rules |
| [27 The web client](/en/web-client/) | The whole browser is one device: one connection, one local database, one tab leading; reads are reported only when visible; the server sends pings; long polling as a fallback | New code to pick the leader; background tabs cannot rely on timers; the web is usually a separate codebase | Products where the web is the main client (chapters 39, 43) |
| [28 Sharing out the conversations](/en/message-owners/) | Each conversation has one owner, assigned on a consistent-hash ring; counters and recent message IDs live in the owner’s memory; hand-off by lease, the new owner continuing from the last seq | The message layer now has state and pauses during hand-offs; one more coordination service; a hot conversation sits on one machine; seqs may have gaps | A group too hot for one machine (chapters 40, 41) |
| [29 A queue before storage](/en/queue-before-storage/) | ACK once the message is in a log partitioned by conversation; cache, push, conversation records and storage each consume it; make a message fetchable before notifying; a new owner continues counting from the cache | One more log cluster; at ACK time a message may not be in storage yet, and conversation records fall minutes behind at peak; consumers must be able to redo work | Whether messages actually arrived (chapter 33); storage slowing down (chapter 36) |
| [30 One SDK, many platforms](/en/sdk-architecture/) | Connection, sending, sync and the local database written as one core, compiled into every native app; the platform layer holds only what depends on the OS | A build per platform; every call crosses a boundary; a bigger app | The web client (chapter 27); end-to-end encryption (chapter 42) |
| [31 Old apps, new servers](/en/compatibility/) | Old apps store what they don’t recognise and show fallback text; version and capabilities reported on connect; logins below the minimum version refused | The protocol carries old things for years; the server has per-version paths | Old and new versions online together during a deploy (chapter 35) |

By layer: five chapters on the connection layer (20, 22–25), two on the message layer (28, 29), one on dispatch (21), four on the client (26, 27, 30, 31).

## 3. v3’s numbers

Both columns use the same assumptions: v2 is one server, 10,000 online, 139 messages a second at peak; v3 is 1 million online, 13,900 a second. Each message reaches 10 devices and 6.67 people on average.

**Servers (at peak)**

| Layer | Resource | v2 | v3 |
|---|---|---|---|
| **Gateways** | Machines | 1, together with the logic | 20, 50,000 connections each; 10 would hold them all, the other half is kept for losing an availability zone (chapter 22) |
| | Memory | 10,000 × 30 KB = 0.3 GB | 30 KB per connection, 30 GB in all, 1.5 GB per gateway; no code changes needed to save memory (chapter 22) |
| | Pushes | 1,390 a second | 139,000 a second, 6,950 per gateway, limit 50,000 |
| | Heartbeats | 42 a second | about 4,170 a second, 208 per gateway (chapter 23) |
| | New connections (TLS handshakes) | 69 a second | about 6,940 a second (one per app open, chapter 26), 347 per gateway, limit 2,000 |
| | Network | about 2.2 Mbit/s of pushes | about 11 Mbit/s per gateway, about 14,000 packets a second |
| | Deploys | each deploy drops all 10,000 people, twice a week | deploying the logic disconnects nobody; done v2’s way, it would be 2 million reconnects a week (chapter 20) |
| **Message servers** | Machines | the same one | one owner per conversation, 4 machines: assuming 5,000 messages a second each, 3 cover the peak, 1 is spare for failures; about 3,480 a second each (chapter 28) |
| | seq | one counter row per conversation in storage (chapter 5) | assigned in the owner’s memory; a new owner continues counting from the recent-message cache (chapters 28, 29) |
| | Recent message IDs | about 4,400 over 32 seconds, 142 KB (chapter 4) | about 6.11 million over 5 minutes that include the new-year minute, 196 MB, spread across the machines (chapter 29) |
| **Online routes** | Entries | a table in the process, 10,000 entries | 1 million rows × 64 bytes = 64 MB (chapter 20) |
| | Writes | about 139 a second (connects, disconnects) | about 13,900 a second |
| | Reads | 927 a second (once per recipient of each message) | about 92,700 a second |
| **Message log** | Appends | none (the to-do is a row in the message’s transaction) | 13,900 a second; 46,300 in the first minute of the new year (chapter 29) |
| | Disk | — | 80 GB a day; kept 3 days, 3 copies, 720 GB |
| **Storage** | Writes (rows a second) | messages and counters 278; conversation records 1,853; read positions and one-to-one receipts at most 1,853 each. About 5,800 in all, 30% of one storage node (chapter 19) | messages 13,900; conversation records about 185,000; read positions and one-to-one receipts at most about 185,000 each. About 570,000 in all, the write capacity of about 28.5 storage nodes: it has to be sharded |
| | Capacity | 0.8 GB of text a day; 20 million conversation records × 51 bytes, about 1 GB | 80 GB of text a day, 29 TB a year, 3 copies; 2 billion conversation records, about 102 GB; 4 TB of photos a day |
| | Sync | 69 a second, each reading only the changed records (chapter 11) | about 6,940 a second |

The biggest item is not the messages but **conversation records and read positions**: each message changes one row for each of 6.67 people, each read changes another, and in one-to-one chats the sender’s row changes too (chapter 15). Over nine tenths of v3’s storage writes are these. This is the bill, at v3’s scale, for v2’s choice of “more writes for fewer reads” (chapter 19); groups of tens of thousands do not take this path (chapter 40).

**Ben’s phone (one day)**

| Resource | v2 | v3 |
|---|---|---|
| Connections | one per app open, 20; and disconnected once more by each deploy, twice a week | one per app open, 20; deploying the logic no longer disconnects; the token (chapter 25) plus version and capabilities (about 10 bytes) are sent when connecting (chapter 31) |
| Traffic | about 500 bytes per sync: a dozen or so changed records, 39 bytes each (chapter 19) | the same; presence is subscribed only for the people on screen (chapter 21) |
| Battery | the background was not settled | heartbeats only in the foreground, after 4 minutes of sending nothing, at most 15 an hour; staying connected in the background all day would be 360, about 1 hour of radio (chapter 23), so the connection is closed in the background (chapter 26) |
| Background | woken by OS push (chapter 17) | the same; when the network changes it reconnects and finishes syncing in about 2 seconds, instead of relying on heartbeats, which take over two minutes on average to notice (chapter 26) |
| Storage | the local database (chapter 7) | the same, plus the SDK core itself (chapter 30); messages it does not recognise are kept as they are and shown after an upgrade (chapter 31) |

In one sentence: most of what v3 adds on the servers is the price of splitting things up: one more hop, one more routing table, one more log, one more cache. The phone has hardly got heavier; what has changed is the rules: when to connect, when to disconnect, what to do with what it doesn’t recognise.

## 4. A gateway dies

8 p.m., peak. The machine running gateway 7 suddenly loses power.

1. **50,000 devices lose their connections.** When a process exits, the OS closes its connections for it, and phones know at once; but this is a power cut, and nothing is sent. A phone only finds out when its next heartbeat gets no answer: up to 4 minutes, plus the few seconds spent waiting for the answer (chapter 23).
2. **Meanwhile, pushes to them go through OS push.** Dispatch, following the routes, hands pushes to gateway 7, about 6,950 a second, and they all fail. By chapter 20, one failed push means the device is treated as offline, and an OS push is sent instead (chapter 17). The messages themselves are all in the log and in storage; only the notifications are late.
3. **The messages they sent are still in their outboxes.** Devices on gateway 7 send about 695 messages a second. Those just sent and not yet ACKed are resent with the same message ID once the phone reconnects; for any that already reached the log, the message server spots the duplicate and returns the original seq (chapters 3, 4).
4. **Reconnects spread out on their own.** A phone that finds itself disconnected first asks the locator which gateway to connect to. The first reconnect is not spread out (chapter 24), but in a power cut each phone only finds out at its own next heartbeat, so the 50,000 are already spread over 4 minutes: 50,000 ÷ 240 seconds is about 208 a second, shared by 19 gateways, so each gets only a dozen or so extra handshakes a second. Even if they all came at once, the remaining 19 gateways would get about 2,600 each and finish the handshakes in 1.3 seconds, under chapter 24’s admission limit (10,000 queued per gateway); nobody is refused.
5. **Check the token, record the location.** The new gateway checks the signature locally, without calling the app’s login service (chapter 25), and writes “this device → me, new connection number” into the routes, overwriting the old row that pointed to gateway 7.
6. **Catch up.** Each device syncs once with its record version and gets the changed conversation records, about 500 bytes; conversations with missing seqs then fill the gaps (chapters 5, 11). About 25 MB for all 50,000.
7. **Presence lags, but does not flicker.** Nobody reported “disconnected” during the power cut, so to their contacts these 50,000 stay “online” throughout; since each reconnects within a few minutes, they never show as “offline” at all (chapter 21). Those who really can’t come back (who happen to have lost their network, say) show as “online” for a few minutes longer.

Not one message is lost, because nothing exists only on a gateway. The price is delay: during those few minutes, these 50,000 people get only OS-push notifications, and catch up on the messages when they are back.

## 5. After v3

v3 achieves one thing: **the same product grows from 10,000 online to 1 million, and deploys, scaling up and a broken machine no longer disconnect everyone.**

But “it runs” and “it keeps running” are different things. With 1 million people online, someone sees every failure, every deploy, every overload. The next six chapters add no new features; they only ask whether the system is healthy:

- did the message actually arrive, and how do we know (chapter 33);
- when more comes in than the system can take, what goes first (chapter 34);
- deploying the gateways themselves without disconnecting anyone (chapter 35);
- storage slows down, OS push fails: how to keep slowness from spreading (chapter 36);
- losing a whole data centre (chapter 37);
- drills and postmortems (chapter 38).

After that, this v3 branches out by product: workplace chat, communities of tens of thousands, live rooms, private messaging, customer service, IM as a platform (chapters 39–44). Each starts from v3’s decisions and keeps, changes or drops them; the last chapter puts them side by side in one table (chapter 45).

Next, chapter 33: a user says “my message never arrived”. It went through seven hops, and every server’s dashboard is green. Where is the message?
