# Chapter 19: v2: groups at 100,000 users

> A message goes into a group of 500. Ana sends it from her laptop, Ben’s phone is closed, several devices must agree, and a wrong message can still be recalled. What did v2 add on top of v1, and what did each piece cost?

IM Systems in Depth · https://im.liko.page/en/v2-groups/

At the end of v1, a message between two people was no longer lost, repeated or reordered. From chapter 10 on, the problem became **more people**: Ana logs in on a new laptop, so where does her conversation list come from (chapter 10)? She has 2,000 conversations, and a sync cannot fetch them all every time (chapter 11). People join and leave groups (chapter 12), and the phone needs the member list (chapter 13). The red badge has to be exact, and the same on every device (chapter 14). The sender wants to know who has read it (chapter 15). What you send must also appear on your other devices (chapter 16). Even with the app closed, you should know a message came (chapter 17). A wrong message can be recalled (chapter 18).

Like chapter 9, this chapter makes no new decision. It does four things: it follows one group message through v2; it puts the nine decisions and their costs in one table; it redoes the server and phone bills from v1 to v2; and it says what v2 cannot hold.

The scale has not changed: still 100,000 daily users, 10,000 online at peak, 139 messages a second. v2 did not add a single user. What it added is groups, devices and state.

## 1. One group message, end to end

Friday evening. On her laptop, Ana sends the hiking group “Saturday’s meeting time is now 8.” Ben’s phone is locked, with the app closed.

**Server: the message layer**

1. **The send check happens inside the lock.** As in v1, taking the seq locks the conversation’s row; at the same time the server checks Ana’s membership range: only if she is still in the group may she send (chapter 12). This message gets seq 4,930.
2. **One transaction writes it all.** The message is stored once, with the count so far and Ana’s running number (chapter 14); one more row is written, the **to-do**, a note for the background job: “hiking group 4,930, update every member’s conversation record” (chapter 11). Commit, then the ACK.

**Server: the dispatch layer**

3. **Push to every online device of everyone whose membership range includes 4,930.** That range is the stretch of seqs a person has in the group (chapter 12). Ana’s own phone gets the push too; only the connection it came from is skipped (chapter 16).
4. **Update 500 conversation records from the to-do.** Each member’s row gets the latest seq and that person’s new record version, one small transaction per person; the sender does not wait (chapter 11).
5. **Devices not in the foreground get an OS push.** Once the to-do has updated Ben’s row, it sees that none of his devices is in the foreground, so it sends a notification through APNs or FCM, with his total unread count as the badge. At most one per conversation per minute, and only one stays on screen (chapter 17).

**Ben’s devices**

6. **Tap the notification, sync once.** The phone asks with its last record version and gets back only the dozen or so rows above it, 39 bytes each, with the unread counts the server has worked out (chapters 11, 14). For each conversation it is behind on, it pulls the newest message (chapter 6).
7. **Open the group; no need to fetch the member list again.** The phone sends the group’s member version; if nothing changed, the answer is 4 bytes (chapter 13).
8. **Read it on the laptop.** The read position only moves up, and when it moves the record version moves too; at the phone’s next sync, the red badge clears (chapter 14).

**Back to Ana**

9. **Who has read it.** Ana taps her message; only then does the server count, from the members’ read positions, “k people have read it” and who they are (chapter 15).
10. **Sent by mistake: recall.** The recall takes a new seq and follows the same path from step 1; the notification on Ben’s lock screen is replaced using the same identifier (chapter 18).

In these ten steps, not one of the v1 rules summed up in chapter 9 changed: a unique key catches repeats, so a message that arrives several times lands on one row (chapters 4, 7); a number is committed in the same transaction as what it stands for, so the order of numbers is the order of commits (chapters 5–7); and a push is only a reminder, and a lost one is made up by pulling by number (chapter 6). v2 just has a few more numbers.

## 2. Switch one off, and what comes back

Each chapter’s simulator or estimate is still in that chapter. Below is what that chapter worked out when one decision is switched off:

| Switched off | What comes back (in that chapter’s simulator or estimate) |
|---|---|
| The server builds conversations | [Chapter 10](/en/fan-out/): the phone pieces its conversation list together from an inbox kept for only 7 days, and of 2,000 conversations finds only **380** |
| Record version | [Chapter 11](/en/conversation-sync/): Ana downloads every conversation’s latest seq each time, **480 KB** a day; by version, 6.7 KB |
| Membership ranges | [Chapter 12](/en/group-membership/): a member table that only says “in or out” lets Cara see **4,836** messages from before she joined, and Ben 64 from while he was out |
| Member version | [Chapter 13](/en/group-sync/): the whole member list on every open, **524 KB** a day; by version, about 0.5 KB |
| Read position | [Chapter 14](/en/unread/): with a counter instead, **46** of the hour’s 62 comparable minutes disagree |
| Receipts from read positions | [Chapter 15](/en/read-receipts/): one receipt per message is **500,000** a day for the hiking group; pushing every read to the whole group is about 5 million pushes a day |
| Pushing to your own other devices | [Chapter 16](/en/many-devices/): pushing only to recipients, a message Ana sends from her phone appears on her open laptop only with that conversation’s next message; the first time, after **20 minutes** |
| Notifications merged per conversation | [Chapter 17](/en/offline-push/): one notification per message is **1,000** a day per person for the hiking group, and FCM drops them all once more than 100 pile up for a phone that is switched off; merged, 406 |
| Recall takes a seq | [Chapter 18](/en/recall-and-edit/): changing it on the server and telling only online devices, a recall after one minute leaves the original text on **24** devices on average for good, and 553 after an hour |

## 3. Three numbers

Almost everything in v2 rests on three numbers that only go up:

| Number | Kept per | Covers |
|---|---|---|
| seq | conversation | messages, joins and leaves (chapter 12), recalls and edits (chapter 18) |
| Record version | person | the conversation list, group info, my groups (chapters 11, 13), read positions (chapter 14), the other side’s read position in a one-to-one chat (chapter 15) |
| Member version | group | the member list (chapter 13) |

Each number is taken in the transaction that writes what it stands for. The phone keeps only three things: each conversation’s seq, its own record version, and each group’s member version. On return it asks, for each, “give me what is above this.”

Put the other way round: **a change that takes no seq and bumps no version is invisible to sync.** Chapter 18’s first idea, changing a message in place on the server, is missed by sync for exactly this reason: Ben’s phone already has the message and its cursor is past it; the change took no seq, so no sync of his will ever pull it.

## 4. The whole v2 map

Compared with v1, the boxes are still on one server, with two additions: the **dispatch layer** is now a block of its own, updating every member’s conversation record from the to-do and sending OS pushes to devices not in the foreground; and the **business layer** sits beside the message path: who is in the group, who may send, and how long a message can still be recalled.

## 5. v2’s nine decisions

| Chapter | Decision | Cost | Revisit when |
|---|---|---|---|
| [10 Who builds the conversation list](/en/fan-out/) | Conversations live on the server, one conversation record per person per conversation; devices copy it, and the local list is only a copy | A new device’s first login fetches every conversation (Ana’s 2,000, about 472 KB) | Each sync asks about every conversation (chapter 11) |
| [11 Syncing the conversation list](/en/conversation-sync/) | For each message, the to-do writes the latest seq and a new record version on every member’s row; the phone fetches only the rows above its version | Each message is written again for each member (about 927 a second at peak); the record lags the message a little; each person’s row that hands out versions is a hot spot | The to-do replaced by the message log (chapter 29); groups with tens of thousands of members (chapter 40) |
| [12 Joining and leaving a group](/en/group-membership/) | Joining and leaving take a seq in the conversation; one row with min_seq and max_seq per person per time they join; who gets a message and how far back they can scroll follow the range | Joining and leaving become a message to the whole group; the member check moves inside the lock; a leaver’s row cannot be deleted | Who may message whom in one-to-one chats (branches); huge groups (chapter 40) |
| [13 Syncing group info and members](/en/group-sync/) | One member version per group, bumped by each member change and stamped on that member’s row; the group’s name, avatar and notice go on conversation records | A column and an index; the phone stores the lists; a very old version means fetching the whole list | Loading members on demand in huge groups (chapter 40) |
| [14 Read position and unread counts](/en/unread/) | The server counts: one read position per person per conversation, only moving up; each message carries the count so far and the sender’s running number | Reading is writing (up to about 927 a second at peak); 8 more bytes per message; two more lookups per read | Per-message read state (chapter 39) |
| [15 Read receipts](/en/read-receipts/) | Receipts come from read positions, with nothing kept apart: in one-to-one chats, the to-do writes the other side’s read position onto the sender’s conversation record; in groups nothing is pushed, and tapping a message counts how many have read it, and who, from an index on (conversation, read position) | One more write per read in one-to-one chats; in groups, one more index that changes with every read; no read times, and no record of messages skipped over | Workplace chat keeping who has not read each message (chapter 39); customer service (chapter 43) |
| [16 Many devices](/en/many-devices/) | Cursors stay on each device; push to every online device of every member, including the sender’s own other devices, skipping only the connection the message came from; a new device fetches every conversation record and each conversation’s newest message, plus one more page for the conversations on its first screen | About 5% more pushes per message; a device table; a new device starts shallow, and local search misses things | History storage and search (chapter 39); multiple devices under end-to-end encryption (chapter 42) |
| [17 Waking an offline phone](/en/offline-push/) | Devices report foreground and background; when none of a person’s devices is in the foreground, a notification goes out through APNs, FCM or a phone maker’s channel; merged per conversation, at most once a minute; the badge is the total unread count on the users row, changed in the same transaction as the conversation record | Several channels to integrate; no promise of delivery or speed; the server also keeps tokens, foreground state and the time of the last notification | How to tell foreground from background (chapter 26); what a notification may carry under end-to-end encryption (chapter 42) |
| [18 Recall and edit](/en/recall-and-edit/) | A recall or an edit takes a seq in the conversation, in the same transaction that marks the original; every device applies it along the path it receives messages on | History only grows; every client must know the new types; one more lookup when counting unread | Old app versions (chapter 31); end-to-end encryption (chapter 42) |

Three of the nine costs are the same trade: **more writes for fewer reads**. Chapter 11 updates a row per member for every message, chapter 14 makes reading a write, and chapter 15 writes the sender’s record once more for each read in a one-to-one chat. In chapter 6, every sync swept 200 conversations; v2 updates a few more rows at write time instead, and sync fetches only what changed.

## 6. v2’s bill

Both columns use the same assumptions: v1 is chapter 9’s column; every number in the v2 column comes from the chapter in brackets.

**The server (10,000 online, at peak)**

| Resource | v1 (chapter 9) | v2 |
|---|---|---|
| CPU | 69 syncs/s, each reading 200 conversations’ latest seqs from the cache, about 13,900 reads in all | Sync reads only the records that changed: about 13 rows each, about 900 rows/s (chapter 11), with one more lookup per row for a recall (chapter 18); new: the to-do runner, about 927 small transactions/s updating conversation records (chapter 11); read reports, also at most 927/s, each with two lookups (chapter 14); member version checks on opening a group, at most about 208/s, mostly answered “no change” (chapter 13) |
| Memory | About 300 MB for 10,000 connections | With a desktop app, one person online may have 1 to 1.5 devices connected (an assumption): 300–450 MB |
| Network bandwidth | Pushes about 2.2 Mbit/s; sync replies 200 × 12 bytes, about 1.3 Mbit/s | About 5% more pushes, to the sender’s own other devices (chapter 16), about 2.3 Mbit/s; sync replies about 13 × 39 = 507 bytes each, about 0.3 Mbit/s; plus OS pushes sent to APNs and FCM, about 1,251/s before merging (chapter 17) |
| NIC packets | About 3,100/s with no loss | About 5% more pushes; sync replies are smaller, and most fit in one packet |
| Disk capacity | Conversation records, 20 million rows × 24 bytes = 480 MB | Each row gains the latest seq and chapter 14’s read position and counts, 51 bytes, about 1.02 GB (leaving out chapter 15’s 4 extra bytes on one-to-one records and chapter 17’s one unread total per person, both small); a member table of 5 million rows, about 105 MB; 8 more bytes per message, 32 MB a day |
| Disk IO | About 139 commits/s, each writing two rows, the message and the conversation | Messages unchanged; conversation records about 1,853 rows/s (chapter 11), read reports another 1,853 (chapter 14), about 3,707 rows in all, 18.5% of what one storage node can write; with the ceiling for one-to-one receipts, about 27.8% (chapter 15) |

In one sentence: v2 turns the server’s reads into writes. Sync drops from about 13,900 cache reads a second to about 900 rows; in exchange come a few thousand small transactions a second, about a fifth of one storage node.

One more cost does not show in the table: **busy groups**. If the hiking group sends 10 messages a second, the conversation records alone take 5,000 changes a second, each changing two rows (the users row for the version and the record), 10,000 rows: half of one storage node (chapter 11). The average adds up, but a single group can eat half of it.

**Ben’s phone (one day)**

| Resource | v1 (chapter 9) | v2 |
|---|---|---|
| Requests | 20 syncs, usually two requests each; 40 messages sent | Syncs unchanged; a report each time he reads somewhere new, assumed once per conversation opened, at most 60; one member version check per group opened; a report on each switch to foreground or background, 40 (chapter 17) |
| Traffic | Each sync downloads 200 latest seqs, 48 KB a day | Each sync fetches only the records that changed, about 10 KB a day; member lists by version, about 0.5 KB a day (chapter 13) |
| Battery | The radio wakes for each push | OS pushes wake it too; merged, a busy group wakes it at most 60 times an hour, and the hiking group 406 times a day (chapter 17) |
| Background | Nothing can wake the phone | An OS push can, with a badge showing the unread count the server worked out (chapters 14, 17) |
| Storage | About 34 MB of text a year | Plus the member lists of groups he has opened (20 KB for the hiking group); conversation records 200 × 39 bytes, about 7.8 KB |

Ana’s laptop is her second device: it syncs and stores a copy of its own. Its first login fetches all 2,000 conversations and each one’s newest message, about 478 KB (chapter 10 had 472 KB; chapter 14 added the read fields to each record); one more page for each of the 20 conversations on the first screen adds 200 KB (chapter 16).

## 7. What v2 cannot hold

v2 is still one server. At 100,000 daily users it is enough. At v3, with 10 million daily users and 1 million online at once, two things break first:

- **One server cannot hold every connection.** Each connection is about 30 KB, so 1 million are 30 GB, more than the 16 GB one machine can give to connections. With the machine assumed in chapter 22, 50,000 connections each normally, that is 20 machines.
- **Every deploy disconnects everyone.** Connections and logic live in the same program, so changing one line of logic means a restart. Today that is 10,000 devices reconnecting; at 2,000 handshakes a second that takes 5 seconds, and since each device syncs once it is back, those 5 seconds see 2,000 syncs a second, 29 times the peak (69 a second).

At v3, 1 million devices on one server would take 500 seconds just to handshake. At two deploys a week, that is 2 million reconnects a week (chapter 20).

Storage will run out too: at v3’s 13,900 messages a second, the conversation records alone need about 92,700 changes a second, 4.6 times one storage node (chapters 10, 11). That comes later in v3 (chapters 28, 29).

Next, chapter 20: take the connections out of the logic, so the gateway only holds connections and the logic can be deployed on its own.
