IM Systems in Depth

Chapter 45

Same decisions, different products

A social app, workplace chat, a community, a live room, a private messenger, customer service and an IM platform all start from the same v3. Which decisions stay, and in which products do they flip? And why do they always flip several at a time?

This chapter is a first draft. The author is still checking it, and its text and numbers may change.

The trunk builds a social app: one-to-one chats between friends, groups of at most 500, records that belong to the user (chapters 1–38). Chapters 39–44 took the same v3 (chapter 32) and built six other products from it, each chapter walking through the trunk’s decisions layer by layer: keep, change, or drop.

This chapter makes no new decision. It puts the six chapters’ answers side by side.

1. How to read the table

The table is wide, so it comes in four parts. Each row is one of the trunk’s decisions, with the chapter that made it in brackets. The first column, “Trunk · social”, is what the trunk does; each column after it is a branch product, headed by the chapter about it. Each cell gives the choice first and the reason in brackets. The branch cells use these marks:

  • =: the chapter keeps it, or leaves it alone (every branch starts from v3, so what it does not mention stays as it was). Words after “=” say how the same design applies in that product.
  • Change, drop, add: the chapter changes it, does without it, or adds something next to it. May change: the chapter considers changing it but has not decided.
  • —: the chapter does not discuss it, and “unchanged” would be a guess.

2. How a message travels

Decision Trunk · social Workplace 39 Community 40 Live 41 Private 42 Support 43 Platform 44
Who gets the push (2, 16) Every online device of every member, the sender’s other devices included; the push is a hint, gaps are pulled by seq = Change: push only to devices looking at the community; the rest get at most one frame a minute (most people are not looking) Change: one bundlebatching打包In a live room, a second’s messages go out as one batch, so each connection gets one push a second, compressed. Pushes no longer grow with the room’s message rate; the cost is half a second’s wait per message on average.See the glossary a second, at most 20 messages a viewer (all a screen can show) Change: the sender encrypts one copy per device; in groups a sender key, one upload (every device has its own keys) = Change: a broadcast to everyone only pushes “go sync” to those online, spread over a minute (ten million people)
Acks and retries (3) ACK once stored; resend if none comes = = — (pushes to viewers may be lost; the ack for a viewer’s own send is not discussed) = = =
Dedup (4) The phone makes the message ID; “sender + ID” is unique = = = Change: the receiving phone dedups (the ID is inside the ciphertext) = Add: the business’s own key, kept 24 hours (its retry is a new call)
Order (5) One seq per conversation; a gap is filled = =, one seq per channel Drop, kept only for gifts (late is useless) Change: one mailbox per device, in arrival order (the server keeps no per-conversation timeline) =, handovers and endings take a seq too =
Send limits (34) 5 a second per device; at most 5,000 recipients a second per conversation; over that, say “retry” = = Change: 20 a second per viewer, the rest dropped quietly by priority = = Change: each tenanttenant租户One business plugged into a shared IM service, such as a shopping app or a game. The service is shared; each tenant’s IDs, rules and quotas are kept apart, so one cannot drag down the rest.See the glossary has its own quota for API calls, broadcasts, callbacks and queues (no tenant drags down the rest)
Copy to another city (37) Copied after the ack; a cross-city failover may lose acked messages May change: copy before the ack, about 30 ms more a message (none may be lost) = = = = =

3. What is stored, and written for whom

Decision Trunk · social Workplace 39 Community 40 Live 41 Private 42 Support 43 Platform 44
Copies of a message (6, 10) One per conversation = =, one per channel Change: a room mailbox keeps only the latest, a screenful for newcomers Change: one ciphertext per device, deleted once fetched, kept at most 30 days = =, a broadcast is stored once too
Conversation recordsconversation record会话记录One row per person per conversation: pinned, muted, the conversation’s latest seq, and later how far they have read. Written on join, leave or a settings change, and from chapter 11 on also when a message arrives (with the latest seq). Each change bumps that person’s record version. The messages are not here.See the glossary (11) Every message updates a row for every member Change: groups of thousands are not updated; worked out when the first screen loads (10 a second would be 3 storage nodes) Drop: they only record which communities you joined (otherwise 100,000 rows a message) Drop (a live room is not in the conversation list) Drop: the phone builds its list (the server cannot read) = (an agent’s workbench is their conversation list) Change: a broadcast writes none; sync asks for one more latest seq
History and new devices (6, 16) The server keeps it all, the user’s; a new device gets the latest first, older on scroll Change: the company’s, kept for years, hot and cold tiers; server-side search; retention and export set by admins =, a channel’s whole history Drop: what passed while disconnected is gone Change: the server keeps none; an old device packs it up for the new one (old ciphertext is unreadable to a new device) Change: the customer’s; one conversation, used for good =
Joining and leaving (12) Takes a seq; history by member rangemembership range成员区间The stretch of seqs a person can see in a group: min_seq is the first, max_seq the last (unset while they are still in it). Joins and leaves take a seq in the conversation like messages, so who gets each message and how far back each person can read are worked out from this range, whenever they are worked out. The group has a pair too: max_seq is the latest seq, and below min_seq nobody sees anything (chapter 12).See the glossary =; leaving the company is one leave per group Change: only the member table changes; what you see comes from roles Drop: joining and leaving only touch a group of connections on the gateway (thousands a second) Change: when someone leaves, everyone replaces their sender key (W4) Change: an agent enters and leaves by joining and leaving, min_seq 1, so they see all history; a handover is one leave and one join =; a conversation holds only one tenant’s users
Member list (13) The phone keeps the whole list, with a version = Change: fetched a screen or a search at a time (the whole list is 4 MB) Drop: the count is an estimate = = =

4. Reading, notifications and recall

Decision Trunk · social Workplace 39 Community 40 Live 41 Private 42 Support 43 Platform 44
Unread counts (14) The server counts: read positionread cursor已读游标How far a person has read in each conversation. It lives on the server, is shared by that person’s devices, and drives unread counts and read receipts. Not the same as the sync cursor.See the glossary plus a count carried by each message; exact Change: keep the read position, then subtract the ranges readread ranges读过的段Kept beside the read position in workplace chat: messages read by jumping ahead are recorded as a range, such as “4,900 to 4,930”, which merges into the read position once the gap is filled. They live in the conversation record row; only a few are kept.See the glossary Change: created on first read; only bold and an @ count Drop Change: the phone counts; the read position goes encrypted to your other devices = Change: a broadcast’s read position is written only once read; the badge is worked out separately
Read receipts (15) From read positions; in groups counted when you open that message Change: add ranges read and a read logread log已读流水An append-only record: each read adds a row with who, which conversation, from which message to which, and the minute. Workplace chat uses it to answer “who has not read it, and when did the others”.See the glossary, so who has not read it and when others did can be looked up (people who jumped ahead must not count as unread) Drop Drop — = =
Offline notifications (17) At most one per conversation a minute, with content; the badge is the unread total = Change: by default only when someone @s me Drop Change: no content, it only wakes the app; the phone works out the badge = Change: a broadcast’s system pushes still go one by one, at low priority
Recall and edit (18) Takes a seq; every device applies it =; the search index follows = (moderators delete messages all the time) — Change: the receiving phone checks the sender and the time limit = =

5. Connections and people

Decision Trunk · social Workplace 39 Community 40 Live 41 Private 42 Support 43 Platform 44
Fan-out (20) The gatewaygateway网关A connection-layer server. Each holds many long-lived connections and handles auth, heartbeats and traffic without reading messages. One of the few stateful parts: a restart drops all its connections.See the glossary knows no business; online routesonline routes在线路由表Which gateway each online device is connected to. Gateways register devices as they connect, and dispatch looks gateways up here.See the glossary find each member’s devices = Change: push from the community’s subscriber list, no route lookup per member Change: the gateway keeps each room’s group of connections; the room routerroom router房间路由The routing for live rooms: for each room it knows only which gateways have viewers, not who they are. A room’s message goes to it, it sends one frame to each of those gateways, and each pushes to the viewers in its own group.See the glossary only lists which gateways have viewers = = Change: the gateway registers connections by tenant, with a cap per tenant
Presence (21) Told only to those looking = =, subscribed only for the people on screen — = = =
Accounts (25) The app’s server signs a token; the IM has its own user IDs Change: accounts belong to the company directory; leaving revokes them = =; guests can only watch this room Change: each device has its own keys; the device list must be right Change: a visitor ID that lasts; handed to the account at login Change: (tenant, user) translated at the edge; one key pair per tenant
Who may message whom (12) In groups, members may send; the rules for one-to-one chats are left to the branches Change: by the directory; outsiders must be added first; checked when a conversation is created Change: computed from roles plus channel overwrites — — Change: it is the assignment itself: only the agent let into the conversation Change: each tenant picks: relationships hosted by the platform, written as conversation state, or a callback per message

6. First, what did not change

Count the branch products’ cells in the four tables (this chapter’s demo.py does it): six branches, 19 rows, 114 cells.

The message layer barely moved. In section 2’s table, acks and retries: 5 of the 6 products keep them; copying to another city, also 5; dedup and a seq per conversation, 4 each. Two more sit outside the table: one owner per conversation (chapter 28), which chapters 39, 40 and 41 say they keep, and the message log before storage (chapter 29), which chapters 39 and 40 say they keep. Only live rooms drop seq, and only the private messenger turns storage into one mailbox per device.

What moved, moved in three places.

  • Distribution: who gets the push, conversation records, fan-out. Only customer service keeps conversation records.
  • State kept per person: unread counts, receipts, history. Only customer service keeps the unread count as it was.
  • The business layer: accounts, who may message whom. No product copies the trunk’s “who may message whom”.

By product, out of the 19 decisions:

Product Kept (=) Changed or added Dropped Not discussed (—)
Support 43 15 4 0 0
Workplace 39 12 7 0 0
Community 40 10 7 2 0
Platform 44 10 9 0 0
Private 42 6 10 1 2
Live 41 3 4 8 4

Customer service changes least: 4 decisions (history, joining and leaving, accounts, who may message whom), all because the conversation belongs to the customer and an agent answers it; nothing about how a message travels changes. It also puts little load on the IM: 2.7 messages a second, 0.019% of v3’s peak of 13,900. Live rooms drop the most, 8. The private messenger changes the most, 10.

7. What flips together, and why

The changes and drops in the table do not come one at a time; they come in groups. Five things push them.

How many people one message goes to. Several of the trunk’s decisions do the same kind of work: once per message per member. Updating each member’s conversation record, counting exact unread, keeping the whole member list on the phone, sending joins and leaves to everyone, pushing to every member’s devices. In a 500-person hiking group each one is affordable. With more people, doing it the trunk’s way looks like this. “Storage nodes” assume chapter 29’s 20,000 rows a second per node; for the live room, “each step” means each once-per-person job, such as pushing or updating conversation records; the platform’s “rate-limited” is chapter 34’s limit of 5,000 recipients a second per conversation.

Product People one message goes to What the trunk’s way costs
The trunk’s hiking group 500 500 records, no problem
Workplace all-hands group (39) 3,000 At 10 messages a second, 30,000 updates (60,000 rows) a second, 3 storage nodes
Community (40) 100,000 At 10 a second, 1,000,000 updates (2,000,000 rows) a second, 10.8 times all of v3’s conversation-record writes
Live room (41) 1,000,000 At 100 a second, 100 million a second for each step, 719 times v3’s peak deliveries
Platform broadcast to everyone (44) 10,000,000 Rate-limited, the last person waits 33 minutes; 20 million rows of the same sentence

So this group always flips together, and always the same way: store the message once, push to whoever is looking, let whoever opens it read it (chapter 10’s read fan-outread fan-out读扩散Keep one copy in the conversation’s timeline, and each reader fetches it from there. Few writes, more reads; suits big groups. All of v1’s and v2’s messages are kept this way (chapters 6, 10); it needs the server to build the chat list and devices to sync it.See the glossary). The community does it first; workplace chat borrows it for groups of thousands; the platform’s broadcast does the same. Live rooms go one step further and let the gateway itself remember who is watching.

Whether some loss is acceptable. The trunk’s seq, gap-filling, history and unread all exist so that nothing goes missing. In a live room, late is useless, and the whole set goes; when the room is busy, the screen cannot keep up, and a viewer sees about one in six of what others say. Yet in the same room, a paid gift’s copy for the streamer goes back to the trunk’s path: stored, with a seq, gaps pulled. Workplace chat goes the other way: nothing should be lost, so chapter 39 considers copying across cities before the ack, about 30 ms more per message.

Who owns the data. In the trunk the records are the user’s. Change the owner, and accounts, who may message whom, how long history stays and how much a newcomer sees change together:

  • Owned by the company (39): accounts live in the company directory; leaving revokes them and leaves every group; history is kept for years, with retention and export set by admins.
  • Owned by the customer (43): the conversation follows the customer and is used for good; an agent who joins sees all of it; after a visitor logs in, the conversation passes to her account.
  • Owned by the tenant (44): every ID carries the tenant; the tenant chooses who may message whom.
  • Only on the user’s devices (42): the server deletes once delivered; history moves between devices.

Who pays. What the payer wants gets added. A company wants “who has not read it, and when did the others” and years of records, so there is a read log and hot and cold tiers (39). A shop wants enough agents and short waits, so there is an assignment service and timestamps for every conversation (43). A tenant wants its own rules and no noisy neighbours, so every tenant gets its own quotas and a cap on connections (44). In a live room, the viewer who sends a gift wants the streamer to see it, so gifts get a reliable path (41). These are mostly added alongside; the trunk’s core does not flip.

Whether the server is trusted. The private messenger’s column changes most, and not because of size (its example group has 100 people). Every cell that needed the server to read content flips at once: how many copies are pushed and stored, dedup, unread, notifications, recall, history. The server keeps only one mailbox per device; the rest moves to the phone. It stores less, since a copy is deleted once delivered: about 1/70 of the trunk after a year. It writes more, since every device gets its own copy, which is then deleted again: about 1.4 times.

8. One product, five questions

The seven products (the trunk plus six branches) on these five things:

Product People per message Can some be lost? Who owns the data Who pays, who sets rules Is the server trusted?
Trunk · social Groups up to 500 No The user The app itself Yes
Workplace 39 All-hands group, 3,000 Even less The company The company, through admins Yes (it needs server search)
Community 40 100,000 members, about 1,000 looking Late by a minute for those not looking The community; anyone allowed can scroll back Whoever runs the community, through roles Yes
Live 41 1,000,000 Yes, except gifts Not stored Viewers who send gifts Yes
Private 42 Group of 100 × 3 devices No Only on the user’s devices The user No
Support 43 The customer plus one agent No The customer The shop Yes
Platform 44 Broadcast to 10 million No; callback rules are only “best effort” The tenant The tenant Yes

Faced with a new product, answer these five questions first, then go back to the four tables: where an answer differs from the trunk’s, the rows it drives are the ones to change. Where the answers are the same, do what the trunk does.

9. This chapter’s decision

That is the end of the branches. What the six products changed most was never “how does a message arrive reliably” but “to whom, recorded for whom, owned by whom”.

Comments

Comments live in GitHub Discussions and need a GitHub account.