Chapter 39
Workplace chat (toB)
HR posts a notice in the 3,000-person all-hands group and asks everyone to confirm by 10:00. Who has not read it? If someone asks two years later, can we still tell?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
The trunk’s app is a social app. Users sign up themselves, add their own friends, and their chat history is their own. This chapter sells the same v3 system to companies.
A company of a few thousand people uses it for work. At 9:00, HR posts in the 3,000-person all-hands group: “@all New leave policy. Please confirm by 10:00.” At 10:30 she wants to know who has not read it, so she can remind them one by one. Two years later someone says “I never got that notice”, and the company needs to see whether and when he read it.
This chapter starts from v3 (chapter 32). It walks the trunk’s decisions layer by layer (keep, change or drop), then goes deep on what workplace chat adds.
1. What workplace chat needs
| Social app (the trunk) | Workplace chat | |
|---|---|---|
| Accounts | Users sign up themselves | The company creates them and an admin manages them; when a person leaves, the account is disabled |
| Who you can talk to | Friends, people in your groups | Colleagues in the company directory; outside people only after you add them |
| Read state | “Read” in one-to-one chats; in a group, tap to see who has read it (chapter 15) | Who has not read each message, and when each person read it, with a way to nudge them |
| Group size | Up to 500 | All-hands groups of a few thousand |
| History | Belongs to the user | Belongs to the company: kept for years, searchable, exportable, deleted by policy |
The first four rows change decisions we already made. The last row is new. Workplace chat has public examples. WeCom (企业微信) has written about its message system: groups of up to 10,000, a list of who has and has not read every message, and messages stored in the cloud for up to 180 days, hundreds of TB in all (WeCom’s IM architecture, 2021, in Chinese). DingTalk (钉钉) has written about its DTIM (DingTalk DTIM, in Chinese, first published on InfoQ).
2. The trunk’s decisions, layer by layer
| Layer | The trunk’s decision | Workplace chat |
|---|---|---|
| Connection | Gateways, heartbeats, reconnects, admission (chapters 20–24) | Keep |
| Connection | The app’s server signs tokens; revocation by a timestamp (chapter 25) | Keep. The company’s single sign-on signs them. Revocation is used often: for everyone who leaves |
| Client | Web client (chapter 27), old apps (chapter 31) | Keep. Desktop and web are the main devices; an admin can require an upgrade for everyone |
| Message | ACK, dedup, seq, conversation owners, message log (chapters 3–5, 28, 29) | Keep |
| Message | Copy to another city after the ACK (chapter 37); if a whole city is lost, the last second or so of acknowledged messages is lost with it | Maybe change: the company’s records must not be lost, so copy before the ACK, 30 ms slower per send, nothing lost even if a whole city goes |
| Dispatch | Every message updates every member’s conversation recordconversation record会话记录One row per person per conversation: pinned, muted, the conversation’s latest seq, and later how far they have read. Written on join, leave or a settings change, and from chapter 11 on also when a message arrives (with the latest seq). Each change bumps that person’s record version. The messages are not here.See the glossary (chapters 10, 11) | Keep for groups of hundreds; change for groups of thousands (see below) |
| Storage | One read position per person per conversation, for unread counts (chapter 14) | Keep |
| Storage | Read receipts worked out from read positions (chapter 15) | Change: sections 4 and 5 |
| Storage | New devices pull only recent messages; search on the device is incomplete (chapter 16) | Change: history kept for years in hot and cold tiers; search on the server (sections 6 and 7) |
| Business | Joining, leaving, syncing members (chapters 12, 13) | Keep; add groups built from departments |
| Business | IM users map to the app’s users (chapter 25) | Change: users belong to the company’s directory (section 3) |
Groups of a few thousand. In chapter 11, every message in a group makes the server update one conversation record per member (a small transaction writing two rows: that person’s version number and their record). In a group of 3,000, one message every two seconds is 1,500 updates/s, 3,000 rows/s, 15% of one storage server (about 20,000 rows/s, chapter 29). That is fine. But a busy group at 10 messages/s is 30,000 updates/s, 60,000 rows/s, three storage servers. DingTalk’s public answer: in big groups, sending a message does not update each member’s red dot and last message; they are computed when the user loads the first screen. That is chapter 40’s answer for communities, and groups of thousands use it too.
The other changes follow, one section each.
3. Accounts belong to the company
In the trunk, Ana signs up herself and the IM gives her a user ID (chapter 25). In workplace chat, the account belongs to the company:
- The directory. The company has departments and members. An admin maintains them, or they are synced from the company’s own HR system. WeCom’s directory API works this way: departments and members can be added, changed and removed, and only an admin can turn on this sync (directory management, in Chinese). An IM user ID no longer means “ana_92 in the app”; it means “this member of this company”.
- Internal and external. Colleagues are internal members. Customers and suppliers are external contacts. In WeCom, employees can add contacts from other companies, and there are two kinds of external contact: WeCom users and WeChat users (customer contact, in Chinese). The two kinds of identity see different things. An external contact cannot see the company’s directory. A group with external people in it is a different kind of group, and it can have different rules for retention and archiving.
- Who may message whom. This follows the directory. People in the same company can message anyone they can find. External people must be added as contacts first. An admin can add rules on top, for example that some departments may not chat with outsiders. All of this is checked when a conversation is created. After that, sending a message checks only chapter 12’s member row; it does not ask the directory on every message.
- Leaving. When an admin marks someone as having left, the IM does three things. It revokes all their tokens with chapter 25’s timestamp and closes their open connections. It removes them from every group (chapter 12: one leave per group). Their one-to-one chats stay visible to the other person, with “left the company” next to the name. Slack does the same when it deactivates a member: the person is removed from all channels and signed out on all devices, and their messages and files are not deleted (Deactivate a member’s account). Their customers must go to someone else. WeCom calls this “inheritance on leaving” (离职继承): a departed member’s customers are assigned to another member, who takes over automatically after 24 hours (assign a departed member’s customers, in Chinese).
The history belongs to the company. When a person leaves, the history stays.
4. Who has not read it: the trunk in an all-hands group
Chapter 15’s read receipts keep nothing of their own. They are worked out from chapter 14’s read position: one seq per person per conversation (the read cursorread cursor已读游标How far a person has read in each conversation. It lives on the server, is shared by that person’s devices, and drives unread counts and read receipts. Not the same as the sync cursor.See the glossary), meaning everything up to that seq has been read. “Who has not read message s” means: of the people in the group when s was sent (chapter 12’s membership rangemembership range成员区间The stretch of seqs a person can see in a group: min_seq is the first, max_seq the last (unset while they are still in it). Joins and leaves take a seq in the conversation like messages, so who gets each message and how far back each person can read are worked out from this range, whenever they are worked out. The group has a pair too: max_seq is the latest seq, and below min_seq nobody sees anything (chapter 12).See the glossary), those whose read position is still below s.
Chapter 14 also decided: jumping to an @ does not move the read position. The messages you jumped over really were not read, so they must not count as read. In a social app that is right. In an all-hands group it goes wrong.
When the notice is posted, many people have unread messages in the group. They see a notification, or the @ next to the red dot, and tap it. That jumps straight to the notice. They read it and reply “Got it”. But they jumped over the messages above the notice (the ones they had not read before it was posted), so their read position is still at the start of that stretch. To HR, they all show as “not read”.
We run it with these assumptions (seed 1):
- when the notice is posted, 30% of members have nothing unread in the group; the others have 1 to 200 unread messages;
- 85% read it within two hours, at times spread evenly over the two hours;
- of those who read it, half jump to it from the @ or a notification, and half open the conversation and read down from their first unread message.
After two hours:
- 2,569 people have read it, and 431 have not;
- by read positions, 1,346 are shown as not having read it: 915 people who read it count as not read, 36% of those who read it. HR would go and nudge every one of them.
With pen and paper: 3,000 people × 85% who read it × one half who jump × 70% with something unread ≈ 892 people. Over 200 seeds, the average is 894 and the most is 947.
There is a second question read positions cannot answer: when. 1,301 people read it before 10:00, but a read position is only a place, with no time. Two years later, “had he read it by 10:00?” has no answer.
The trunk is not wrong. In a social group, “read by N” is only nice to know. In workplace chat, the list is used to nudge people and to settle disputes.
5. The fix: read ranges, plus a read log
Keep the read position, and add two things beside it:
- Read rangesread ranges读过的段Kept beside the read position in workplace chat: messages read by jumping ahead are recorded as a range, such as “4,900 to 4,930”, which merges into the read position once the gap is filled. They live in the conversation record row; only a few are kept.See the glossary (listed among chapter 14’s other answers). The read position says “everything from the start up to here is read”. A message read by jumping is kept as a separate range, for example “4,900 to 4,930”. When the person scrolls back and fills the gap, the two ranges merge into the read position. The ranges live in the same conversation record row. Usually there are none or a few, and only a few are kept (for example 8).
- A read logread log已读流水An append-only record: each read adds a row with who, which conversation, from which message to which, and the minute. Workplace chat uses it to answer “who has not read it, and when did the others”.See the glossary. Every read appends one row: who, which conversation, from which message to which, and the minute. Rows are only appended, never changed. The phone reports its read position at most once a second (chapter 14), and reports in the same minute that join up are merged into one row. DingTalk does something similar: the client merges several reads in one conversation into one request, and the server merges the read requests for each message, for example a minute of requests into one.
Then:
- Who has not read message s: of the people in the group when s was sent, those whose read position is below s and whose read ranges do not include s. For 3,000 people, a position each (12 bytes) plus a few ranges, a tap reads a little over 36 KB.
- When he read it: the first row in his read log for this conversation that includes s. To the minute.
- Unread counts still come from chapter 14’s read position. The countable messages inside a read range are subtracted from the unread count; chapter 14’s “count so far”, stored on every message (how many countable messages the conversation has up to that one), gives that number with one subtraction. Say Ana has read to 4,800, the latest is 5,000, and she has a read range from 4,900 to 4,930: the count so far at 4,930 minus the count at 4,899 is 31, so her unread count goes from 200 to 169 (assuming every message after 4,800 is an ordinary one that counts).
In the all-hands group above: of the people who jumped, 915 had an unread stretch before the notice, so 915 ranges are written. The log has 2,569 rows, one per reader. Exactly those 431 people are shown as not having read it. Over 200 seeds, none is wrong. The 1,301 who read it before 10:00 can be found in the log.
How the sender sees the numbers go up. While HR has the group open, each of her messages shows “N unread” under it. There is no need to push every single read. A background job reads the read log (a consumer, like those behind chapter 29’s message log), gathers a minute’s worth and tells her “these messages’ numbers changed”, and her phone fetches them.
6. Kept for years: hot and cold
At v3’s volume: 400 million messages a day, 200 bytes each.
- 80 GB a day, 29.2 TB a year, 87.6 TB with three replicas;
- 5 years is 146 TB, 438 TB with three replicas.
But most reads touch only recent messages. So history is split in two tiers:
- hot: the last 30 days, 2.4 TB, 7.2 TB with three replicas, on fast disks as in the trunk;
- cold: the older 143.6 TB, on cheap disks, with erasure codingerasure coding纠删码Data is split into blocks and parity blocks are computed from them; any blocks, up to the number of parity blocks, can be lost and rebuilt. With 6 data and 3 parity blocks the overhead is 50%, against 200% for three replicas. It suits cold data.See the glossary instead of three replicas. For example, every 6 blocks of data get 3 parity blocks: any 3 can be lost and the data can still be rebuilt, for only 50% extra. 143.6 TB × 1.5 ≈ 215 TB.
About 223 TB in all, 51% of keeping everything as three replicas. The cost is that cold reads are slow: scrolling back two years means a wait. WeCom does the same and keeps cold data on cheap SATA disks.
Text is the small part. With chapter 8’s assumption that 5% of messages are 200 KB photos, that is 4 TB a day, 50× the text, 1.46 PB a year, and work files are bigger still. They live in object storageobject storage对象存储A storage service that keeps whole files by name (S3, for example); images and files go here, not into the message table. The phone gets a short-lived upload address and uploads to it directly; downloads usually go through a CDN.See the glossary, need hot and cold tiers too, and fall under the same retention rules below.
The read log is kept just as long: about 600 million rows a day (10 million people opening about 60 conversations a day each, chapter 7’s assumption), 28 bytes a row, 16.8 GB a day, about a fifth of the text.
7. Search
Chapter 16 said a new device pulls only recent messages, so search on the device is incomplete. With years of history, search moves to the server.
- Where it comes from: the search index is one more consumer behind the message logmessage log消息日志A durable, append-only queue: the hand-off between receiving and delivering. A message counts as received once it is in the log, and the server ACKs then. It usually runs on a message queue, such as a topic partitioned by conversation.See the glossary (chapter 29), and builds the index as the log grows. Recalls, edits (chapter 18) and deletions by policy reach it the same way.
- How big: assume the index is about the size of the text, another 80 GB a day.
- Who can find what: only messages in your own conversations, and only from the time you were in the group (chapter 12’s membership range). Do not build an index per person: a message reaches 6.67 people on average (chapter 6’s assumption), so one copy each is 533 GB a day. Build the index per conversation and filter at query time by “which conversations am I in, and from which message”.
- How to shard it: by company looks natural, but a big company becomes a hot spot. Slack first split its databases by workspace; a large customer could outgrow one shard, so it moved messages to sharding by channel (Scaling Datastores at Slack with Vitess, 2020). The trunk already shards by conversation (chapter 28), so we keep that.
8. Retention and export
The history belongs to the company, so the company decides how long it is kept and who may see it.
- Retention. An admin chooses “keep everything” or “delete after N days”. On Slack’s paid plans, data is kept for the lifetime of the workspace by default; it can be changed to delete automatically, and deletions run once a day (Customize data retention). Deleting must reach every copy: hot, cold, replicas, the search index, the read log, files, and every device’s local database (like a recall in chapter 18, an event goes down to the devices).
- Deleting is writing too. If every company kept only one year, 400 million messages would be deleted every day, 4,630 a second on average, as many as are written. Squeezed into four hours at night, that is about 28,000 rows/s, one and a half storage servers, before the index and the files. So cold data is best laid out by company and month: when it expires, the whole block is dropped instead of deleting row by row.
- Export. For compliance, a company wants to export every conversation. On Slack’s Business+ plan, exporting all channels and direct messages needs an application (Export your workspace data). WeCom has “conversation archiving” (会话内容存档): the company chooses which employees are archived and fetches their work conversations through an API (conversation archiving, in Chinese). Export follows the permissions from section 7: what the exporter can see is recorded, and can be audited.
9. Cost
- One more log row per read. About 16.8 GB a day, about 20,000 rows/s at peak, one storage server. One row per message per reader (16 bytes a row) would be 42.7 GB a day and about 93,000 rows/s at peak, 4.6 storage servers.
- Read ranges make the unread count harder. Messages inside ranges must be subtracted. When there are too many ranges, only the latest few are kept, and older jumps are only in the log.
- Times only to the minute. Merging each minute into one row is what keeps the rows down.
- Deleting must reach every copy (section 8). Miss one, and it is not really deleted.
- A departed person’s phone. The server can tell it to wipe its local database only when it connects. A phone that never goes online again keeps its copy.
10. Other answers
- One row per message per reader. The most direct: it knows when, and it knows jumps. The cost is rows: a group of 500 with 1,000 messages a day writes 500,000 rows a day; if each member opens it 20 times a day, the log needs only 10,000.
- WeCom. One copy of each message per member (write fan-outwrite fan-out写扩散When a message is sent, write one entry into each recipient’s inbox; reading means reading your own inbox. More writes, fast reads; suits conversations with few members. Chapter 6 weighs it; chapter 10 shows it goes with client-built chat lists, where the phone builds its chats from the inbox. Neither v1 nor v2 uses it.See the glossary), which the article says serves toB needs such as receipts and “delete only on my side”. The read list of a receipt is stored in the sender’s copy: when someone reads it, a new message pointing at the original is appended. Phones confirm several messages as read at once, and the server processes them together: one confirmation per reader, and, after a queue, several per sender. At the online peak this saved 44% of writes in all. Requests for the same account go through a queue one at a time, so two people’s reads do not overwrite each other.
- DingTalk DTIM. Ordinary messages are stored once; what differs per person (messages for one person only, messages one person deleted) goes into that person’s inbox. Read state is written when a message is stored. Reads in big groups are merged.
- A bitmap per message. One bit per group member, 375 bytes per message in a group of 3,000. It says who read it, not when.
- Small groups only, kept for a few days. As chapter 15 noted, in Telegram’s public API, “who read this group message, and when” (
messages.getMessageReadParticipants) is only available in groups up to a size limit (100 members by default), and the receipts are kept only for a while (7 days by default), then deleted (method, client configuration). A social product can save that way; an office IM’s all-staff group has thousands of members and must be checked two years later, the opposite. - Like the trunk, read positions only. Accept that jumps count as unread and that there are no times. Enough for a social product.
11. This chapter’s decision
Workplace chat makes read state and history heavier. The next chapter goes the other way: a community of 100,000 people, most of whom rarely visit. How can each message cost less?
Comments
Comments live in GitHub Discussions and need a GitHub account.