# Chapter 39: Workplace chat (toB)

> HR posts a notice in the 3,000-person all-hands group and asks everyone to confirm by 10:00. Who has not read it? If someone asks two years later, can we still tell?

IM Systems in Depth · https://im.liko.page/en/workplace-chat/

The trunk's app is a social app. Users sign up themselves, add their own friends, and their chat history is their own. This chapter sells the same v3 system to companies.

A company of a few thousand people uses it for work. At 9:00, HR posts in the 3,000-person all-hands group: “@all New leave policy. Please confirm by 10:00.” At 10:30 she wants to know who has not read it, so she can remind them one by one. Two years later someone says “I never got that notice”, and the company needs to see whether and when he read it.

This chapter starts from v3 ([chapter 32](/en/v3-million-online/)). It walks the trunk's decisions layer by layer (keep, change or drop), then goes deep on what workplace chat adds.

## 1. What workplace chat needs

| | Social app (the trunk) | Workplace chat |
|---|---|---|
| Accounts | Users sign up themselves | The company creates them and an admin manages them; when a person leaves, the account is disabled |
| Who you can talk to | Friends, people in your groups | Colleagues in the company directory; outside people only after you add them |
| Read state | “Read” in one-to-one chats; in a group, tap to see who has read it (chapter 15) | Who has not read each message, and when each person read it, with a way to nudge them |
| Group size | Up to 500 | All-hands groups of a few thousand |
| History | Belongs to the user | Belongs to the company: kept for years, searchable, exportable, deleted by policy |

The first four rows change decisions we already made. The last row is new. Workplace chat has public examples. WeCom (企业微信) has written about its message system: groups of up to 10,000, a list of who has and has not read every message, and messages stored in the cloud for up to 180 days, hundreds of TB in all ([WeCom's IM architecture](http://www.52im.net/thread-3631-1-1.html), 2021, in Chinese). DingTalk (钉钉) has written about its DTIM ([DingTalk DTIM](http://www.52im.net/thread-4012-1-1.html), in Chinese, first published on InfoQ).

## 2. The trunk's decisions, layer by layer

| Layer | The trunk's decision | Workplace chat |
|---|---|---|
| Connection | Gateways, heartbeats, reconnects, admission (chapters 20–24) | Keep |
| Connection | The app's server signs tokens; revocation by a timestamp ([chapter 25](/en/accounts/)) | Keep. The company's single sign-on signs them. Revocation is used often: for everyone who leaves |
| Client | Web client (chapter 27), old apps (chapter 31) | Keep. Desktop and web are the main devices; an admin can require an upgrade for everyone |
| Message | ACK, dedup, seq, conversation owners, message log (chapters 3–5, 28, 29) | Keep |
| Message | Copy to another city after the ACK ([chapter 37](/en/losing-a-data-centre/)); if a whole city is lost, the last second or so of acknowledged messages is lost with it | Maybe change: the company's records must not be lost, so copy before the ACK, 30 ms slower per send, nothing lost even if a whole city goes |
| Dispatch | Every message updates every member's conversation record (chapters 10, 11) | Keep for groups of hundreds; change for groups of thousands (see below) |
| Storage | One read position per person per conversation, for unread counts ([chapter 14](/en/unread/)) | Keep |
| Storage | Read receipts worked out from read positions ([chapter 15](/en/read-receipts/)) | Change: sections 4 and 5 |
| Storage | New devices pull only recent messages; search on the device is incomplete (chapter 16) | Change: history kept for years in hot and cold tiers; search on the server (sections 6 and 7) |
| Business | Joining, leaving, syncing members (chapters 12, 13) | Keep; add groups built from departments |
| Business | IM users map to the app's users (chapter 25) | Change: users belong to the company's directory (section 3) |

**Groups of a few thousand.** In chapter 11, every message in a group makes the server update one conversation record per member (a small transaction writing two rows: that person's version number and their record). In a group of 3,000, one message every two seconds is 1,500 updates/s, 3,000 rows/s, 15% of one storage server (about 20,000 rows/s, chapter 29). That is fine. But a busy group at 10 messages/s is 30,000 updates/s, 60,000 rows/s, three storage servers. DingTalk's public answer: in big groups, sending a message does not update each member's red dot and last message; they are computed when the user loads the first screen. That is chapter 40's answer for communities, and groups of thousands use it too.

The other changes follow, one section each.

## 3. Accounts belong to the company

In the trunk, Ana signs up herself and the IM gives her a user ID (chapter 25). In workplace chat, the account belongs to the company:

- **The directory.** The company has departments and members. An admin maintains them, or they are synced from the company's own HR system. WeCom's directory API works this way: departments and members can be added, changed and removed, and only an admin can turn on this sync ([directory management](https://developer.work.weixin.qq.com/document/path/90193), in Chinese). An IM user ID no longer means “ana_92 in the app”; it means “this member of this company”.
- **Internal and external.** Colleagues are internal members. Customers and suppliers are external contacts. In WeCom, employees can add contacts from other companies, and there are two kinds of external contact: WeCom users and WeChat users ([customer contact](https://developer.work.weixin.qq.com/document/path/92109), in Chinese). The two kinds of identity see different things. An external contact cannot see the company's directory. A group with external people in it is a different kind of group, and it can have different rules for retention and archiving.
- **Who may message whom.** This follows the directory. People in the same company can message anyone they can find. External people must be added as contacts first. An admin can add rules on top, for example that some departments may not chat with outsiders. All of this is checked **when a conversation is created**. After that, sending a message checks only chapter 12's member row; it does not ask the directory on every message.
- **Leaving.** When an admin marks someone as having left, the IM does three things. It revokes all their tokens with chapter 25's timestamp and closes their open connections. It removes them from every group (chapter 12: one leave per group). Their one-to-one chats stay visible to the other person, with “left the company” next to the name. Slack does the same when it deactivates a member: the person is removed from all channels and signed out on all devices, and their messages and files are not deleted ([Deactivate a member's account](https://slack.com/help/articles/204475027-Deactivate-a-members-account)). Their customers must go to someone else. WeCom calls this “inheritance on leaving” (离职继承): a departed member's customers are assigned to another member, who takes over automatically after 24 hours ([assign a departed member's customers](https://developer.work.weixin.qq.com/document/path/94081), in Chinese).

The history belongs to the company. When a person leaves, the history stays.

## 4. Who has not read it: the trunk in an all-hands group

Chapter 15's read receipts keep nothing of their own. They are worked out from chapter 14's **read position**: one seq per person per conversation (the read cursor), meaning everything up to that seq has been read. “Who has not read message s” means: of the people in the group when s was sent (chapter 12's membership range), those whose read position is still below s.

Chapter 14 also decided: **jumping to an @ does not move the read position.** The messages you jumped over really were not read, so they must not count as read. In a social app that is right. In an all-hands group it goes wrong.

When the notice is posted, many people have unread messages in the group. They see a notification, or the @ next to the red dot, and tap it. That jumps straight to the notice. They read it and reply “Got it”. But they jumped over the messages above the notice (the ones they had not read before it was posted), so their read position is still at the start of that stretch. To HR, they all show as “not read”.

We run it with these assumptions (seed 1):

- when the notice is posted, 30% of members have nothing unread in the group; the others have 1 to 200 unread messages;
- 85% read it within two hours, at times spread evenly over the two hours;
- of those who read it, half jump to it from the @ or a notification, and half open the conversation and read down from their first unread message.

After two hours:

- 2,569 people have read it, and 431 have not;
- by read positions, **1,346** are shown as not having read it: 915 people who read it count as not read, 36% of those who read it. HR would go and nudge every one of them.

With pen and paper: 3,000 people × 85% who read it × one half who jump × 70% with something unread ≈ 892 people. Over 200 seeds, the average is 894 and the most is 947.

There is a second question read positions cannot answer: **when.** 1,301 people read it before 10:00, but a read position is only a place, with no time. Two years later, “had he read it by 10:00?” has no answer.

The trunk is not wrong. In a social group, “read by N” is only nice to know. In workplace chat, the list is used to nudge people and to settle disputes.

## 5. The fix: read ranges, plus a read log

Keep the read position, and add two things beside it:

1. **Read ranges** (listed among chapter 14's other answers). The read position says “everything from the start up to here is read”. A message read by jumping is kept as a separate range, for example “4,900 to 4,930”. When the person scrolls back and fills the gap, the two ranges merge into the read position. The ranges live in the same conversation record row. Usually there are none or a few, and only a few are kept (for example 8).
2. **A** **read log****.** Every read appends one row: who, which conversation, from which message to which, and the minute. Rows are only appended, never changed. The phone reports its read position at most once a second (chapter 14), and reports in the same minute that join up are merged into one row. DingTalk does something similar: the client merges several reads in one conversation into one request, and the server merges the read requests for each message, for example a minute of requests into one.

Then:

- **Who has not read message s**: of the people in the group when s was sent, those whose read position is below s and whose read ranges do not include s. For 3,000 people, a position each (12 bytes) plus a few ranges, a tap reads a little over 36 KB.
- **When he read it**: the first row in his read log for this conversation that includes s. To the minute.
- **Unread counts** still come from chapter 14's read position. The countable messages inside a read range are subtracted from the unread count; chapter 14's “count so far”, stored on every message (how many countable messages the conversation has up to that one), gives that number with one subtraction. Say Ana has read to 4,800, the latest is 5,000, and she has a read range from 4,900 to 4,930: the count so far at 4,930 minus the count at 4,899 is 31, so her unread count goes from 200 to 169 (assuming every message after 4,800 is an ordinary one that counts).

In the all-hands group above: of the people who jumped, 915 had an unread stretch before the notice, so 915 ranges are written. The log has 2,569 rows, one per reader. Exactly those 431 people are shown as not having read it. Over 200 seeds, none is wrong. The 1,301 who read it before 10:00 can be found in the log.

**How the sender sees the numbers go up.** While HR has the group open, each of her messages shows “N unread” under it. There is no need to push every single read. A background job reads the read log (a consumer, like those behind chapter 29's message log), gathers a minute's worth and tells her “these messages' numbers changed”, and her phone fetches them.

## 6. Kept for years: hot and cold

At v3's volume: 400 million messages a day, 200 bytes each.

- **80 GB** a day, **29.2 TB** a year, 87.6 TB with three replicas;
- 5 years is 146 TB, **438 TB** with three replicas.

But most reads touch only recent messages. So history is split in two tiers:

- **hot**: the last 30 days, 2.4 TB, 7.2 TB with three replicas, on fast disks as in the trunk;
- **cold**: the older 143.6 TB, on cheap disks, with erasure coding instead of three replicas. For example, every 6 blocks of data get 3 parity blocks: any 3 can be lost and the data can still be rebuilt, for only 50% extra. 143.6 TB × 1.5 ≈ 215 TB.

About 223 TB in all, 51% of keeping everything as three replicas. The cost is that cold reads are slow: scrolling back two years means a wait. WeCom does the same and keeps cold data on cheap SATA disks.

Text is the small part. With chapter 8's assumption that 5% of messages are 200 KB photos, that is 4 TB a day, 50× the text, 1.46 PB a year, and work files are bigger still. They live in object storage, need hot and cold tiers too, and fall under the same retention rules below.

The read log is kept just as long: about 600 million rows a day (10 million people opening about 60 conversations a day each, chapter 7's assumption), 28 bytes a row, 16.8 GB a day, about a fifth of the text.

## 7. Search

Chapter 16 said a new device pulls only recent messages, so search on the device is incomplete. With years of history, search moves to the server.

- **Where it comes from**: the search index is one more consumer behind the message log (chapter 29), and builds the index as the log grows. Recalls, edits (chapter 18) and deletions by policy reach it the same way.
- **How big**: assume the index is about the size of the text, another 80 GB a day.
- **Who can find what**: only messages in your own conversations, and only from the time you were in the group (chapter 12's membership range). Do not build an index per person: a message reaches 6.67 people on average (chapter 6's assumption), so one copy each is 533 GB a day. Build the index per conversation and filter at query time by “which conversations am I in, and from which message”.
- **How to shard it**: by company looks natural, but a big company becomes a hot spot. Slack first split its databases by workspace; a large customer could outgrow one shard, so it moved messages to sharding by channel ([Scaling Datastores at Slack with Vitess](https://slack.engineering/scaling-datastores-at-slack-with-vitess/), 2020). The trunk already shards by conversation (chapter 28), so we keep that.

## 8. Retention and export

The history belongs to the company, so the company decides how long it is kept and who may see it.

- **Retention.** An admin chooses “keep everything” or “delete after N days”. On Slack's paid plans, data is kept for the lifetime of the workspace by default; it can be changed to delete automatically, and deletions run once a day ([Customize data retention](https://slack.com/help/articles/203457187-Customize-message-and-file-retention-policies)). Deleting must reach every copy: hot, cold, replicas, the search index, the read log, files, and every device's local database (like a recall in chapter 18, an event goes down to the devices).
- **Deleting is writing too.** If every company kept only one year, 400 million messages would be deleted every day, 4,630 a second on average, as many as are written. Squeezed into four hours at night, that is about 28,000 rows/s, one and a half storage servers, before the index and the files. So cold data is best laid out by company and month: when it expires, the whole block is dropped instead of deleting row by row.
- **Export.** For compliance, a company wants to export every conversation. On Slack's Business+ plan, exporting all channels and direct messages needs an application ([Export your workspace data](https://slack.com/help/articles/201658943-Export-your-workspace-data)). WeCom has “conversation archiving” (会话内容存档): the company chooses which employees are archived and fetches their work conversations through an API ([conversation archiving](https://developer.work.weixin.qq.com/document/path/91360), in Chinese). Export follows the permissions from section 7: what the exporter can see is recorded, and can be audited.

## 9. Cost

- **One more log row per read.** About 16.8 GB a day, about 20,000 rows/s at peak, one storage server. One row per message per reader (16 bytes a row) would be 42.7 GB a day and about 93,000 rows/s at peak, 4.6 storage servers.
- **Read ranges make the unread count harder.** Messages inside ranges must be subtracted. When there are too many ranges, only the latest few are kept, and older jumps are only in the log.
- **Times only to the minute.** Merging each minute into one row is what keeps the rows down.
- **Deleting must reach every copy** (section 8). Miss one, and it is not really deleted.
- **A departed person's phone.** The server can tell it to wipe its local database only when it connects. A phone that never goes online again keeps its copy.

## 10. Other answers

- **One row per message per reader.** The most direct: it knows when, and it knows jumps. The cost is rows: a group of 500 with 1,000 messages a day writes 500,000 rows a day; if each member opens it 20 times a day, the log needs only 10,000.
- **WeCom.** One copy of each message per member (write fan-out), which the article says serves toB needs such as receipts and “delete only on my side”. The read list of a receipt is stored in the sender's copy: when someone reads it, a new message pointing at the original is appended. Phones confirm several messages as read at once, and the server processes them together: one confirmation per reader, and, after a queue, several per sender. At the online peak this saved 44% of writes in all. Requests for the same account go through a queue one at a time, so two people's reads do not overwrite each other.
- **DingTalk DTIM.** Ordinary messages are stored once; what differs per person (messages for one person only, messages one person deleted) goes into that person's inbox. Read state is written when a message is stored. Reads in big groups are merged.
- **A bitmap per message.** One bit per group member, 375 bytes per message in a group of 3,000. It says who read it, not when.
- **Small groups only, kept for a few days.** As chapter 15 noted, in Telegram’s public API, “who read this group message, and when” (`messages.getMessageReadParticipants`) is only available in groups up to a size limit (100 members by default), and the receipts are kept only for a while (7 days by default), then deleted ([method](https://core.telegram.org/method/messages.getMessageReadParticipants), [client configuration](https://core.telegram.org/api/config)). A social product can save that way; an office IM’s all-staff group has thousands of members and must be checked two years later, the opposite.
- **Like the trunk, read positions only.** Accept that jumps count as unread and that there are no times. Enough for a social product.

## 11. This chapter's decision

**Decision card**

- Problem: Workplace chat must know, for every message, who has not read it and when each person did. The list must be exact and kept for years. The trunk works receipts out from read positions: in an all-hands group of 3,000, people who jump to a message from an @ or a notification still count as not having read it (seed 1: 915 of the 2,569 who read it), and there are no times.
- Choice: Keep the read position for unread counts. Beside it, keep read ranges (for jumps) and an append-only read log merged per minute. Who has not read it = those in the group at the time whose read position and read ranges do not include the message; when = from the log. Accounts belong to the company's directory, checked with external contacts when a conversation is created. History in hot and cold tiers; a search index built from the message log; retention and export set by the admin.
- Cost: The read log is about 16.8 GB a day, about one storage server at peak; unread counts must subtract messages in ranges; times only to the minute; deletion must reach cold data, the index, the log, files and devices; new services for the directory, retention and export.
- Revisit when: Groups of tens of thousands (chapter 40); no server search under end-to-end encryption (chapter 42); one IM service for many companies (chapter 44).
- Other answers: One row per message per reader; WeCom (write fan-out, receipts stored in the sender's copy, one account at a time); DingTalk DTIM (stored once plus a personal inbox, merged reads); a bitmap per message; Telegram (small groups only, receipts kept 7 days); read positions only.

Workplace chat makes read state and history heavier. The next chapter goes the other way: a community of 100,000 people, most of whom rarely visit. How can each message cost less?
