IM Systems in Depth

Glossary

Each idea in this series has one name in each language. Here is what each means in this series; names you will meet elsewhere follow “also”. Only terms used in published chapters are listed; each new chapter adds its own.

Basics

schemaschema(消息定义)

One file that says which fields each message has, their types and their numbers, from which the server’s and every app’s code is generated. Protobuf’s .proto files are one kind.

also: IDL, .proto file

Explained in: Side trip W2 Encoding, Chapter 31 Old apps, new servers

varintvarint(变长整数)

Small numbers in few bytes: 7 bits a byte, the top bit meaning “more follows”. 0–127 take one byte; 300 is AC 02. Protobuf writes integers and field numbers this way.

also: variable-length integer

Explained in: Side trip W2 Encoding

long-lived connection长连接

A TCP or WebSocket connection a device keeps open to the server. With it, the server can push a message at any time instead of waiting to be asked.

also: persistent connection

Explained in: Chapter 2 From polling to push, Chapter 20 Split the gateway

ACKACK(确认)

The receiver’s “got it” to the sender. In this series, the server’s ACK means the message is in the message log, not that anyone has received it.

Explained in: Chapter 3 Acks and retries

sync/async boundary同步与异步的分界

The line between what the sender waits for and what it does not. In this series it is the message log: the sender waits until the append, and all delivery after it is asynchronous.

Explained in: Chapter 29 A queue before storage

write fan-out写扩散

When a message is sent, write one entry into each recipient’s inbox; reading means reading your own inbox. More writes, fast reads; suits conversations with few members. Chapter 6 weighs it; chapter 10 shows it goes with client-built chat lists, where the phone builds its chats from the inbox. Neither v1 nor v2 uses it.

also: fan-out on write

Explained in: Chapter 6 Offline sync, Chapter 10 Who builds the conversation list

read fan-out读扩散

Keep one copy in the conversation’s timeline, and each reader fetches it from there. Few writes, more reads; suits big groups. All of v1’s and v2’s messages are kept this way (chapters 6, 10); it needs the server to build the chat list and devices to sync it.

also: fan-out on read

Explained in: Chapter 6 Offline sync, Chapter 10 Who builds the conversation list

business layer业务层

The rules: accounts, friends, groups, permissions, moderation. It sits beside the message path: the layers call it when they need to, and messages do not flow through it.

Explained in: Chapter 12 Joining and leaving a group, Chapter 14 Read position and unread counts, Chapter 25 Accounts: the app’s users and the IM’s

membership range成员区间

The stretch of seqs a person can see in a group: min_seq is the first, max_seq the last (unset while they are still in it). Joins and leaves take a seq in the conversation like messages, so who gets each message and how far back each person can read are worked out from this range, whenever they are worked out. The group has a pair too: max_seq is the latest seq, and below min_seq nobody sees anything (chapter 12).

also: min_seq / max_seq

Explained in: Chapter 12 Joining and leaving a group

before-send callback发送前回调

Before storing a message, the IM calls the business’s service to ask “may this go through?” and lets it through, refuses it or rewrites it accordingly. The rules are always current, but every send waits an extra round trip, and when the business is down the IM can only let everything through or refuse everything (branch: IM as a platform).

Explained in: Chapter 44 IM as a platform

end-to-end encryption端到端加密

A message is encrypted on the sending device and only the recipients’ devices can open it; the server passes on bytes it cannot read. Each device has its own keys, so the unit of encryption is the device, not the person.

also: E2EE

Explained in: Chapter 42 Private messenger, Side trip W4 Group encryption

tenant租户

One business plugged into a shared IM service, such as a shopping app or a game. The service is shared; each tenant’s IDs, rules and quotas are kept apart, so one cannot drag down the rest.

Explained in: Chapter 44 IM as a platform, Chapter 45 Same decisions, different products

Client

member version成员版本号

A per-group number that goes up with every member change (join, leave, group nickname, role), stamped on that member’s row. The phone keeps the member list with its version and sends it when opening the group; the server answers with just the version if nothing changed, otherwise only the changed rows (chapter 13).

Explained in: Chapter 13 Syncing group info and members

IM SDK

The client library inside every app: the outbox and retries, sync, the local database, and the API the app uses.

Explained in: Chapter 7 The local database, Chapter 30 One SDK, many platforms

outbox发件箱

The messages on the client that have not been ACKed yet. After a disconnect or a restart, they are resent from here.

Explained in: Chapter 3 Acks and retries, Chapter 7 The local database

local database本地数据库

The phone’s own copy of messages, conversations and the sync cursor, usually SQLite. The screen reads only it, and everything from the network is written into it first; a cursor is committed in the same transaction as the messages it stands for.

Explained in: Chapter 7 The local database, Chapter 30 One SDK, many platforms

resumable upload断点续传

A file is uploaded in parts and the server remembers which have arrived; after a drop the phone asks how far it got and sends only the rest instead of starting over.

also: chunked upload, multipart upload

Explained in: Chapter 8 Images and files

message ID消息 ID

A unique ID the client gives each message before sending it, unchanged on resends. The server uses it to recognise a resend.

also: client message ID

Explained in: Chapter 4 Duplicates

sync cursor同步游标

What a device keeps as “I have everything up to here”. In this series it is two things: the conversation-record version, and each conversation’s last seq. A device that comes back asks what changed after that version and each conversation’s latest seq, then pulls what it is behind on.

Explained in: Chapter 6 Offline sync, Chapter 7 The local database, Chapter 16 Many devices

record version记录版本

A per-person number that goes up with each change to their conversation records; it is taken under a lock on that person’s users row, so version order is commit order. The phone keeps the version it has and asks “what changed after this” on return; a group it was added to, or a chat with new messages, shows up here.

Explained in: Chapter 6 Offline sync, Chapter 11 Syncing the conversation list, Chapter 16 Many devices

sender keys发送者密钥

In an end-to-end encrypted group, each device makes a chain key of its own and first sends it to every device in the group over pairwise sessions; after that each message is encrypted once and uploaded once. The cost: when someone leaves, every device makes a new key and sends it out again.

also: Sender Key

Explained in: Chapter 42 Private messenger, Side trip W4 Group encryption

Connection layer

gateway网关

A connection-layer server. Each holds many long-lived connections and handles auth, heartbeats and traffic without reading messages. One of the few stateful parts: a restart drops all its connections.

also: connection server

Explained in: Chapter 20 Split the gateway, Chapter 22 A million connections

locator接入调度

The service that tells a device which gateway to connect to. The device asks it first, then connects.

Explained in: Chapter 20 Split the gateway, Chapter 24 Reconnect storms

heartbeat心跳

A small packet sent while a connection is idle. It finds connections that have died, and keeps carrier equipment from forgetting connections that stay silent too long.

Explained in: Chapter 23 Heartbeats

NATNAT(地址转换)

Equipment at the carrier or in a home router that swaps the phone’s private address for a public address and port and records the mapping; packets pushed down find the phone only through it. A mapping idle too long is deleted, which is why long-lived connections need heartbeats.

also: network address translation

Explained in: Chapter 23 Heartbeats, Chapter 33 Where did my message go?, Side trip W3 TCP or UDP

head-of-line blocking队头阻塞

TCP delivers in order: when one packet is lost, the data behind it that has already arrived waits in the kernel for the resend. QUIC removes this only between streams; within one stream it remains.

Explained in: Side trip W3 TCP or UDP

QUIC

A transport protocol on top of UDP that does its own resending, encryption and congestion control; HTTP/3 runs on it. One connection carries many independent streams, each in order within itself.

Explained in: Side trip W3 TCP or UDP

connection migration连接迁移

QUIC identifies a connection by the connection ID in each packet, not by address and port, so when a phone moves from Wi-Fi to 4G the connection carries on. The load balancer must route by connection ID too. The trunk does not use it: a network change means a reconnect and a sync.

Explained in: Side trip W3 TCP or UDP

Message layer (receive)

seq序号(seq)

The number of each message in a conversation, assigned in order by the message service that owns it. It sets the order, lets a client see which message is missing, and tells a returning phone which conversations it is behind on.

also: sequence number

Explained in: Chapter 5 Ordering, Chapter 12 Joining and leaving a group

dedup去重

When a message ID arrives again, return the first result instead of storing another copy. The network can only deliver at least once; dedup makes each message show once.

Explained in: Chapter 4 Duplicates

sharding by conversation按会话分片

Each conversation is owned by one message service. Its seq is then assigned by a single owner, which is what keeps the order.

Explained in: Chapter 28 Sharing out the conversations

message log消息日志

A durable, append-only queue: the hand-off between receiving and delivering. A message counts as received once it is in the log, and the server ACKs then. It usually runs on a message queue, such as a topic partitioned by conversation.

Explained in: Chapter 29 A queue before storage

idempotency key幂等键

A key of the business’s own that its server sends with a message, such as an order number plus the notice type. However many times the same key comes, the result is the first one: the platform remembers the message ID the key produced and returns it on a retry instead of sending again.

Explained in: Chapter 44 IM as a platform

Dispatch layer (deliver)

latest-seq sweep最新 seq 一览

On return the phone asks for the latest seq of all its conversations at once; the server reads them from a cache in one batch, one key per conversation. Any conversation whose seq is above the phone’s has new messages. Its reads grow with the number of conversations, not with the number of new messages.

Explained in: Chapter 6 Offline sync, Chapter 11 Syncing the conversation list

online push在线推送

A push through a gateway, over the connection, to an online device. It is a hint: if it is lost, the device catches up by pulling.

Explained in: Chapter 2 From polling to push, Chapter 20 Split the gateway

OS push系统推送

A notification sent through Apple’s APNs, Google’s FCM or a phone maker’s channel to a phone with no live connection. A third party, best effort, with no guarantee.

also: offline push, push notification

Explained in: Chapter 17 Waking an offline phone

sync service同步服务

Answers a device that comes back: what changed in its conversation records, each conversation’s latest seq, and then the messages it is behind on, page by page (from the timeline, or from the inbox with write fan-out).

Explained in: Chapter 6 Offline sync, Chapter 16 Many devices

broadcast conversation广播会话

Where a notice to all of a tenant’s users goes, stored once. It has no member rows and is in nobody’s conversation records; a phone asks for its latest seq during sync and pulls when it is newer.

Explained in: Chapter 44 IM as a platform

room router房间路由

The routing for live rooms: for each room it knows only which gateways have viewers, not who they are. A room’s message goes to it, it sends one frame to each of those gateways, and each pushes to the viewers in its own group.

Explained in: Chapter 41 Live rooms, Chapter 45 Same decisions, different products

batching打包

In a live room, a second’s messages go out as one batch, so each connection gets one push a second, compressed. Pushes no longer grow with the room’s message rate; the cost is half a second’s wait per message on average.

Explained in: Chapter 41 Live rooms, Chapter 45 Same decisions, different products

Storage layer

conversation record会话记录

One row per person per conversation: pinned, muted, the conversation’s latest seq, and later how far they have read. Written on join, leave or a settings change, and from chapter 11 on also when a message arrives (with the latest seq). Each change bumps that person’s record version. The messages are not here.

Explained in: Chapter 6 Offline sync, Chapter 10 Who builds the conversation list, Chapter 11 Syncing the conversation list, Chapter 14 Read position and unread counts

read cursor已读游标

How far a person has read in each conversation. It lives on the server, is shared by that person’s devices, and drives unread counts and read receipts. Not the same as the sync cursor.

Explained in: Chapter 14 Read position and unread counts, Chapter 15 Read receipts

message store消息存储

The long-term history of messages, one timeline per conversation. Dispatch writes it after reading the log, so it lags the ACK a little.

Explained in: Chapter 29 A queue before storage

timeline时间线

All of a conversation’s messages in seq order. With read fan-out, everyone reads from here.

Explained in: Chapter 10 Who builds the conversation list

inbox信箱

A per-person index of “new messages for you”, with its own per-user sequence. Write fan-out writes it, and devices pull after their position in it. Chapters 6 and 10 weigh it against asking every conversation; this series’ v2 does not use it.

also: sync queue

Explained in: Chapter 6 Offline sync, Chapter 10 Who builds the conversation list

online routes在线路由表

Which gateway each online device is connected to. Gateways register devices as they connect, and dispatch looks gateways up here.

Explained in: Chapter 20 Split the gateway

object storage对象存储

A storage service that keeps whole files by name (S3, for example); images and files go here, not into the message table. The phone gets a short-lived upload address and uploads to it directly; downloads usually go through a CDN.

Explained in: Chapter 8 Images and files

read ranges读过的段

Kept beside the read position in workplace chat: messages read by jumping ahead are recorded as a range, such as “4,900 to 4,930”, which merges into the read position once the gap is filled. They live in the conversation record row; only a few are kept.

also: ranges read

Explained in: Chapter 39 Workplace chat (toB), Chapter 45 Same decisions, different products

read log已读流水

An append-only record: each read adds a row with who, which conversation, from which message to which, and the minute. Workplace chat uses it to answer “who has not read it, and when did the others”.

Explained in: Chapter 39 Workplace chat (toB), Chapter 45 Same decisions, different products

erasure coding纠删码

Data is split into blocks and parity blocks are computed from them; any blocks, up to the number of parity blocks, can be lost and rebuilt. With 6 data and 3 parity blocks the overhead is 50%, against 200% for three replicas. It suits cold data.

Explained in: Chapter 39 Workplace chat (toB)

store and forward存储转发

The server holds a message only until the device fetches it: deleted once fetched, and after a while if never fetched (30 days at WhatsApp). History lives on the users’ own devices, not on the server.

Explained in: Chapter 42 Private messenger