IM Systems in Depth

Chapter 44

IM as a platform

One IM serves dozens of apps at once: a shopping app wants to notify ten million users, a game’s callback service goes down, a clinic lets patients talk only to their own doctor. What has to be split by business, and what stays shared?

This chapter is a first draft. The author is still checking it, and its text and numbers may change.

Up to chapter 38, this IM served one app: Ana, Ben, the hiking group, and every rule set by that app.

Now it serves many at once: a shopping app, a game, a clinic’s consultation app. Cloud IM services (Tencent Cloud IM, RongCloud, NetEase Yunxin) are sold this way, and large companies often let several of their apps share one IM. Each business that plugs in is a tenanttenant租户One business plugged into a shared IM service, such as a shopping app or a game. The service is shared; each tenant’s IDs, rules and quotas are kept apart, so one cannot drag down the rest.See the glossary: like the tenants of one building, they share the building, and each has its own locked door.

This chapter’s question: when one IM serves many tenants, what has to be split by tenant, and what stays shared?

The short answer first: how messages are stored, ordered and synced does not change at all. What changes is what each tenant brings with it: its IDs, its notices to everyone, its retries, its rules, and how much of the shared resources it can take.

1. Every ID carries its tenant

The shopping app has a user called ana_92, and so does the game. They are not the same person.

Chapter 25 already has half the answer: the IM gives each app user an 8-byte ID of its own, through a mapping keyed by (app, ana_92). So IDs inside the IM never collide. What is left is the edge:

  • Every call from outside carries its tenant, and the platform turns (tenant, ana_92) into the internal ID at the edge. Every Tencent Cloud IM server API call carries an SDKAppID, “the application identifier the app gets from the IM console” (REST API intro).
  • Nothing from outside names an internal ID. Otherwise the shopping app could fill in a game user’s internal ID and message him.
  • Conversations and groups belong to a tenant too. A conversation holds only people of one tenant.
  • Tokens are signed with each tenant’s own key. In chapter 25 the app’s server signs the login token with its private key and the gateway checks it with the public key; now each tenant has its own key pair, and the gateway picks the public key of the tenant named in the token.

2. A notice to everyone

8 p.m. The shopping app wants to send “The sale starts now” to all 10,000,000 of its users.

What would the trunk do? Treat them as members of one conversation: the message is stored once, and each recipient’s conversation recordconversation record会话记录One row per person per conversation: pinned, muted, the conversation’s latest seq, and later how far they have read. Written on join, leave or a settings change, and from chapter 11 on also when a message arrives (with the latest seq). Each change bumps that person’s record version. The messages are not here.See the glossary is updated (chapters 10 and 11). Chapter 11 counted each update as one small transaction that writes two rows: the person’s record version, and the record itself.

  • 10,000,000 record updates in all: 20,000,000 rows.
  • One storage node writes 20,000 rows a second (chapter 29): 1,000 seconds. All 20 of v3’s record shards together (chapter 36; say each writes as fast as one storage node): 50 seconds, and at peak chat’s own record updates already take 46% of them (185,000 rows a second, chapter 32).

But the trunk is not crushed by this: chapter 34 limits each conversation to 5,000 recipients a second; for one message to ten million people, we take that to mean its record updates go out at 5,000 people a second. Nothing breaks. What is wrong is something else:

  • The last person waits 33 minutes (10,000,000 ÷ 5,000 = 2,000 seconds). The last “The sale starts now” arrives at 8:33, when it is no use.
  • For that half hour, half of one storage node is busy (5,000 people × 2 rows = 10,000 rows a second) writing twenty million rows that all say the same thing.

Public cloud IMs choose the slow side too:

  • Tencent Cloud IM’s push to all members keeps cloud history (“roaming”) and unread counts by default, with a “push concurrency limit of 200 people per second” (all-member/tag push): 13 h 53 min for ten million.
  • Easemob’s app-wide broadcast is written into the server-side conversation list, kept in cloud history and can count as unread, at most 1,000 users a second (Easemob docs): 2 h 47 min.

The fix: store it once, read it on open

Anything sent to everyone becomes chapter 10’s read fan-outread fan-out读扩散Keep one copy in the conversation’s timeline, and each reader fetches it from there. Few writes, more reads; suits big groups. All of v1’s and v2’s messages are kept this way (chapters 6, 10); it needs the server to build the chat list and devices to sync it.See the glossary:

  • One broadcast conversationbroadcast conversation广播会话Where a notice to all of a tenant’s users goes, stored once. It has no member rows and is in nobody’s conversation records; a phone asks for its latest seq during sync and pulls when it is newer.See the glossary per tenant. Every user of the tenant counts as in it, but no member rows and no conversation records are written. “The sale starts now” is one message in it, written once.
  • Ask about it during sync. Each sync fetches only the conversation records whose version has gone up (chapter 11); the broadcast conversation has no record, so it is never among them. So each sync also asks for the broadcast conversation’s latest seq: one more number the server reads from its cache. If the broadcast conversation’s seq is ahead of the phone’s, the phone fetches that one message; everyone fetches the same one, from cache.
  • Those online get a “go sync” nudge. 10% online is 1,000,000 people; spread over a minute that is 16,667 pushes a second, 1.7% of what the 20 gateways can push (50,000 a second each, 1,000,000 in all).
  • The read position is written only on reading. As with any conversation (chapter 14), a read cursorread cursor已读游标How far a person has read in each conversation. It lives on the server, is shared by that person’s devices, and drives unread counts and read receipts. Not the same as the sync cursor.See the glossary row is written when he looks at it; for someone who never opens the app, nothing is written.

NetEase Yunxin’s broadcasts seem to have this shape: “at most the latest 10,000 offline broadcast messages are kept”, and broadcasts “do not support push, roaming or creating a conversation” (broadcast messages). The docs don’t say how they are stored.

Sending to some. A list of, say, 2,000 people (a size assumed for the example) is like a big group: ordinary write fan-outwrite fan-out写扩散When a message is sent, write one entry into each recipient’s inbox; reading means reading your own inbox. More writes, fast reads; suits conversations with few members. Chapter 6 weighs it; chapter 10 shows it goes with client-built chat lists, where the phone builds its chats from the inbox. Neither v1 nor v2 uses it.See the glossary, but in the tenant’s own low-priority queue, so it does not compete with chat. A tag (a label the tenant puts on users, such as “Shanghai” or “member”) that matches millions of people is a broadcast again: a broadcast conversation per tag, or a condition on the broadcast, filtered by the person’s tags during sync.

3. The business’s own retries

Notices like “Your order has shipped” are sent by the business’s server through the IM’s API. When a call times out, it retries; but the first call may have gone through, with only its answer lost.

Chapter 4’s dedup relies on the message IDmessage ID消息 IDA unique ID the client gives each message before sending it, unchanged on resends. The server uses it to recognise a resend.See the glossary the sender makes, which catches resends on the network; a business retry is a new call, and to the IM it is a second message. Chapter 4’s “in practice” said it: this takes the business’s own ID.

Tencent Cloud IM’s single-chat API builds its dedup key from the sender, MsgSeq and MsgRandom (two numbers the caller fills in) and a checksum of the content, and “the key is valid for 120 seconds” (send a single-chat message). A retry within two minutes is caught; a job that runs again five minutes later is not.

The platform’s way: the server API also takes a key of the business’s own, usually called an idempotency keyidempotency key幂等键A key of the business’s own that its server sends with a message, such as an order number plus the notice type. However many times the same key comes, the result is the first one: the platform remembers the message ID the key produced and returns it on a retry instead of sending again.See the glossary: however many times the same key comes, the result is the same as the first time. Say, the order number plus the notice type. The platform keeps (tenant, key) → message ID for 24 hours under a unique constraint; when the same key comes again, it returns the first message’s ID and sends nothing. How much is that? v3 handles 400,000,000 messages a day; if a tenth are sent by servers, that is 40,000,000, at 64 bytes each (the key and the message ID): 2.56 GB a day.

4. Who may message whom

Chapter 12 settled who may post in a group: its members. What about private chats? Every tenant has its own rules: a social app lets only friends message each other; a shopping app lets a buyer with an order message the seller; a clinic lets patients message only their own doctor; nearly all let you block someone. These rules live with the tenant (the business layerbusiness layer业务层The rules: accounts, friends, groups, permissions, moderation. It sits beside the message path: the layers call it when they need to, and messages do not flow through it.See the glossary); the IM does not know them.

The obvious way: before storing each message, the IM calls the tenant’s service and asks “may this go through?” Normally the tenant calls the IM; here the IM calls back into the tenant, hence a callback, and this one is a before-send callbackbefore-send callback发送前回调Before storing a message, the IM calls the business’s service to ask “may this go through?” and lets it through, refuses it or rewrites it accordingly. The rules are always current, but every send waits an extra round trip, and when the business is down the IM can only let everything through or refuse everything (branch: IM as a platform).See the glossary. Every message waits one more round trip, say 20 ms; and when the tenant’s service fails, the IM has only two choices: wait for the timeout and let it through, or refuse everything.

Here are two minutes of private messages to Ana: Cara is a friend, Ben is blocked at 0:20, and there is a stranger (Ana accepts friends only). From 1:00 to 1:30 the tenant’s service is down:

business downAna blocks BenCaraa friendBenblocked at 0:20StrangerAna: friends only0:000:301:001:302:00

allowed, deliverednot allowed, refusednot allowed, deliveredallowed, refused

12 that should have been refused reached Ana. The business was asked 64 times; each send waited an extra 20 ms normally, and the full 2 s while it was down.

  • A timeout lets it through: in the half minute it is down, all 12 messages from Ben and the stranger get through.
  • A timeout refuses: nothing leaks, but 5 of Cara’s are refused.
  • Either way, every message waits on the tenant, and in the half minute it is down each waits the full 2 seconds. At v2’s peak of 139 a second, that is 139 × 30 = 4,170 messages stuck.

Who may talk to whom rarely changes; messages flow all the time. So the decision can move to the moment a relation changes (pick “Read the IM’s own state” in the scene):

  • When a relation changes, the tenant writes it into the IM there and then. Two people become friends: the tenant calls the server API to open a private conversation. An order opens a buyer–seller conversation. Ana blocks Ben: Ben’s member row in their private conversation is marked “may not post”. Only after that write does Ana see “blocked”.
  • A send checks only the IM’s own state. The send’s transaction already locks the conversation’s row and checks that the sender is a member (chapter 12); it also checks whether his member row says “may not post”. A block locks that row before changing it, so a message Ben is sending either comes before the block or is refused.

On a platform, no one answer is right for every tenant, so all three are offered and each tenant chooses:

  • The platform hosts friends and blocklists. They are stored on the platform, which checks them itself on each send. Handy for a social app; but buyer and seller, patient and doctor, the platform does not know. Tencent Cloud IM offers this, yet sending a single-chat message “does not check the friend relationship by default”; it is switched on in the console (friend management): a customer-service product can hardly make users add the agent as a friend first.
  • The tenant writes relations in as conversation state: the way above. It does not depend on the tenant being up; the cost is two copies, which need reconciling.
  • A callback per message: for rules that change daily or depend on the content.

5. Callbacks before and after a send

There are two kinds. Before a send: the IM waits for the tenant’s answer, which can let it through, refuse it, or change the content (say, replace a banned word). After a send: the message has gone; the tenant is told so it can record it or copy it into its own systems. OpenIM’s docs say after-event callbacks are asynchronous, and their answer cannot change the result (webhooks overview).

What if a before-send callback times out? The public defaults:

Timeout After a timeout
Tencent Cloud IM 2 s; for the before-send-message callback it cannot be changed delivered by default, the console can switch to not delivering; no retry
NetEase Yunxin 2 s by default, settable let through by default, settable to block
RongCloud 5 s 3 more tries, then delivered by default; with widespread timeouts in a short time, callbacks pause for 1 minute
OpenIM (open source, self-hosted) 5 s in the config file the shipped config has failedContinue: true: on failure the send goes on

Sources: Tencent Cloud IM callbacks, NetEase Yunxin third-party callback setup, RongCloud message callback, OpenIM webhooks.yml.

All four let the message through by default: when a tenant’s service is down, its users should still be able to chat. The cost is that rules kept in a callback are only “best effort”.

Callbacks can also pass one tenant’s failure on to others. Assume a big tenant sends 1,000 messages a second at its peak (a round number chosen for the example), each with a callback. By chapter 36’s Little’s law, the number waiting at once = arrivals a second × how long each waits: normally each takes 20 ms, so 1,000 × 0.02 = 20 are waiting; when its service hangs, each waits the full 2 seconds, and 1,000 × 2 = 2,000 sends are waiting at once. If they share a pool with other tenants, everyone slows down. So:

  • Each tenant’s callbacks have their own slots (chapter 36’s bulkhead): at most so many may wait at once; when they are full, a new one gets the tenant’s chosen default (let through or refuse) at once, with no queueing.
  • After many timeouts, stop asking for a while (a circuit breaker), then try again. RongCloud’s one-minute pause is exactly that.

Telegram’s Bot API is a platform calling third parties too: new messages for a bot are pushed to the address the bot has set, like an after-send callback that blocks nothing. The bot fills in its own slots: at most so many connections at once, 1 to 100; if the answer is not a 2xx, Telegram retries a reasonable number of times and then gives up (setWebhook).

6. Quotas, and the shared connections

Everything a tenant can start has its own quota, that is, a cap: at most so many a second or a day. Public examples:

  • Server API calls: Tencent Cloud IM’s single-chat API, 200 a second by default.
  • Broadcasts: Tencent Cloud, 100 a day, more than 1 second apart; NetEase Yunxin, 10 a minute; Easemob, one per 30 minutes and 3 a day.
  • Bot messages: Telegram asks bots to send no more than 1 message a second in one chat, allows at most 20 a minute in a group, and about 30 a second for bulk notifications; with paid broadcasts enabled, up to 1,000 a second (Bots FAQ).

Callback slots and queue lengths are capped too. The largest tenants get their own queues and message-log partitions (chapters 29 and 34): when one falls behind, only it is late.

The connection layer is shared by all tenants. The gateway registers each connection under the tenant in its token, keeps tenants apart when it forwards, and caps each tenant’s connections. Inside one company it is the same: an article on 百度Geek说 says that “if every business maintains its own long-connection service, there is a large cost of duplicated development and maintenance”, so messaging, live streams, push and others “share one long connection” (repost on 52im, original title 千万级高性能长连接Go服务架构实践; our translation).

7. The cost

  • A broadcast is in nobody’s conversation records. Chapter 17’s badge is a total kept on the users row, adjusted each time a conversation record changes; the broadcast conversation has no record, so until Ana opens the app the total does not include “The sale starts now”. For the badge to count it while the app is closed, the server has to take the broadcast conversation’s latest seq separately and add the ones she has not read.
  • OS push cannot be avoided. Making ten million closed phones show a notification still takes ten million OS pushes; they can only go at low priority, slowly.
  • Quotas need someone to set them. Too small and a tenant cannot send; too large and they don’t stop the noisy neighbour: one tenant fills the shared resources and the others slow down.
  • Every line of code must carry the tenant. Miss one, and one tenant’s data may go to another.

8. Other answers

  • Write fan-out for broadcasts too, rate-limited. Tencent Cloud IM (by default) and Easemob, as in section 2: the broadcast goes into everyone’s history and unread count, and ten million people take hours.
  • A separate deployment for the largest tenants. The strongest isolation and the highest cost; for a few big customers.
  • Plugins instead of callbacks. A self-hosted IM can run rules as plugins inside the IM’s process, with no network. turms loads plugins from its plugins directory when the server starts (turms plugin docs).
  • A copy of the relations in the IM, synced asynchronously. The tenant sends an event when a relation changes and the IM catches up; the tenant does not wait for the IM, but a block takes a few seconds to apply.

9. This chapter’s decision

That completes the six branches. The same decision, in another product, may come out the other way; the last chapter puts them in one table.

Comments

Comments live in GitHub Discussions and need a GitHub account.