Chapter 41
Live rooms
A live stream puts 1,000,000 people in one room. Thousands join and leave every second, and the screen has space for only a dozen or so comments. Can the trunk's group design cope? If not, what has to change?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
8 p.m., and a product launch goes live. 1,000,000 people enter the same live room, as many as the whole v3 app has online at peak. Comments scroll across the bottom of the video: someone asks the price, someone types “here!”, someone sends a rocket gift.
The trunk’s hiking group has 500 members; every message must arrive, and a member who comes back must be able to catch up. A live room is the other way round:
| Hiking group | Live room | |
|---|---|---|
| People | 500 | 1,000,000 |
| Joining and leaving | a few times a day | thousands of times a second |
| Every message must arrive | yes | no; losing some is fine |
| Arriving late | still must arrive, even seconds late | useless: the video has moved on |
| Unread, read, history | yes | no; a newcomer only needs to see the last screenful |
This chapter’s question: both are “one message to many people”. Where does the trunk’s group design break in front of 1,000,000 people, and what has to change?
Among public sources, “百度直播消息服务架构实践” (Baidu’s live-stream messaging service in practice, 百度Geek说, 2021; repost on 52im) states the goal most clearly: at most N messages a second per room, delivered to a million viewers within S seconds, with N ≤ 20 and S ≤ 2. It also says that one screen usually shows no more than 10–20 messages, and that at more than 20 a second the screen never stops scrolling. This chapter’s numbers are of the same scale.
1. Run the trunk’s group design
When the trunk sends a group message (chapter 32’s path), it does these steps for every recipient:
- Who gets it: message N goes to the members whose rangemembership range成员区间The stretch of seqs a person can see in a group: min_seq is the first, max_seq the last (unset while they are still in it). Joins and leaves take a seq in the conversation like messages, so who gets each message and how far back each person can read are worked out from this range, whenever they are worked out. The group has a pair too: max_seq is the latest seq, and below min_seq nobody sees anything (chapter 12).See the glossary holds N (chapter 12).
- Conversation recordconversation record会话记录One row per person per conversation: pinned, muted, the conversation’s latest seq, and later how far they have read. Written on join, leave or a settings change, and from chapter 11 on also when a message arrives (with the latest seq). Each change bumps that person’s record version. The messages are not here.See the glossary: one row per member, stamped with a new version (chapter 11).
- Look up the online routesonline routes在线路由表Which gateway each online device is connected to. Gateways register devices as they connect, and dispatch looks gateways up here.See the glossary: which gateway this person’s devices are connected to (chapter 20).
- Push: one copy per device (chapter 2).
- Pull: a device that sees a skipped number pulls the missing messages (chapter 5).
- Read position: for a chat on screen, reported at most once a second (chapter 14).
With 1 message a second in the room, each step is 1,000,000 operations a second. But a hot room has 100 a second: chat, gifts, “so-and-so joined”, likes. The first four steps are done for every message, so:
100 messages a second × 1,000,000 people = 100,000,000 a second for each step, 719 times v3’s whole-site peak of 139,000 deliveries a second
- Conversation records: 100,000,000 small transactions a second, each writing two rows (chapter 11): 200,000,000 rows. At 20,000 rows a second per storage node, that is 10,000 nodes.
- Online routes: 100,000,000 lookups a second, 1,079 times chapter 32’s whole-site peak (92,700 a second).
- Pushes: at 50,000 pushes a second per gatewaygateway网关A connection-layer server. Each holds many long-lived connections and handles auth, heartbeats and traffic without reading messages. One of the few stateful parts: a restart drops all its connections.See the glossary, 2,000 gateways.
Joins and leaves are worse. Chapter 12 decided that a join or a leave also takes a seqseq序号(seq)The number of each message in a conversation, assigned in order by the message service that owns it. It sets the order, lets a client see which message is missing, and tells a returning phone which conversations it is behind on.See the glossary: it is a system message to the whole group. Assume 2,000 joins and 2,000 leaves a second (Baidu’s article says that one room’s peak is under 20,000 joins a second, and under 20,000 leaves). That is 4,000 messages a second, each to 1,000,000 people: 4,000,000,000 deliveries a second, 40 times the room’s actual chat.
Each of these steps is useful in the hiking group. A live room is simply a different product.
2. First, drop what is not needed
Three of these a live room does not need at all:
- Read position: there is no unread count, and nobody cares who has read what.
- Pulls to fill gaps: what is lost stays lost. The trunk’s phone pulls as soon as it sees a skipped number; a live room is allowed to drop messages (section 6 will even drop most of them on purpose), so 1,000,000 phones see gaps every second: 1,000,000 pulls a second.
- Conversation record: a live room is not on the chat list, and leaving ends it.
Public products choose the same: Tencent Cloud IM’s “live-stream group” has no member limit, no unread count and no stored history, and by default receives only online pushes (group system); join and leave notices are still sent, but not stored. In this chapter, joining and leaving no longer take a seq or get stored either; “so-and-so joined” is at most a hint that can be lost.
Three steps are left: who is in the room, where they are connected, pushing to them. Baidu’s article also counts six steps, drops three, and keeps exactly these three (its six include “find the user’s devices”, which our route table answers in the same lookup; ours include the conversation record).
But these three are still 100,000,000 a second, so they have to be done another way.
3. Move fan-out into the gateways
Copying one message many times to reach many people is called fan-out. In the trunk, fan-out happens behind the gateways: the dispatch layer (the services behind the gateways that deliver messages) finds each person and pushes to each one.
Notice something about the three steps: “who is in the room” and “where are they connected” both end in finding connections, one by one. And those connections already live on the gateways.
So let the gateways keep track: a room is a group of connections on each gateway.
- Joining: the phone sends a frame “join room R” on its own connection. The gateway puts this connection into its own “connections in R”. If it is the first viewer of R on this gateway, it tells the room routerroom router房间路由The routing for live rooms: for each room it knows only which gateways have viewers, not who they are. A room’s message goes to it, it sends one frame to each of those gateways, and each pushes to the viewers in its own group.See the glossary: “I have viewers of R.”
- Leaving, or disconnecting: the gateway takes the connection out of the group; if the group is now empty, it tells the room router. After a reconnect, the phone must join the room again.
- Sending: the room’s message server (one owner per conversation, chapter 28) hands the message to the room router. The router knows only “which gateways have viewers of R”; v3 has 20 gateways in all (chapter 22), so that is at most 20 frames. Each gateway pushes to the connections in its group.
1,000,000 viewers spread over those 20 gateways is 50,000 on each. Behind the gateways, nothing is kept “once per viewer” any more:
- each message is 20 frames behind the gateways; at 100 a second, 2,000 frames;
- no online-route lookups, no conversation records;
- with 2,000 joins a second, the router hardly changes: only when the first viewer arrives on a gateway, or the last one leaves.
Baidu’s article calls this multicast (mcast): the group’s membership is not stored separately but kept on each long-connection instance, and the multicast route is a set of long-connection instances. In its article every join goes through the business layer, which updates the multicast route and the instance’s list of connections; changing the route only when a gateway’s first viewer arrives or its last one leaves is this chapter’s version. Twitch’s public description points the same way; see section 11.
But one thing has not changed: each gateway still pushes every message 50,000 times. At 100 a second that is 5,000,000 pushes a second, 100 times its capacity. Added up, it is still 100,000,000 a second and 2,000 gateways. Fan-out has moved; it has not shrunk.
4. One batch a second
Viewers don’t care whether each message arrives on its own. So pack a second’s messages into one batchbatching打包In a live room, a second’s messages go out as one batch, so each connection gets one push a second, compressed. Pushes no longer grow with the room’s message rate; the cost is half a second’s wait per message on average.See the glossary, and push each connection once a second:
Pushes: 100,000,000 a second → 1,000,000 a second. 1,000,000 ÷ 50,000 per gateway = 20 gateways
The cost is delay: a message waits half a second in the batch on average, and at most 1 second (demo.py simulates 100,000 messages: 0.50 s on average, 99% within 0.99 s). That leaves room under the 2-second goal.
20 gateways are exactly full, with nothing to spare. For a real event you would plan at 80%, which is 25 gateways; big streams usually need capacity added in advance.
Some messages cannot wait half a second, such as “the stream has ended”. They can be sent at once, in a batch of their own. But know the price: every extra batch means 50,000 more pushes on each gateway, a whole second of its push capacity. So only a few rare kinds may jump the queue.
5. Bytes: compress, then cap
The number of pushes is solved; the bytes are not. A live-room message is much bigger than a chat message: it carries the sender’s nickname, avatar, level and badges, and a gift carries its effect settings. Baidu’s article assumes 2 KB on average, while the trunk’s chat message is 200 bytes. This chapter uses 2 KB.
One gateway, pushing all 100 messages a second to 50,000 people:
50,000 × 100 messages × 2 KB × 8 = 80 Gbit/s
A network card is 10 Gbit/s. Once batched, a second’s messages share the same field names and format, so they compress well. Baidu measured a ratio of 6.7 : 1, about 15% of the original size. Where to compress is this chapter’s own choice: the same batch goes to all 50,000 people on this gateway, so it can be compressed once, in the application, and the compressed bytes sent to everyone; with WebSocket’s own compression, each connection compresses separately, 50,000 times. But 15% of 80 is still 12 Gbit/s, more than the card.
So there is also a cap: at most 20 messages a second per viewer. That is all a screen can hold; more cannot be read anyway.
50,000 × 20 messages × 2 KB × 8 = 16 Gbit/s, or 2.4 Gbit/s compressed: 24% of the card
On the phone: 20 messages a second is 6 KB a second compressed (48 kbit/s), 21.6 MB for an hour of watching; without the cap, 108 MB.
6. Which 20 to keep: priority
100 messages come in each second, and each viewer gets 20. Which ones? By priority:
- Control messages: the stream has ended, the video switches, someone is muted. Few, and they must arrive.
- Gifts: someone paid, and everyone wants to see them.
- What the streamer and the moderators say.
- Ordinary chat.
- “So-and-so joined” and likes: the most numerous and the least important. With 2,000 joins a second, showing them all would fill 100 screens; show a few, and turn the likes into one number.
Say there are 5 gifts a second, all kept. That leaves 15 picked from the other 95: a viewer sees 16% of what others say, about one message in six. The viewer’s own message is always seen: the phone shows it on its own screen as soon as it is sent.
This is what Tencent Cloud IM does in chapter 34’s “other answers”: drop what is over the limit without telling the sender. The trunk did not choose it: in the hiking group, Ana would think her message had gone out. In a live room, nobody minds missing one more “here!” from someone else.
7. Gifts take another, reliable path
Gifts are different. The sender paid money, and the streamer must see it at once and say thank you. An ordinary viewer missing one gift animation is fine; the streamer missing one is an incident.
So a gift goes out twice:
- The copy for all viewers rides in the batches above, at high priority, and can still be lost.
- The copy for the streamer takes the trunk’s old path: stored with a seq and pushed to the streamer; the streamer’s app pulls when it sees a skipped number (chapter 5). If the connection breaks, it polls with plain HTTP requests. There is one recipient, so 5 gifts a second is 5 writes a second: no strain at all for the trunk’s design.
The gift itself is an order, recorded first by the business’s payment system; the message is only a notice. Baidu’s article says they opened a separate, reliable multicast channel for gifts, with push plus pull on the streamer’s side, and that gift messages reach their target more than 99.9% of the time. WeChat’s live chat room met the same problem: a lost co-hosting signal broke co-hosting, and a lost big-gift animation brought complaints, so important messages were marked with a priority and clients fetch the important ones first (WeChat live chat room).
8. The room mailbox: what a newcomer sees
Someone who just entered the room must not see an empty screen; they should see the last screenful. With 2,000 people joining a second, each wanting the last 20 messages:
2,000 × 20 messages × 2 KB = 80 MB a second, 12 MB compressed, and all of it the same 20 messages
So each room gets a mailbox: recent messages in time order. Everything worth keeping (chat, gifts) is written into it, and it can be paged back by count or by time. Baidu’s article does the same: each multicast group gets a shared mailbox (mcast-mbox) that takes users’ messages and anything to be kept, and history is fetched by time range and count. Keeping the last screenful in memory is this chapter’s addition.
This chapter saves one step more: every gateway already receives each of the room’s batches, so let it keep the last 20 itself. Each gateway sees only 100 joins a second (2,000 ÷ 20); it answers them directly, without asking anything behind it.
9. The trunk’s decisions: keep, change, drop
| Trunk decision | In a live room |
|---|---|
| The gateway holds connections and knows no business (chapter 20) | Change: the gateway keeps one more thing, each room’s group of connections |
| Online routes find the gateway by person and device (chapter 20) | Change: the room router knows only “which gateways have viewers” |
| Accounts and tokens (chapter 25) | Keep: a viewer who has not logged in gets a guest identity that may only watch this room |
| Heartbeats (chapter 23) | Change: Baidu’s article says that after joining a multicast group the heartbeat becomes shorter and adjustable by the server, to notice a broken connection sooner, reconnect, and join the room again |
| One owner per conversation (chapter 28) | Keep: the room’s owner does the batching, the cap and the priority |
| seq and gap filling (chapter 5) | Drop, except for gifts |
| Conversation records, read position, offline push (chapters 11, 14, 17) | Drop |
| A join or leave is a message (chapter 12) | Drop: joining and leaving change only a gateway’s group of connections |
| Limits (chapter 34) | Change: 20 messages a second per viewer; over that, dropped by priority without telling the sender |
10. The cost
- The gateway now knows some business logic. Chapter 20 said gateways know no business logic and are rarely redeployed. Now they keep rooms’ groups of connections, and batch and compress. Keep the room’s rules behind them, so the gateway knows only “groups” and “batches”.
- Half a second of delay. Fine for a live stream, wrong for chat.
- Most messages are seen by nobody. In a busy room, a viewer sees one in six of what others say.
- A disconnect loses messages. Whatever passed during the few seconds of reconnecting is gone; on return, the viewer sees only the last screenful from the mailbox.
- The viewer count is an estimate. Each gateway knows how many are in its group; adding them up takes a few seconds, so “1,000,000 watching” on screen is not exact. WeChat built a separate system for its online count and for “how many have watched”.
- One owner per room. All comments pass through one message server. At chapter 28’s 5,000 messages a second per server, a few hundred comments a second are fine; any hotter, and taking in comments has to be split too.
- Two paths. Ordinary messages and gifts follow two sets of logic, to be tested and watched separately.
11. Other answers
- Pull instead of push. WeChat’s live chat room uses read fan-outread fan-out读扩散Keep one copy in the conversation’s timeline, and each reader fetches it from there. Few writes, more reads; suits big groups. All of v1’s and v2’s messages are kept this way (chapters 6, 10); it needs the server to build the chat list and devices to sync it.See the glossary and long polling: each room stores its messages once, and phones pull them. It does not push over WebSocket because, it says, push can lose messages and needs pull as a fallback anyway, and push needs an exact list of who is online at every moment. Its cache keeps only the last 2,000 messages; when a room is busy all the time, requests pile up, so the polling interval grows with the number of people online, and in a test with 4,000,000 online, requests fell by 58% (WeChat live chat room).
- Fan-out in layers. Twitch’s 2015 engineering overview says its chat Edge receives and distributes messages between clients and backend services, and that Pubsub and Edge “form a hierarchical message distribution system which executes massive fanout”; another service aggregates room membership across all the edges (Twitch Engineering).
- The server tells the client how often to come. YouTube’s live chat API returns a
pollingIntervalMilliswith each reply, telling the client how long to wait before polling again; the first request returns only the most recent messages. It also offersstreamList, a connection kept open while the server pushes (liveChatMessages). - Send to every server. Chapter 20’s Socket.IO Redis adapter: keep no list of which servers have viewers; send every message to all of them. Simplest when there are few servers.
- A fixed limit per group. Tencent Cloud IM allows 40 messages a second per group; over that, high-priority messages go first and the rest are not delivered (chapter 34).
Further reading: the author’s post Group Chat vs. Live Stream Messaging: A Comparison, which compares group chat and live-stream messaging point by point.
12. This chapter’s decision
A live room trades “every message must arrive” for “fast enough”. The next chapter goes the other way: a private messenger wants the server to know as little as possible, not even what a message says.
Comments
Comments live in GitHub Discussions and need a GitHub account.