# Chapter 41: Live rooms

> A live stream puts 1,000,000 people in one room. Thousands join and leave every second, and the screen has space for only a dozen or so comments. Can the trunk's group design cope? If not, what has to change?

IM Systems in Depth · https://im.liko.page/en/live-rooms/

8 p.m., and a product launch goes live. 1,000,000 people enter the same live room, as many as the whole v3 app has online at peak. Comments scroll across the bottom of the video: someone asks the price, someone types “here!”, someone sends a rocket gift.

The trunk's hiking group has 500 members; every message must arrive, and a member who comes back must be able to catch up. A live room is the other way round:

| | Hiking group | Live room |
|---|---|---|
| People | 500 | 1,000,000 |
| Joining and leaving | a few times a day | thousands of times a second |
| Every message must arrive | yes | no; losing some is fine |
| Arriving late | still must arrive, even seconds late | useless: the video has moved on |
| Unread, read, history | yes | no; a newcomer only needs to see the last screenful |

This chapter's question: **both are “one message to many people”. Where does the trunk's group design break in front of 1,000,000 people, and what has to change?**

Among public sources, “百度直播消息服务架构实践” (Baidu's live-stream messaging service in practice, 百度Geek说, 2021; [repost on 52im](http://www.52im.net/thread-3515-1-1.html)) states the goal most clearly: at most N messages a second per room, delivered to a million viewers within S seconds, with N ≤ 20 and S ≤ 2. It also says that one screen usually shows no more than 10–20 messages, and that at more than 20 a second the screen never stops scrolling. This chapter's numbers are of the same scale.

## 1. Run the trunk's group design

When the trunk sends a group message (chapter 32's path), it does these steps for every recipient:

1. **Who gets it**: message N goes to the members whose range holds N (chapter 12).
2. **Conversation record**: one row per member, stamped with a new version (chapter 11).
3. **Look up the online routes**: which gateway this person's devices are connected to (chapter 20).
4. **Push**: one copy per device (chapter 2).
5. **Pull**: a device that sees a skipped number pulls the missing messages (chapter 5).
6. **Read position**: for a chat on screen, reported at most once a second (chapter 14).

With 1 message a second in the room, each step is 1,000,000 operations a second. But a hot room has 100 a second: chat, gifts, “so-and-so joined”, likes. The first four steps are done for every message, so:

> 100 messages a second × 1,000,000 people = **100,000,000 a second for each step**, 719 times v3's whole-site peak of 139,000 deliveries a second

- Conversation records: 100,000,000 small transactions a second, each writing two rows (chapter 11): 200,000,000 rows. At 20,000 rows a second per storage node, that is 10,000 nodes.
- Online routes: 100,000,000 lookups a second, 1,079 times chapter 32's whole-site peak (92,700 a second).
- Pushes: at 50,000 pushes a second per gateway, 2,000 gateways.

Joins and leaves are worse. Chapter 12 decided that a join or a leave also takes a seq: it is a system message to the whole group. Assume 2,000 joins and 2,000 leaves a second (Baidu's article says that one room's peak is under 20,000 joins a second, and under 20,000 leaves). That is 4,000 messages a second, each to 1,000,000 people: 4,000,000,000 deliveries a second, 40 times the room's actual chat.

Each of these steps is useful in the hiking group. A live room is simply a different product.

## 2. First, drop what is not needed

Three of these a live room does not need at all:

- **Read position**: there is no unread count, and nobody cares who has read what.
- **Pulls to fill gaps**: what is lost stays lost. The trunk's phone pulls as soon as it sees a skipped number; a live room is allowed to drop messages (section 6 will even drop most of them on purpose), so 1,000,000 phones see gaps every second: 1,000,000 pulls a second.
- **Conversation record**: a live room is not on the chat list, and leaving ends it.

Public products choose the same: Tencent Cloud IM's “live-stream group” has no member limit, no unread count and no stored history, and by default receives only online pushes ([group system](https://cloud.tencent.com/document/product/269/1502)); join and leave notices are still sent, but not stored. In this chapter, joining and leaving no longer take a seq or get stored either; “so-and-so joined” is at most a hint that can be lost.

Three steps are left: **who is in the room, where they are connected, pushing to them**. Baidu's article also counts six steps, drops three, and keeps exactly these three (its six include “find the user's devices”, which our route table answers in the same lookup; ours include the conversation record).

But these three are still 100,000,000 a second, so they have to be done another way.

## 3. Move fan-out into the gateways

Copying one message many times to reach many people is called **fan-out**. In the trunk, fan-out happens behind the gateways: the dispatch layer (the services behind the gateways that deliver messages) finds each person and pushes to each one.

Notice something about the three steps: “who is in the room” and “where are they connected” both end in finding connections, one by one. And those connections already live on the gateways.

So let the gateways keep track: **a room is a group of connections on each gateway.**

- **Joining**: the phone sends a frame “join room R” on its own connection. The gateway puts this connection into its own “connections in R”. If it is the first viewer of R on this gateway, it tells the **room router**: “I have viewers of R.”
- **Leaving, or disconnecting**: the gateway takes the connection out of the group; if the group is now empty, it tells the room router. After a reconnect, the phone must join the room again.
- **Sending**: the room's message server (one owner per conversation, chapter 28) hands the message to the room router. The router knows only “which gateways have viewers of R”; v3 has 20 gateways in all (chapter 22), so that is at most 20 frames. Each gateway pushes to the connections in its group.

1,000,000 viewers spread over those 20 gateways is 50,000 on each. Behind the gateways, nothing is kept “once per viewer” any more:

- each message is 20 frames behind the gateways; at 100 a second, 2,000 frames;
- no online-route lookups, no conversation records;
- with 2,000 joins a second, the router hardly changes: only when the first viewer arrives on a gateway, or the last one leaves.

Baidu's article calls this multicast (mcast): the group's membership is not stored separately but kept on each long-connection instance, and the multicast route is a set of long-connection instances. In its article every join goes through the business layer, which updates the multicast route and the instance's list of connections; changing the route only when a gateway's first viewer arrives or its last one leaves is this chapter's version. Twitch's public description points the same way; see section 11.

But one thing has not changed: **each gateway still pushes every message 50,000 times.** At 100 a second that is 5,000,000 pushes a second, 100 times its capacity. Added up, it is still 100,000,000 a second and 2,000 gateways. Fan-out has moved; it has not shrunk.

## 4. One batch a second

Viewers don't care whether each message arrives on its own. So pack a second's messages into one batch, and push each connection once a second:

> Pushes: 100,000,000 a second → **1,000,000 a second**. 1,000,000 ÷ 50,000 per gateway = **20 gateways**

The cost is delay: a message waits half a second in the batch on average, and at most 1 second (demo.py simulates 100,000 messages: 0.50 s on average, 99% within 0.99 s). That leaves room under the 2-second goal.

20 gateways are exactly full, with nothing to spare. For a real event you would plan at 80%, which is 25 gateways; big streams usually need capacity added in advance.

Some messages cannot wait half a second, such as “the stream has ended”. They can be sent at once, in a batch of their own. But know the price: every extra batch means 50,000 more pushes on each gateway, a whole second of its push capacity. So only a few rare kinds may jump the queue.

## 5. Bytes: compress, then cap

The number of pushes is solved; the bytes are not. A live-room message is much bigger than a chat message: it carries the sender's nickname, avatar, level and badges, and a gift carries its effect settings. Baidu's article assumes 2 KB on average, while the trunk's chat message is 200 bytes. This chapter uses 2 KB.

One gateway, pushing all 100 messages a second to 50,000 people:

> 50,000 × 100 messages × 2 KB × 8 = **80 Gbit/s**

A network card is 10 Gbit/s. Once batched, a second's messages share the same field names and format, so they compress well. Baidu measured a ratio of 6.7 : 1, about 15% of the original size. Where to compress is this chapter's own choice: the same batch goes to all 50,000 people on this gateway, so it can be compressed once, in the application, and the compressed bytes sent to everyone; with WebSocket's own compression, each connection compresses separately, 50,000 times. But 15% of 80 is still 12 Gbit/s, more than the card.

So there is also a cap: **at most 20 messages a second per viewer.** That is all a screen can hold; more cannot be read anyway.

> 50,000 × 20 messages × 2 KB × 8 = 16 Gbit/s, or **2.4 Gbit/s** compressed: 24% of the card

On the phone: 20 messages a second is 6 KB a second compressed (48 kbit/s), 21.6 MB for an hour of watching; without the cap, 108 MB.

## 6. Which 20 to keep: priority

100 messages come in each second, and each viewer gets 20. Which ones? By priority:

1. **Control messages**: the stream has ended, the video switches, someone is muted. Few, and they must arrive.
2. **Gifts**: someone paid, and everyone wants to see them.
3. **What the streamer and the moderators say.**
4. **Ordinary chat.**
5. **“So-and-so joined” and likes**: the most numerous and the least important. With 2,000 joins a second, showing them all would fill 100 screens; show a few, and turn the likes into one number.

Say there are 5 gifts a second, all kept. That leaves 15 picked from the other 95: a viewer sees 16% of what others say, about one message in six. The viewer's own message is always seen: the phone shows it on its own screen as soon as it is sent.

This is what Tencent Cloud IM does in chapter 34's “other answers”: drop what is over the limit without telling the sender. The trunk did not choose it: in the hiking group, Ana would think her message had gone out. In a live room, nobody minds missing one more “here!” from someone else.

## 7. Gifts take another, reliable path

Gifts are different. The sender paid money, and the streamer must see it at once and say thank you. An ordinary viewer missing one gift animation is fine; the streamer missing one is an incident.

So a gift goes out twice:

- **The copy for all viewers** rides in the batches above, at high priority, and can still be lost.
- **The copy for the streamer** takes the trunk's old path: stored with a seq and pushed to the streamer; the streamer's app pulls when it sees a skipped number (chapter 5). If the connection breaks, it polls with plain HTTP requests. There is one recipient, so 5 gifts a second is 5 writes a second: no strain at all for the trunk's design.

The gift itself is an order, recorded first by the business's payment system; the message is only a notice. Baidu's article says they opened a separate, reliable multicast channel for gifts, with push plus pull on the streamer's side, and that gift messages reach their target more than 99.9% of the time. WeChat's live chat room met the same problem: a lost co-hosting signal broke co-hosting, and a lost big-gift animation brought complaints, so important messages were marked with a priority and clients fetch the important ones first ([WeChat live chat room](http://www.52im.net/thread-3376-1-1.html)).

## 8. The room mailbox: what a newcomer sees

Someone who just entered the room must not see an empty screen; they should see the last screenful. With 2,000 people joining a second, each wanting the last 20 messages:

> 2,000 × 20 messages × 2 KB = 80 MB a second, 12 MB compressed, and all of it the same 20 messages

So each room gets a **mailbox**: recent messages in time order. Everything worth keeping (chat, gifts) is written into it, and it can be paged back by count or by time. Baidu's article does the same: each multicast group gets a shared mailbox (mcast-mbox) that takes users' messages and anything to be kept, and history is fetched by time range and count. Keeping the last screenful in memory is this chapter's addition.

This chapter saves one step more: every gateway already receives each of the room's batches, so let it keep the last 20 itself. Each gateway sees only 100 joins a second (2,000 ÷ 20); it answers them directly, without asking anything behind it.

## 9. The trunk's decisions: keep, change, drop

| Trunk decision | In a live room |
|---|---|
| The gateway holds connections and knows no business (chapter 20) | **Change**: the gateway keeps one more thing, each room's group of connections |
| Online routes find the gateway by person and device (chapter 20) | **Change**: the room router knows only “which gateways have viewers” |
| Accounts and tokens (chapter 25) | **Keep**: a viewer who has not logged in gets a guest identity that may only watch this room |
| Heartbeats (chapter 23) | **Change**: Baidu's article says that after joining a multicast group the heartbeat becomes shorter and adjustable by the server, to notice a broken connection sooner, reconnect, and join the room again |
| One owner per conversation (chapter 28) | **Keep**: the room's owner does the batching, the cap and the priority |
| seq and gap filling (chapter 5) | **Drop**, except for gifts |
| Conversation records, read position, offline push (chapters 11, 14, 17) | **Drop** |
| A join or leave is a message (chapter 12) | **Drop**: joining and leaving change only a gateway's group of connections |
| Limits (chapter 34) | **Change**: 20 messages a second per viewer; over that, dropped by priority without telling the sender |

## 10. The cost

- **The gateway now knows some business logic.** Chapter 20 said gateways know no business logic and are rarely redeployed. Now they keep rooms' groups of connections, and batch and compress. Keep the room's rules behind them, so the gateway knows only “groups” and “batches”.
- **Half a second of delay.** Fine for a live stream, wrong for chat.
- **Most messages are seen by nobody.** In a busy room, a viewer sees one in six of what others say.
- **A disconnect loses messages.** Whatever passed during the few seconds of reconnecting is gone; on return, the viewer sees only the last screenful from the mailbox.
- **The viewer count is an estimate.** Each gateway knows how many are in its group; adding them up takes a few seconds, so “1,000,000 watching” on screen is not exact. WeChat built a separate system for its online count and for “how many have watched”.
- **One owner per room.** All comments pass through one message server. At chapter 28's 5,000 messages a second per server, a few hundred comments a second are fine; any hotter, and taking in comments has to be split too.
- **Two paths.** Ordinary messages and gifts follow two sets of logic, to be tested and watched separately.

## 11. Other answers

- **Pull instead of push.** WeChat's live chat room uses read fan-out and long polling: each room stores its messages once, and phones pull them. It does not push over WebSocket because, it says, push can lose messages and needs pull as a fallback anyway, and push needs an exact list of who is online at every moment. Its cache keeps only the last 2,000 messages; when a room is busy all the time, requests pile up, so the polling interval grows with the number of people online, and in a test with 4,000,000 online, requests fell by 58% ([WeChat live chat room](http://www.52im.net/thread-3376-1-1.html)).
- **Fan-out in layers.** Twitch's 2015 engineering overview says its chat Edge receives and distributes messages between clients and backend services, and that Pubsub and Edge “form a hierarchical message distribution system which executes massive fanout”; another service aggregates room membership across all the edges ([Twitch Engineering](https://blog.twitch.tv/en/2015/12/18/twitch-engineering-an-introduction-and-overview-a23917b71a25)).
- **The server tells the client how often to come.** YouTube's live chat API returns a `pollingIntervalMillis` with each reply, telling the client how long to wait before polling again; the first request returns only the most recent messages. It also offers `streamList`, a connection kept open while the server pushes ([liveChatMessages](https://developers.google.com/youtube/v3/live/docs/liveChatMessages/list)).
- **Send to every server.** Chapter 20's Socket.IO Redis adapter: keep no list of which servers have viewers; send every message to all of them. Simplest when there are few servers.
- **A fixed limit per group.** Tencent Cloud IM allows 40 messages a second per group; over that, high-priority messages go first and the rest are not delivered (chapter 34).

Further reading: the author's post [Group Chat vs. Live Stream Messaging: A Comparison](https://blog.liko.page/2024/07/11/Group-Chat-vs-Live-Stream-Messaging-A-Comparison/), which compares group chat and live-stream messaging point by point.

## 12. This chapter's decision

**Decision card**

- Problem: 1,000,000 people in one room, 100 messages a second, thousands joining and leaving. The trunk's group design does six steps per message for every person, the first four each 100,000,000 a second; joins and leaves are messages too, 4,000,000,000 deliveries a second.
- Choice: Drop read position, gap filling and conversation records; move fan-out into the gateways: each gateway keeps the room's connections it holds, and a room router knows only which gateways have viewers. The room's owner packs one batch a second, at most 20 messages per viewer chosen by priority, and pushes it compressed. Gifts take a separate reliable path with seq to the streamer; a room mailbox gives newcomers the last screenful.
- Cost: Gateways know about rooms; 0.5 s of delay on average; in a busy room a viewer sees one in six comments; messages during a disconnect are lost; the viewer count is an estimate; one owner per room; two message paths.
- Revisit when: When one room's comments are more than one message server can take in; when a live room needs replay or chat search; when the same long-connection layer also serves group chat and notifications (chapter 44).
- Other answers: Read fan-out with long polling and an interval set by the number online (WeChat live chat room); layered fan-out with Edge and Pubsub (Twitch); a polling interval given by the server (YouTube live chat); send to every server (Socket.IO); a fixed limit per group (Tencent Cloud IM).

A live room trades “every message must arrive” for “fast enough”. The next chapter goes the other way: a private messenger wants the server to know as little as possible, not even what a message says.
