IM Systems in Depth

Chapter 16

Many devices

Ana replies to Ben from her phone, yet her laptop, open on the desk, doesn’t show the reply. She buys a new tablet and logs in for the first time. How much chat history should it download?

This chapter is a first draft. The author is still checking it, and its text and numbers may change.

Ana is at her laptop, the desktop app open on her chat with Ben. She picks up her phone and replies to Ben: “See you downstairs at seven.” She looks up at the laptop: the reply isn’t there.

Twenty minutes later Ben replies “OK”. Only then do both lines appear on the laptop at once: hers and Ben’s.

What this chapter decides: when a person has several devices, how each one gets the messages it should have, including her own sent from another device; and how much a new device takes on its first login. Keeping the read position and unread counts the same on every device was settled in chapter 14.

1. First, what does the product need?

Approach Example What the server does
One device per account Logging in on a new device logs the old one out One connection per person in the routing table
One device of each kind One phone, one computer; a second of the same kind logs the first out Logs out by device type at login
Several, each on its own The phone is off, the laptop still sends and receives Every device has to get every message
Several, end-to-end encrypted WhatsApp’s multi-device The sender encrypts a copy for each device (chapter 42)

The trunk is the third kind: Ana has a phone and a laptop, 1.5 devices per person on average across the series, and each device connects to the server and catches up by itself, without relying on another device being online.

2. Coming back from offline: already solved

Start with the device that was offline. Ana’s laptop was closed all night. When she opens it in the morning, how does it know what it missed?

Since chapter 6, each device keeps its own sync cursorsync cursor同步游标What a device keeps as “I have everything up to here”. In this series it is two things: the conversation-record version, and each conversation’s last seq. A device that comes back asks what changed after that version and each conversation’s latest seq, then pulls what it is behind on.See the glossary: a record versionrecord version记录版本A per-person number that goes up with each change to their conversation records; it is taken under a lock on that person’s users row, so version order is commit order. The phone keeps the version it has and asks “what changed after this” on return; a group it was added to, or a chat with new messages, shows up here.See the glossary (a number that goes up each time one of her conversation records changes), plus the last seqseq序号(seq)The number of each message in a conversation, assigned in order by the message service that owns it. It sets the order, lets a client see which message is missing, and tells a returning phone which conversations it is behind on.See the glossary of each conversation (each message’s number in the conversation, counting up without gaps). The laptop asks with its own version, “what changed after this?”, then pulls the conversations it is behind on; the phone asks with the phone’s. The server keeps no “how far has it got” per device.

Ana’s own messages are part of this. In chapter 11, for every message the dispatch layer moves the conversation record of every member to the new latest seq, the sender included. Ana’s version goes up, so at its next sync the laptop sees that conversation is behind and pulls back the line she sent from her phone.

So “one cursor per device” was already there in v1.

3. The online moment: the sender is left out

The problem comes while the laptop is open. An online device doesn’t rely on sync; it relies on pushes (chapter 2). And chapter 2’s push was written as: “find each of Ben’s connections and write the message to them.” It pushes to the recipients, not the sender, because the sender already has the message.

With one device, that is right. With two, Ana’s laptop didn’t send this message, yet it is skipped because it belongs to the sender. It has to wait until:

  • the conversation’s next message is pushed, the seq jumps, and it pulls the missing ones in between (chapter 5);
  • or the next sync (reconnecting, coming back to the foreground).

The twenty minutes at the start were the wait for Ben’s next line. In a quiet conversation, the wait can be long.

This is not a new problem. The open protocol XMPP has an extension written just for it, Message Carbons: a user switches devices mid-conversation, and part of the conversation is left behind on the other one. Its answer is to copy every message the person sends or receives to their other online devices.

4. The fix: push to every device, skipping only the connection it came from

The dispatch rule becomes:

  • Push to every online device of every member of the conversation, including the sender’s own other devices;
  • Skip only the connection the message came from. That phone already has the message, and its seq comes back with the ACK (chapter 5).

Before, it was “the members, minus the sender”; now it is “every connection of every member, minus the one it came from”. The routing table was already “user → a set of connections” (chapter 2), so it doesn’t change.

When the laptop receives Ana’s own message, it writes it to the local database as usual; the sender is Ana, so it shows on the right. (sender_id, msg_id) is unique in the local database (chapter 7), so a message that arrives once by push and once by sync is stored only once.

The numbers:

  • One message in the hiking group has to reach 500 people × 1.5 devices = 750 devices: online ones by push, offline ones by syncing when they come back.
  • The extra: with 1.5 devices per person on average, minus the one that sent it, each message goes to 0.5 more of the sender’s devices on average. At the v2 peak of 139 messages a second, that is about 70 more pushes a second, 5% more than the 1,390 deliveries a second before (139 messages, each reaching 10 devices on average).

5. A new device: how deep on the first pull?

Ana buys a tablet and logs in for the first time. It has nothing; its record version is 0.

Chapter 10 set the minimum: take every conversation recordconversation record会话记录One row per person per conversation: pinned, muted, the conversation’s latest seq, and later how far they have read. Written on join, leave or a settings change, and from chapter 11 on also when a message arrives (with the latest seq). Each change bumps that person’s record version. The messages are not here.See the glossary and the newest message of each conversation, which is enough to sort the chat list. A record now carries chapter 14’s read position and unread count, 39 bytes; a message is about 200 bytes:

  • Ana (2,000 conversations): about 478 KB.
  • An average person (200 conversations): about 48 KB.

But with only the newest message, opening a conversation on the tablet shows a single line, and scrolling up waits for the network. Some depths to choose from:

How deep Ana’s tablet Opening a conversation
The newest message of each conversation (chapter 10) 478 KB The first open waits one round trip
Plus one page (50 messages) for each of the 20 conversations on the first screen 200 KB more The ones she uses most are there at once
All history 267 received and 40 sent a day (chapter 7): about 110,000 messages a year, at 200 bytes each 22 MB of text; 67 MB for three years Everything, searchable on the device

Photos aren’t included: as in chapter 8, the full image downloads only when opened.

The trunk suggests the second. The rest is marked as a “gap”, as in chapter 7 (a local marker: this stretch is still on the server), and filled in when the conversation is opened or scrolled up to it. Full history is not pulled at login.

Telegram’s API splits it the same way: the conversation list comes a page at a time from messages.getDialogs, each page carrying the last message of each of its conversations (messages.dialogsSlice); opening one and scrolling up calls messages.getHistory for a page before a given message. The difference is that even the conversation list is paged there; this chapter takes it all at once.

The cost is that search on a new device is incomplete: the device has only what it has pulled. Full search means either searching on the server or slowly pulling history in the background. The trunk leaves it for now; products that keep years of history and need search are chapter 39’s workplace chat.

6. The cost

  • About 5% more pushes per message, to the sender’s own other devices.
  • Devices have to be managed. The server needs a devices table: which devices are logged in, and on which platform. Logging out, or removing a device from another one, deletes its row and closes its connection; chapter 17 also keeps the device’s push token here. Telegram’s API shows such a table: account.getAuthorizations lists every logged-in session with its device model, platform, app version, last active time and IP (authorization), and account.resetAuthorization logs one of them out.
  • A new device starts shallow. Scrolling up waits for the network, and local search is incomplete.
  • Each device syncs on its own. Every message and every read is pulled once by each device; the server’s reads grow with the number of devices, not people.

7. Other answers

  • Allow only one device. The simplest: one connection per person in the routing table, and a new login logs out the old one. It is a product limit, not a technical one.
  • An inbox per device. The server keeps a queue for each device and deletes each message once it is taken. Devices don’t affect each other; the cost is one write per device for each message: one message in the hiking group is 750 writes.
  • Deliver to one device only. XMPP’s core specification lets the server hand a message sent to a person to just their “most available” device, such as the one with the highest priority (RFC 6121 §8.5.2.1). Later, Carbons copied messages to the other online devices, and server-side message archives (XEP-0313) let every device catch up on history.
  • The sender sends a copy to each device. WhatsApp’s multi-device (2021): messages are end-to-end encrypted and the server can’t decrypt them, so the sending client encrypts and sends once for each device on both sides. When a new device connects, the phone packs the recent conversations’ messages into an encrypted bundle and sends it over. Up to 4 devices besides the phone. Chapter 42 has the details.
  • A few messages per room on the first sync. Matrix’s first sync can use a filter to limit how many messages each room returns; a timeline that isn’t complete is marked limited, and prev_batch pages back (syncing). The same kind of answer as this chapter’s “newest message of each conversation, the rest later”.

8. This chapter’s decision

Every open device now gets the message. But the app on Ben’s phone is closed, its connection long gone; the message waits in the conversation, and he doesn’t know. Next: waking an offline phone.

Comments

Comments live in GitHub Discussions and need a GitHub account.