Chapter 21
Presence
Ben comes online. Of his 200 contacts, whom should we tell?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
Ana opens her chat with Ben. Under his name it says “online”, or “last seen 19:42”. That line is presence.
The previous chapter’s online routesonline routes在线路由表Which gateway each online device is connected to. Gateways register devices as they connect, and dispatch looks gateways up here.See the glossary table already knows whether Ben has a connection. The hard part is not knowing; it is whom to tell. Ben has 200 contacts. Each time he comes online or goes offline, do all of them need to know at once?
1. First, what does the product need?
| What the product shows | What the server does |
|---|---|
| Nothing | Nothing |
| Only a rough idea, such as “seen recently” | Keeps a last-seen time and looks it up when a chat is opened; no pushes |
| “Online” or “last seen 19:42” in an open chat, and a green dot in the contact list | When someone is watching, tells them at once when it changes (this chapter) |
| Every person’s status across the whole list: online, away, do not disturb | The workplace approach, with big lists and more states (chapter 39) |
Telegram’s API has both kinds: an exact “online” and a last-seen time, and rough ones like “last seen recently”, “last seen last week” and “last seen last month” (UserStatus). When the exact time cannot be shown (for example because of privacy settings), it gives the rough kind (userStatusRecently).
The trunk (the one product this series grows from v0 to v3) picks the third tier: online or last seen, in the chat header and in the contact list.
2. What counts as a change
On a phone, “online” means the app is open and connected. Once it goes to the background, the connection is closed 10 seconds later (chapter 26).
Ana opens the app 20 times a day; each visit is two changes, one online and one offline, so 40 in all. Many visits are quick: she replies to one message and switches straight back. If going offline counts only after a short wait, those round trips merge away. The series assumes 10 changes per person per day after merging.
With 10 million daily users at 10 a day, that is about 1,160 changes a second on average, online or offline; taking the peak as 3 times the average, about 3,500 a second.
3. The first idea: tell every contact
The most direct way, and the way XMPP, the standard instant-messaging protocol, does it: when someone’s status changes, the server sends it to all their contacts; contacts who are offline get nothing, and receive everyone’s current status in one go when they come online (RFC 6121, sections 4.2–4.4).
With our numbers:
- For each change, about 10% of the 200 contacts are online, so it goes to 20 people.
- At peak, 3,500 changes a second × 20 = about 69,000 pushes a second. At v3’s peak, message pushes are 139,000 a second; presence pushes alone come to half that number.
Most of these pushes are seen by nobody. Ana is chatting with Cara; her other contacts come and go, and the pushes still reach her phone: 69,000 a second spread over 1 million online devices is about 250 an hour each.
And it grows with the number of contacts. Someone with 2,000 contacts pushes to 200 people each time they come or go, 10 times as many as an ordinary user.
4. Tell only those who are watching
Turn it around: instead of Ben changing and the server working out who should know, Ana says first “these are the people I am looking at.”
- The device reports whom it is watching. The phone sends the server the whole list of people on its screen (Ben in the chat header, the dozen or so rows showing in the contact list). When the screen changes it sends a new list, which replaces the old one entirely.
- The server first answers with the current status. For each person on the list, online now or last seen at what time, in one reply.
- After that, it pushes only on a change. Ben comes online; the server looks up who is watching Ben, and pushes only to those devices.
This chapter assumes an average of 10 people on each online device’s screen. With 1 million devices online, that is 10 million “who is watching whom” entries. Spread over 10 million daily users, each person is watched by exactly one device on average, so on average each change is pushed once.
- Pushes: from about 69,000 a second down to about 3,500, 20 times fewer.
- But the lists have to be sent too: a new one each time the app or a chat is opened. At 20 app opens and 60 chat opens a day (chapter 7’s assumptions), that is 800 million lists a day across 10 million users, about 9,260 a second on average, and about 27,800 lists a second at peak (3 times the average). Each one gets a reply with current statuses; at 10 people a list, about 280,000 status lookups a second.
Together, frames sent to devices drop from about 69,000 a second to about 31,000, a little more than half saved. The biggest saving is that it no longer grows with the number of contacts: someone with 2,000 contacts coming or going is pushed only to the one or two devices watching them.
Slack does the same: the client sends presence_sub with the full set of users to subscribe to each time, and the subscription lasts only as long as that WebSocket connection; the docs suggest at most 500 users (Slack docs).
5. Subscriptions live with the connection
“Who is watching whom” is state that has to be kept. Where, and how is it not lost?
- Split across a few presence servers by the person being watched. One server is in charge of Ben: how many connections he has, when he was last seen, and who is watching him.
- A subscription lives only on its connection. When the connection closes, the gateway tells the services behind it (chapter 20), and all of that device’s subscriptions are deleted; after reconnecting, the phone sends the list on its screen again. Since each list is the whole list, sending it again adds nothing and loses nothing.
- Size: 10 million subscriptions at 16 bytes each, plus an index to find them by device, about 320 MB, spread across a few machines.
6. One thing to get right: wait before going offline
When a gateway restarts (chapter 24), the devices on it (50,000 normally, on the machine assumed in chapter 22) disconnect together and connect back seconds to tens of seconds later. Each one goes “offline, online” once:
- Telling every online contact: 100,000 changes, each pushed to 20 people, 2 million pushes in all.
- Telling only those watching: about 100,000.
Better still is for them not to count as changes at all: when a connection closes, wait a while first; if it reconnects within that time, it never went offline. Wait 30 seconds, say, and most devices are back after a gateway restart, with no push at all. Section 2’s “reply to one message and switch straight back” round trips merge away the same way.
LinkedIn’s presence works like this: if a connection drops and reconnects within d seconds, it stays online as long as the heartbeats resume; each online person has a delayed trigger (LinkedIn Engineering blog, 2018).
It is late the other way too. A connection can also die silently: the phone goes into a lift and loses signal, and the gateway never receives any “disconnect”. The gateway can only find out through heartbeats: the phone sends something at least every 4 minutes (a heartbeat when it has nothing else to send), and only after 12 minutes (3 intervals) with nothing heard does the gateway count the connection as dead (chapter 23). For those 12 minutes, Ben still shows as “online”. A whole gateway dying is worse: it never gets to tell the services behind it that its connections are gone. Its devices are updated only when they connect to another gateway; for those that never come back, something else has to void everything listed under that gateway (chapter 20).
7. The cost
- Subscriptions are state to maintain. Every screen change sends a list, about 27,800 a second at peak, more than the pushes themselves.
- Status is late. Going offline shows only after a wait; when a connection dies silently, “online” lingers for over ten minutes.
- People not watching do not know. While Ana is not looking at Ben, she does not learn when he comes or goes; it is looked up when she opens his chat. That is the point, but features like “tell me when a friend comes online” have to be built separately.
- Who may see whom must be checked first. A subscription has to ask the business layer whether Ben has hidden his status from Ana.
8. Other answers
- Tell every online contact (XMPP, section 3). Simplest when contacts are few and every status is wanted; the cost follows the number of contacts.
- Fetch once on open, refresh on a timer while watching. No subscriptions to keep; refreshing every 30 seconds is about 33,000 requests a second, and a change shows up to 30 seconds late.
- “Online” as a lease that expires. Telegram’s
userStatusOnlinecarriesexpires, “Time to expiration of the current online status”; clients report online or offline withaccount.updateStatus(userStatusOnline, account.updateStatus). Past that time the “online” no longer counts, without waiting for a heartbeat timeout. - Shard by person, push only what is visible. Slack hashes users onto its presence servers, and a client gets notices only about the users it can see (Real-time Messaging).
- Show nothing. The cheapest answer.
9. This chapter’s decision
Presence and pushes both rest on the gateways’ connections. How many connections should one gateway hold? Next: a million connections.
Comments
Comments live in GitHub Discussions and need a GitHub account.