Chapter 17
Waking an offline phone
Ben’s app is closed and its connection long gone. Ana @mentions him in the hiking group; the message waits in the conversation. How does he find out?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
Ben finishes dinner and puts his phone face down on the table. In the hiking group, Ana writes “@Ben did you buy the train tickets for tomorrow morning?”.
The message is stored in the conversation, and Ben’s conversation record (one row per person per conversation, chapter 11) is updated. But Ben’s app is not in the foreground, and its connection dropped long ago. Chapter 2’s push goes over the long-lived connection; with no connection there is nothing to push over, and he won’t know until he next opens the app himself.
What this chapter decides: when the app is not in the foreground, how to let Ben know there is a message, without a thousand messages a day driving him mad.
1. Only the OS can wake it
Soon after an app goes to the background, the OS suspends it, and its connection stops with it. The only thing that can get a message to the phone then is the OS’s own push service, which stays connected all the time:
- iPhone: Apple’s APNs.
- Android: Google’s FCM. But in mainland China most Android phones have no Google services (Alibaba Cloud Mobile Push’s explanation), so FCM can’t be used.
- Android in mainland China: Huawei, Honor, Xiaomi, OPPO and vivo each have their own service (the phone makers’ push services), and each has to be integrated separately.
The flow is the same everywhere: when the app starts, it asks the OS for a push token (device token) and hands it to the server, which stores it in chapter 16’s devices table. To notify that device, the server takes the token to the matching service and asks it to send. In Telegram’s API, the client hands over its token with account.registerDevice and says which service it belongs to; the list has 13 kinds, including APNs, FCM, Huawei push and web push. The notification arrives, the OS shows it, and the app doesn’t need to wake. This is an OS pushOS push系统推送A notification sent through Apple’s APNs, Google’s FCM or a phone maker’s channel to a phone with no live connection. A third party, best effort, with no guarantee.See the glossary, a different thing from the online pushonline push在线推送A push through a gateway, over the connection, to an online device. It is a hint: if it is lost, the device catches up by pulling.See the glossary over the long-lived connection.
2. Which devices to notify
“Devices with no connection” sounds like the answer, but it isn’t. Just after the app goes to the background, its connection is still in the server’s routing table (which connection each of a person’s devices is on, chapter 2): the phone is already suspended, and the server only notices when the heartbeat times out (chapter 23). Judged by “is there a connection”, the messages of those few minutes are pushed into a connection nobody reads, and no notification goes out.
So what counts is foreground or background: the app tells the server whenever it goes to the background or comes back to the foreground. A device that has reported background, or whose connection has dropped (a killed app has no time to report), is not in the foreground, and gets an OS push.
With several devices there is one more question: Ana is reading on her laptop; should her phone sound? This chapter suggests not: if any of her devices is in the foreground, no notification is sent. The badge on the phone is corrected by the next notification, or when the app is opened.
3. One notification per message
The most direct way: for each message, send a notification to every device not in the foreground.
The hiking group has 1,000 messages a day, so each member gets 1,000 notifications. It is not just noisy; the services can’t take it either:
- They can’t store that many. While a phone is off or has no network, the service holds notifications for it. APNs keeps just one per app; FCM keeps at most 100 on Android, and beyond that drops them all, so the phone comes back to a single “too many” message (FCM, APNs).
- They throttle. Send to the same token too fast and APNs answers 429
TooManyRequests, “Too many requests were made consecutively to the same device token” (APNs error codes). - Messages must be classified. The phone makers’ services split messages into “communication” and “marketing”: instant messaging counts as communication, with no limit on numbers, but you have to apply to each maker for it; without that, it counts as marketing, with only one to five a day per device, depending on the maker (Alibaba Cloud’s summary of each maker’s rules).
At the v2 peak: 139 messages a second, each reaching 10 devices on average.
At peak only one in ten people has the app open; assuming the same for devices, the other nine in ten all need a notification: 139 × 10 × 90%, about 1,251 a second, before counting muted chats or combining.
4. The fix: one notification per conversation
A notification is not a copy of the message; it is a nudge: “new messages in the hiking group”. So combine them by conversation:
- One per conversation on screen. A new notification for the same conversation replaces the old one: APNs uses
apns-collapse-id(at most 64 bytes), FCM uses the notification’stag(field reference), both set to the conversation ID. The text reads “Hiking group: Cara said… (12 messages)”. - At most once a minute per conversation. The server keeps, for each device and conversation, when it last sent; anything arriving within the minute waits until the minute is up and goes out combined. A busy hour is at most 60.
- Muted conversations send nothing, and don’t count towards the badge; if someone @mentions you, it still sends (chapter 14 keeps unread @mentions).
A day in the hiking group, with chat coming in bursts (demo.py: 20 bursts of 50 messages, one every half minute on average), combines 1,000 messages into 406 notifications. 60% fewer, but still a lot. A busy group can’t be brought down by combining alone; in the end it is up to the user to mute it.
Why not use the service’s own combining? FCM has a collapse_key too, but a device keeps at most 4 different keys (same page). With one key per conversation, those run out as soon as someone has more than a few conversations. So combining happens on the server, and replacing happens on the phone.
5. The badge
The number on the app icon is the sum of the unread counts of Ben’s conversations that aren’t muted. The phone is suspended and can’t work it out, so only the server can put it in the notification: APNs’s badge field (payload).
Chapter 14 already has the server work out each conversation’s unread count. But adding up Ben’s 200 conversations for every notification means reading 1,251 × 200, about 250,000 rows a second at peak.
The fix is to keep the total on Ben’s row in the users table. In chapters 11 and 14, every change to a conversation record locks this row to take a version number; in the same transaction, add the difference between the record’s unread count before and after to the total. Muting or unmuting a conversation takes its unread count off or adds it back.
The timing has to be right too. In chapter 11 the online push goes out as soon as the message is stored; the conversation records change a little later, through the to-do (a row written when the message is sent, saying “update every member’s conversation”, later done one small transaction per person). If the notification went out with the online push, the total would not include this message yet, and the badge would be one short. So the to-do sends the notification: once Ben’s row is updated, the new total is in hand; then it checks whether he has a device in the foreground, and if none does, it sends.
This is not the kind of counter chapter 14 rejected. That one adds one per message: if a to-do dies halfway and runs again, the same message is added twice. This one adds a difference: Ben has 5 unread in the hiking group; the to-do writes the new latest seq and he has 6, so the total goes up by 1. When the to-do runs again, the row already says 6 and still says 6 after the write; the difference is 0 and the total stays put.
6. A notification is only a nudge
Notifications come with no guarantee: the service may be late or lose them, and the user may have turned notifications off. As in chapter 2, a push is only a nudge; the pull is what counts:
- A notification doesn’t move the cursor. The sync cursor on the phone moves only with sync (chapter 6).
- After opening, sync as usual. The app comes to the foreground, connects, and syncs by record version; messages are pulled from the conversation, not taken from the notification.
- Tokens expire. When a token is no longer valid, APNs answers 410, meaning stop sending here (APNs error codes); the server deletes it from the devices table.
7. The numbers
- One per message: 1,000 notifications a day for each person in the hiking group. A phone switched off for a day: FCM fills up at 100 and drops them all.
- Combined by conversation: at most 60 an hour; for
demo.py’s day, 406. - Peak: about 1,251 notifications a second to send, before counting muted chats or combining.
- Badge: adding up on the spot is about 250,000 rows a second; kept on the users row and sent by the to-do, there are no extra reads.
8. The cost
- Several services to integrate. APNs, FCM, and several phone makers in mainland China, each with its own tokens, format and rules, plus applying for message classification.
- Neither delivery nor speed is guaranteed. Low-priority notifications may be held and delivered together (APNs); if FCM sees that high-priority messages keep producing no notification, it may lower them to normal priority (FCM).
- Read on another device, the notification stays on this one. Clearing it quietly on an iPhone takes a background push, and Apple advises no more than two or three of those an hour (background updates). The trunk doesn’t clear it, and lets the next notification put the badge right.
- A little later than an online push. The notification waits until the to-do reaches that person.
- The server keeps more. Each device’s token and foreground state, the last notification time per device per conversation, and an unread total per person.
9. Other answers
- The message text in the notification, or just “new message”. With the text, users can read it without opening the app; with just “new message”, the content doesn’t pass through Apple’s, Google’s or the phone makers’ servers. Products with end-to-end encryption (chapter 42) have another way: the notification carries only ciphertext, and on an iPhone the app’s notification extension (
mutable-content) rewrites the content on the phone before it is shown. Telegram’s API shows one more way: the template for a group message carries the sender, the group name and the text, and with FCM the client can hand over a key when it registers its token, so the notification content is encrypted before it goes to the service (push notifications). - Send only background pushes, and let the app fetch and show the messages itself. The app decides everything that is shown, but background pushes are low priority and the OS limits how many there are.
- One notification per message. For a small product with few messages it is the simplest, and it never hits the services’ limits.
- Your own always-on connection. The app keeps its own connection in the background, common on Android in the early years. Today’s OSes limit what apps can run in the background, so this option keeps shrinking.
10. This chapter’s decision
Ben knows there is a message now. He opens it, and Ana’s line has become “This message was recalled”. Yet the notification is still on his lock screen, his tablet already synced the original, and the other members’ devices hold hundreds more copies. Next: recall and edit.
Comments
Comments live in GitHub Discussions and need a GitHub account.