Chapter 26
App lifecycle
Ben walks out of the office and his phone switches from Wi-Fi to 4G. The app is open, yet Ana’s messages arrive two minutes late. On a phone, who decides when the connection should close and when it should reconnect?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
At 6 in the evening Ben walks out of the office reading his chat with Ana, and his phone switches from the office Wi-Fi to 4G. Ana sends three messages in a row; nothing appears on his screen. Two minutes later, all three pop up at once.
The server did nothing wrong. All three messages were pushed on time, into Ben’s old connection on the Wi-Fi. That connection had stopped working, but Ben’s app did not know and kept waiting.
Here is the problem: on a phone, whether a connection is alive is decided not by the app, still less by the server, but by the phone’s OS. In this chapter the app listens to the OS, and passes what it hears on to the server.
1. How phone OSes treat apps in the background
The two systems have different rules, but the same direction: an app in the background should do as little as possible.
iOS. When an app goes to the background, the system calls one of its callbacks and gives it 5 seconds to wrap up; shortly after the callback returns, the app is suspended: it runs no code at all (Apple docs). For a little more time it can request a background task, which it must end as soon as it is done, or the system terminates the app. While the app is suspended, the system may also reclaim its sockets, which closes the connection (TN2277, archived).
Android. An app in the background still has “several minutes” to run background services, after which the system stops them (Android 8 background limits). A phone that is unplugged, with the screen off and left still for a while, enters Doze, a low-power mode: the system suspends network access and only lets it through in maintenance windows that come further and further apart (Doze docs). Many phone makers also clean up background apps more aggressively; dontkillmyapp.com lists them by maker (a community site, not official documentation).
The upshot is simple: once an app leaves the screen, its long-lived connectionlong-lived connection长连接A TCP or WebSocket connection a device keeps open to the server. With it, the server can push a message at any time instead of waiting to be asked.See the glossary cannot be relied on. The earlier chapters already counted this way: “online” means the app is open, and that is where v3’s peak of a million online comes from.
2. Watch it break: the network changed, the connection didn’t notice
A TCP connection is tied to the addresses at both ends. When Ben’s phone changes network, its address changes and the old connection can no longer be used.
Sometimes the OS makes the old socket report an error at once, and the app knows straight away. But not always: Android’s docs say that once a new network becomes the default, the remaining connections on the old network are forcefully terminated only “at some point later” (docs). Until then, the app writes into a connection that no longer works and sees no error at all.
An app that does not listen to the OS can only find out through chapter 23’s heartbeat: wait for the next heartbeat (chapter 23 sends one after 4 minutes with nothing sent, so 0 to 4 minutes later), then 10 more seconds without a reply, then reconnect. 130 seconds on average, 250 at worst, and all that time Ben’s screen is on and no new messages arrive.
The OS does know the network changed, and tells the app: Android has a callback for changes to the default network (same docs), and iOS has NWPathMonitor (docs). Listen for that event and reconnect as soon as the network changes:
the event (assumed within 1 second) + a new connection, 0.6 seconds (chapter 2: one round trip each for TCP, TLS 1.3 and the upgrade) + one round trip to sync, 0.2 seconds = about 1.8 seconds
3. Going to the background: keep the connection or close it?
Chapter 17 already settled half of this: when the app goes to the background it tells the server, and from then on the server sends this device an OS pushOS push系统推送A notification sent through Apple’s APNs, Google’s FCM or a phone maker’s channel to a phone with no live connection. A third party, best effort, with no guarantee.See the glossary. The other half is the connection itself.
Keeping it is the natural choice: Ana switches away to copy a verification code and is back in seconds; the connection is still there and nothing needs reconnecting. But on iOS the app is suspended a few seconds later while the connection stays open: the phone’s OS kernel receives packets and acknowledges them on its behalf, and the server sees nothing wrong. It has to wait for chapter 23’s check: the gateway closes it only 12 minutes after it last heard from it. In the foreground the app sends something at least every 4 minutes, so when it goes to the background its last send was 0 to 4 minutes earlier; take that from the 12 minutes, and after the app is suspended the connection hangs on for another 8 to 12 minutes, 10 on average.
v3 has 10 million people a day, each opening the app 20 times, and every open ends with the app going to the background. Taking the peak as 3 times the average, that is about 6,944 a second. 6,944 new ones a second, each hanging on for 600 seconds, means the number hanging at any moment is the two multiplied:
6,944 × 600 seconds ≈ 4.17 million unused connections hanging open, more than the million really in use
The OS may reclaim the socket earlier, so this is an upper bound; but even a part of it lands on gateways that chapter 22 sized for only a million connections. On Android the app stays alive in the background for a while; the connection is alive too, and keeps sending heartbeats and using battery until the OS stops the app.
So close it. The price of closing is a new connection every time the app returns to the foreground. The choice here is to keep it 10 seconds, then close: whoever comes back within 10 seconds needs no reconnect, and at peak the connections left hanging are only 6,944 × 10 seconds’ worth, about 70,000. 10 seconds fits within iOS’s 5 seconds plus a background task.
4. The fix: a state machine driven by OS events
Both problems are the same flaw: the connection looks only at its own socket, not at the OS. The fix is to write the connection as a state machine, with both OS events and connection events as its inputs.
| State | What it does | Events that leave it |
|---|---|---|
| Disconnected | Nothing | Back to the foreground → Connecting |
| Connecting | TCP, TLS, with the token (chapter 25) | Success → Syncing; failure → Waiting to retry |
| Syncing | Catches up on conversations and messages as in chapters 6 and 11 | Done → Online |
| Online | Receives pushes, sends heartbeats | To the background → Background; network changed, heartbeat timed out → Connecting; server closed it → Waiting to retry |
| Waiting to retry | Backoff with random jitter (chapter 3; so that when a gateway restarts, not everyone comes back at the same moment, chapter 24) | Time is up, network is back → Connecting |
| Background | Has reported going to the background; closes in 10 seconds | Back to the foreground → Online; 10 seconds pass → Disconnected |
Four rules:
- When the network changes, reconnect at once. Don’t wait for the old connection to report an error; drop it and connect on the new network. No backoff the first time; only if that fails, back off with jitter as in chapter 3. Telegram’s client library TDLib asks for the same: call
setNetworkTypewhenever the network changes, even if its type stays the same (say, a different Wi-Fi), and it reopens all network connections to cut the delay of switching networks (TDLib docs). - Going to the background: report first, then close. In the few seconds the OS gives, report “background” as in chapter 17, and close the connection 10 seconds later. If that frame does not get out (say there happens to be no network), the server can only fall back on heartbeats, back to 8 to 12 minutes, and messages in that time get no notification.
- Back in the foreground: connect, then sync. Chapter 6’s “sync once on every open” becomes, here, “connect once and sync once every time the app returns to the foreground”. Whoever comes back within 10 seconds still has the connection, and just reports “foreground”.
- Killed or crashed: nothing to do. When the process is gone, the OS kernel closes its sockets, and the server usually receives the close; if it doesn’t, there are still heartbeats.
5. The numbers
- Noticing the network changed: from 130 seconds on average to about 1.8 seconds.
- Unused connections hanging open: at peak, from up to about 4.17 million to about 70,000.
- The price is more connections: a new connection every time the app returns to the foreground, about 6,944 handshakes a second at peak. Spread over chapter 22’s 20 gateways, that is about 347 a second each; each can do 2,000 a second, so this takes 17%. This counts connections, one per open; chapter 21 counts presence changes, where quick visits are merged away, so its peak is only about 3,500 online-or-offline changes a second.
6. Cost
- Every open means connecting and syncing. The handshakes use the gateways’ CPU (the 17% above), and opening the app takes about a second longer.
- The rules vary by platform, version and maker. iOS and Android each have their own, and on Android it also depends on the maker; this state machine has to change as the OSes are upgraded.
- In the background, everything rests on OS push. Once the connection is closed, background messages can only come through APNs, FCM and phone makers’ channels (chapter 17), and none of them guarantees delivery. A notification is only a nudge; the sync on returning to the foreground is the source of truth.
7. Other answers
- Keep the connection in the background too. Android allows a foreground service, at the price of a notification that stays in the notification bar, and of heartbeats all the time: one every 4 minutes, each waking the radio for 10 seconds, keeps the radio on about 1 hour more a day. iOS does not let ordinary chat apps do this.
- Wake the app with a VoIP push. Since iOS 13, an app that receives a VoIP push must report an incoming call to CallKit, or it is terminated (PushKit docs). It only suits real calls.
- Sync in the background with silent pushes. Apple says these pushes are low priority and throttled, and not to send “more than two or three per hour” (docs); Ana receives about 267 messages a day on average (the series’ number), 11 an hour, far more than that.
- QUIC connection migration. QUIC identifies a connection by a connection ID rather than by addresses, so when the phone changes network the connection keeps working (RFC 9000 section 9); a server can also declare that it does not support this. It only solves network switches, not the background.
8. This chapter’s decision
On a phone, the app at least has OS push to fall back on. In a browser, Ana opens three tabs at once, and once she closes them nothing can wake her. Next: the web client.
Comments
Comments live in GitHub Discussions and need a GitHub account.