# Chapter 23: Heartbeats

> Ben’s app is open and the connection looks fine, yet none of Ana’s messages arrive. When did the connection die, and who should notice first?

IM Systems in Depth · https://im.liko.page/en/heartbeats/

It is evening and the hiking group is quiet. Ben’s app is open and the connection icon is green. Ana asks in the group what time they meet tomorrow, and Ben’s phone stays silent. Over twenty minutes later Ben posts something himself, and only then do several messages pop up at once.

Nothing ever said the connection had closed, yet it had stopped working long before. This chapter answers how a long-lived connection can die without a sound, and who notices it, and how soon.

## 1. Why an open connection dies

Between the phone and the server sits the carrier’s **NAT** (network address translation). The phone has an address inside the carrier’s private network; the NAT swaps it for a public address and port, and remembers the pairing. Packets the server pushes down can only find the phone through that entry.

NAT entries have an idle timeout: if no packet passes for a while, the entry is deleted. After that:

- **The server** writes a push into the connection; the packet reaches the NAT, finds no owner, and is dropped. TCP retransmits, and Linux by default keeps retransmitting for 13 to 30 minutes before giving up (`tcp_retries2` in [tcp(7)](https://man7.org/linux/man-pages/man7/tcp.7.html)). All that time, Ben is still online in the online routes (chapter 20: which gateway each device is connected to); and the OS push of [chapter 17](/en/offline-push/) goes only to devices that are not online, so it is not sent either.
- **The phone** receives nothing and sees no error. It only learns the connection is gone when it sends something and that fails (it gets a “connection reset”, or never gets an answer).

TCP has its own keepalive, but it is off by default, and even when on it starts probing only after 2 hours idle ([RFC 1122](https://www.rfc-editor.org/rfc/rfc1122) requires a default of at least 2 hours; Linux defaults to 7,200 seconds). So “TCP is reliable” does not help here: what is reliable is that data gets retransmitted, not that the connection stays up.

## 2. How long the network allows

The standard is generous: [RFC 5382](https://www.rfc-editor.org/rfc/rfc5382) requires a NAT to keep an idle TCP connection for at least 2 hours and 4 minutes, just long enough for TCP’s default keepalive.

Real networks are much shorter:

- A 2011 measurement ([Wang et al., SIGCOMM 2011](https://web.eecs.umich.edu/~zmao/Papers/netpiculet.pdf)) of 73 carriers’ 3G networks found 11 under 10 minutes, 4 of them at 5 minutes or less.
- In the same measurement, the firewall of a large US carrier sent a forged “reset” after 255 seconds idle, cutting the connection.
- The WeChat team measured several networks between late 2013 and early 2014 ([article](http://www.52im.net/thread-120-1-1.html), original by WeMobileDev, reposted by 52im): China Mobile 2G/3G and China Unicom 2G at 5 minutes, China Telecom 3G over 28 minutes.

All of these are measurements of 2G/3G networks more than ten years ago, and say nothing about today’s 4G/5G. Newer evidence is only personal experience: in 2020 a person running his own XMPP chat server wrote that on a French mobile network he had to bring TCP keepalives down to 270 seconds (4.5 minutes); “Anything more and the NAT gateway would kill the TCP session” ([blog](https://blog.wirelessmoves.com/2020/09/carrier-grade-nat-timeouts-and-how-to-configure-your-xmpp-server.html)). That is one network, not a measurement. Besides, nobody can tell the phone what this network’s timeout is, and it differs from network to network. This series assumes **5 minutes**.

## 3. Heartbeats

The fix is to never let the connection stay quiet too long: a **heartbeat**.

1. **The client sends it.** When the client has sent nothing into the connection for 4 minutes, it sends a tiny heartbeat frame, and the gateway answers. Any frame the client sends counts: if Ben sends a message, the clock starts over. Frames it receives do not count: the gateway can only know the client is alive by hearing from it (section 4).
2. **The server sets the interval.** On connect, the gateway puts the interval in its first frame to the client. Changing it then needs no new app release.

Why the client? The gateway only has to answer, instead of scheduling sends for each of a million connections; and the phone knows best which network it is on and whether the app is in the foreground or background.

Why not WebSocket’s own Ping frame ([RFC 6455](https://www.rfc-editor.org/rfc/rfc6455#section-5.5.2), which the other side must answer with a Pong)? Because the browser’s WebSocket API has only `send` and `close` ([MDN](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket)); a web page cannot send a Ping. An ordinary application frame works the same on every client.

Below is a quiet hour of Ben’s: he receives 15 messages and sends 6, and the NAT forgets the connection after 5 minutes without a packet. Four settings, one run each:

- **No heartbeat**: 9 of the 15 fall into the NAT; the most delayed is 26.4 minutes late, waiting until Ben sends something himself, finds the connection dead, reconnects and syncs.
- **Every 6 minutes**, a little longer than the NAT: only 1 is dropped, but the connection dies 3 times, each time needing a reconnect, a handshake and a sync.
- **Every 4 minutes**: none dropped, 11 heartbeats.
- **Every 30 seconds**: none dropped, 116 heartbeats.

Over 200 quiet hours: with no heartbeat, 36% of pushes fall into the NAT, and 191 of the 200 hours have some; every 4 minutes, none are dropped, at 12.2 heartbeats an hour on average (an hour has 15 four-minute slots; Ben’s own messages stand in for a few).

The model in the figure makes a few assumptions: any passing packet refreshes the NAT entry; on a forgotten connection, pushed packets are silently dropped, while Ben sending anything finds out at once that the connection is dead; and the sync after the phone reconnects (chapters 6 and 11) brings back everything missed. So dropped messages are not lost, only late, and without a notification.

## 4. Finding dead connections

Heartbeats do a second job: when the connection really breaks (Ben goes into the subway, the phone runs out of battery), both sides should learn it quickly.

- **The client**: if a heartbeat gets no reply in 10 seconds, it closes the connection, reconnects and syncs. While the app is in the foreground, it knows within 4 minutes 10 seconds at most.
- **The gateway**: if a connection receives nothing for **3 intervals** (12 minutes), it closes it and removes this device of Ben’s from the online routes. Messages for him then go by chapter 17’s OS push.

Why does the gateway wait 3 intervals, not 1? In the background, the OS delays the app’s timers; on a poor network, a heartbeat may need several retransmissions to arrive. Close too early, and a phone that is alive has to reconnect for nothing. The price of waiting long: for those 12 minutes Ben is still “online” in the routes, and the roughly 6.7 messages sent to him at peak (267 a day per person, 3 times that rate at peak, over 12 minutes) get no notification.

On a gateway, a million connections do not each need a timer: each connection records only “the latest time I must hear from it”, put into time buckets, and scanning one bucket a second is enough (a timing wheel).

## 5. The numbers

- **Server**: a million connections, one heartbeat every 4 minutes, is about **4,170** a second; across 20 gateways, 208 each a second, about 0.4% of a gateway’s push capacity. Even every 30 seconds is only 3.3%. Nearly every connection pays this: only a client that sends something itself saves a heartbeat, and most people receive more than they send.
- **Phone**: the cost of heartbeats is battery. As chapter 2 said, each time the radio wakes it stays up about 10 seconds before going back to low power (a measurement published in 2012 found [11.6 seconds](https://web.eecs.umich.edu/~zmao/Papers/Thesis/JunxianHuangThesis2013.pdf) on a US LTE network).
- A phone doing nothing but keeping the connection: one heartbeat every 4 minutes is 360 a day, each keeping the radio up about 10 seconds, so **1 hour** a day; every 30 seconds is 2,880 a day, **8 hours**.

This battery bill only counts while the screen is off and the connection still open: in the foreground, the screen uses far more power than the radio. iOS does not let apps keep background connections, and most Android phones stop them too (chapter 26); so the phone’s OS keeps one connection itself and all apps’ notifications share it. That is APNs and FCM. Where FCM is unavailable, Android apps keep their own background connections and this bill is theirs; that is exactly the situation the WeChat article is about.

## 6. Cost

- **Battery and detection speed trade against each other.** Halve the interval and the radio is up twice as long; lengthen it and dead connections are found later.
- **5 minutes is a guess.** On a network whose timeout is shorter than 4 minutes, connections still die; each time, the next heartbeat finds out and the client reconnects.
- **There will be false alarms.** On a poor network replies are slow, and reconnecting after 10 seconds without one sometimes drops a connection that was still alive.

## 7. Other answers

- **Find the interval per network**: WeChat’s smart heartbeat. In the foreground it uses a fixed interval; in the background it probes per network (mobile data by type, Wi-Fi by name): after 3 short heartbeats in a row succeed, it tries a longer one; if that works it lengthens again, if it fails it steps back, and it settles a little below the last interval that worked. Once a week it probes upward again. In the staged rollout, 70% of users reached the maximum interval ([article](http://www.52im.net/thread-120-1-1.html)).
- **The server sets the interval, with a random first beat**: Discord’s gateway gives a `heartbeat_interval` in its first frame (Hello), and the client waits `heartbeat_interval * jitter`, “where jitter is any random value between 0 and 1”, before the first heartbeat; if a heartbeat is not acknowledged, it treats the connection as “zombied”, closes it, reconnects and resumes the session ([Discord gateway docs](https://docs.discord.com/developers/events/gateway)). The docs say the random first beat is there so that not too many clients reconnect at the same moment and cause a burst of traffic (chapter 24).
- **The server waits only 1.5 intervals**: MQTT 3.1.1 has the client send PINGREQ when idle, and the server disconnects after 1.5 keep-alive periods with no packet at all ([§3.1.2.10](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html)). With 4 minutes that is 6 minutes: faster than 12, with more false alarms.
- **The client tells the server how long to wait**: Telegram’s MTProto has a ping, `ping_delay_disconnect`, that carries a `disconnect_delay`: the server starts a timer when it arrives and closes the connection if the next one has not come within that many seconds. The docs’ example is a ping every 60 seconds with a 75-second delay ([MTProto service messages](https://core.telegram.org/mtproto/service_messages)). The client sets the wait, and it is only 1.25 intervals; in this chapter the gateway sets it, and waits 3 intervals.
- **TCP keepalive**: between servers, it works once the interval is shortened; but it only proves the other side’s OS kernel is alive. If the process is stuck, the kernel still answers.
- **Lengthen the timeouts of the boxes in between**: if the load balancer or proxy is your own, set its timeout longer. For example, nginx proxying WebSocket cuts the connection after 60 seconds without data by default ([nginx docs](https://nginx.org/en/docs/http/websocket.html)). The carrier’s NAT cannot be changed. In the 2011 measurement, the carrier that cut connections at 255 seconds gave 1,600 seconds to one port only: 5228, used by Google’s push.

## 8. This chapter’s decision

**Decision card**

- Problem: When a connection stays quiet too long, a NAT quietly forgets it: pushes fall into nothing, the server thinks the device is online and sends no OS push, and the phone thinks it is still connected. TCP keepalive starts only after 2 hours.
- Choice: The client sends a heartbeat when it has sent nothing for 4 minutes (shorter than the assumed 5-minute NAT timeout); any frame it sends counts, and the server sets the interval on connect. No reply in 10 seconds: reconnect and sync. A gateway that hears nothing for 3 intervals (12 minutes) closes the connection, deletes the route and switches to OS push.
- Cost: About 4,170 heartbeats a second on the server, under 1% of gateway capacity; a phone waking only to keep the connection keeps its radio up about 1 hour a day; a dead connection can take the gateway 12 minutes to find; the NAT timeout is a guess.
- Revisit when: Many devices dropping and returning at once (chapter 24); the app going to the background or switching networks (chapter 26); timers in the browser (chapter 27).
- Other answers: A smart heartbeat that adapts per network (WeChat); a server-set interval with a random first beat (Discord); a 1.5× timeout (MQTT); a client-set wait (Telegram); TCP keepalive; longer timeouts on your own proxies.

Heartbeats let one phone know it has dropped off. But when a whole availability zone’s gateways stop together, 500,000 phones learn it in the same second, and come back in the same second. Next: reconnect storms.
