Side trip W3
TCP or UDP
On TCP, one lost packet holds up every message behind it. How much slower does that make chat? When should you use UDP or QUIC?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
Chapter 22 put a million TCP long-lived connectionslong-lived connection长连接A TCP or WebSocket connection a device keeps open to the server. With it, the server can push a message at any time instead of waiting to be asked.See the glossary on 20 gatewaysgateway网关A connection-layer server. Each holds many long-lived connections and handles auth, heartbeats and traffic without reading messages. One of the few stateful parts: a restart drops all its connections.See the glossary. People often ask: when TCP loses one packet, everything behind it waits; why not UDP, or QUICQUICA transport protocol on top of UDP that does its own resending, encryption and congestion control; HTTP/3 runs on it. One connection carries many independent streams, each in order within itself.See the glossary, which runs on UDP?
Chapter 2 also left three things for this page: some networks block UDP; QUIC’s connection migrationconnection migration连接迁移QUIC identifies a connection by the connection ID in each packet, not by address and port, so when a phone moves from Wi-Fi to 4G the connection carries on. The load balancer must route by connection ID too. The trunk does not use it: a network change means a reconnect and a sync.See the glossary; WebSocket over HTTP/3. This side trip first works out how much slower this TCP habit makes chat, then looks at who really uses UDP and QUIC, and finally lists what a chat connection would need before moving over.
First, two terms. UDP just sends datagrams one by one: it does not resend what is lost and does not keep order. QUIC is a transport protocol on top of UDP that does its own resending, encryption and congestion control; HTTP/3 runs on it. One QUIC connection can carry many independent streams, each in order within itself.
1. One lost packet holds up everything behind it
TCP promises to hand bytes to the application in order. Say the server sends Ben three messages in a row; the packet with the first one is lost, and the packets with the second and third reach the phone. TCP cannot hand those two up first, because it only accepts “the next byte due”. They wait in the kernel until the first one is resent and arrives, and then all three come up together. This is head-of-line blockinghead-of-line blocking队头阻塞TCP delivers in order: when one packet is lost, the data behind it that has already arrived waits in the kernel for the resend. QUIC removes this only between streams; within one stream it remains.See the glossary: the one at the head of the line is missing, and everyone behind it is stuck.
Side trip W1 mentioned several messages arriving in one read; the held-up messages coming out together after a loss are one of the reasons.
How long the wait is depends on how fast the sender notices the loss. There are two ways:
- Later packets give evidence: 3 more packets arrive after it, the receiver’s TCP acknowledgments (ACKs; TCP’s own, not the server’s ACK from chapter 3) all say “I’m still missing that one”, and the sender resends at once (fast retransmit). The wait = the time to send 3 more packets + one round trip.
- A timer: if nothing follows, the sender can only wait for its retransmission timer. RFC 6298 says this timer should be at least 1 second; Linux lowers the floor to 200 ms, so in practice it is about one round trip plus 200 ms (
TCP_RTO_MINin the kernel source). With the series’ 200 ms round trip on mobile, that is 400 ms.
If the resend is lost too, the timer doubles.
2. Estimate: chat, a live room, a call
Now look at it from the side of one packet that wasn’t lost. If any packet sent shortly before it was lost and hasn’t been resent yet, it has to wait too; “shortly” means within the “wait” worked out above. That stretch holds “packets per second × wait” packets on average, and the chance that none of them was lost is (1 − loss rate) to that power. So the share held up by someone else’s loss is about:
1 − (1 − loss rate) ^ (packets per second × wait)
Three kinds of traffic, at two loss rates: 1% (a decent mobile network, an assumption here) and 10% (the series’ bad network). The round trip is 200 ms throughout.
- Ben’s chat: at v1’s peak, 1,390 deliveries per second shared by 10,000 people online is 0.139 messages per second each, one every 7.2 seconds (the same in v3: 139,000 shared by a million). Waiting for 3 more packets would take 21.6 s, so the timer always comes first: 400 ms. In those 400 ms, on average only 0.06 packets are sent.
- A busy live room: assume 20 messages per second to each viewer (an assumption here, each in its own packet; chapter 41 caps a viewer at 20 a second and packs a second’s messages into one batch, and with one push a second it looks more like chat). 3 more packets arrive within 150 ms; add a round trip and the resend goes after 350 ms (sooner than the timer), with 7 packets sent in between.
- A voice call: one audio packet every 20 ms, 50 a second. 3 more packets take 60 ms; add a round trip and the resend goes after 260 ms, with 13 packets sent in between.
demo.py runs 200,000 packets of each, and the results roughly match the formula (a little under it for the live room and the call: a packet that is lost itself counts as waiting for its own, not as held up by someone else’s). In the table, “in order” is TCP: a message waits when its own packet is lost and also when someone else’s is; “on its own” is each message delivered separately (UDP, or one QUIC stream per message), waiting only for its own. Lateness is measured from when it would have arrived with no loss:
| Loss 1%: held up by someone else | Loss 10%: held up by someone else | Loss 10%: how late the slowest 1% are (in order / on its own) | |
|---|---|---|---|
| Ben’s chat | 0.1% | 0.6% | 758 / 400 ms |
| Live room, 20 a second | 6.5% | 49.3% | 1,349 / 400 ms |
| Call, 50 packets a second | 11.7% | 72.3% | 1,540 / 660 ms |
Across the three, a message held up by someone else waits 0.14 to 0.31 s longer on average.
In the chat row, nothing breaks. Under 1% are held up by someone else, by a few tenths of a second, the same size as the 400 ms a message waits when its own packet is lost. The reason is simple: chat is sparse. When a packet is lost, there is almost never another one behind it.
In the live room, on a bad network half the messages are held up, and they are held up in runs: the screen pauses, then several messages pop out at once.
A call hates lateness most. An audio packet that arrives after its playback time is as good as lost. Calling apps keep a small buffer, say 100 ms, and drop anything later than that:
- over TCP, 8.5% of the audio is later than the buffer at 1% loss, and 67.9% at 10%; one loss turns into a run of stutters;
- over UDP, without resending, only 1% (or 10%) is missing, and the rest is on time; the decoder can patch the gaps roughly from the sound around them.
That is why real-time audio and video go over UDP: better lost than late.
3. Who uses UDP and QUIC
Call audio and video: UDP. Calls in browsers and apps mostly use WebRTC, whose media goes over UDP. In Discord’s 2018 post on its voice architecture, the client exchanges UDP messages with the voice server, while events such as messages and presence come over a separate WebSocket to the gateway. Where UDP is blocked, WebRTC can go through a TURN relay over TCP or TLS (RFC 8656); the sound is worse, but it works.
An app’s ordinary requests: more and more over QUIC.
- In its SIGCOMM 2017 paper, Google estimated that 7% of Internet traffic was QUIC; search latency fell by 8.0% on desktop and 3.6% on mobile, and YouTube’s rebuffer rate by 18.0% and 15.3%.
- Meta wrote in 2020 that more than 75% of its Internet traffic used QUIC and HTTP/3; for the Facebook app’s dynamic requests, compared with HTTP/2, errors fell by 6% and tail latency by 20%.
Most of these gains come from one fewer handshake round trip, surviving network changes and better loss recovery, not only from the absence of head-of-line blocking.
Chat connections: in public documents, mostly still TCP. Discord’s gateway is a WebSocket; Telegram’s MTProto lists its transports as TCP, WebSocket, HTTP and HTTPS, with no UDP.
One more thing to note: QUIC only removes head-of-line blocking between streams. Within a stream, data is still delivered in order. Put every chat message on one stream and a lost packet still holds up everything behind it. To really separate them, you need a stream per conversation (or per message).
4. Moving the chat connection to QUIC: four things first
One: a TCP way in where UDP is blocked. In the Google paper above, data from 2016 show that 4.4% of clients could not use QUIC (UDP blocked, or the path’s MTU too small: the largest packet that gets through is under the 1,200 bytes QUIC requires), mostly on corporate networks, likely behind enterprise firewalls; another 0.3% of clients were on networks that rate-limit UDP. RFC 9308 cites measurements of 3% to 5% of networks blocking all UDP, and says plainly that an application on QUIC must either accept failing on those networks or be ready to fall back to another protocol.
For v3: 4.4% of a million online is 44,000 phones, 2,200 per gateway, that can only come in over TCP.
In that paper, Chrome tries both: it connects over QUIC and over TLS/TCP at once, holding the TCP attempt back by up to 300 ms, and uses whichever connects first. The gateways then have to run and test both protocols.
Two: a load balancer that routes by connection ID. QUIC does not identify a connection by “address + port” but by a connection ID in each packet. When the phone moves from Wi-Fi to 4G, its address changes, the connection ID doesn’t, and the connection carries on (RFC 9000 section 9). This is connection migration.
But the layer-4 load balancer in front of the gateways (it looks only at IP addresses and ports, not inside the packets) usually hashes the address and port to pick a gateway. A new address means a new hash. With 20 gateways, the new address lands on the old gateway with a chance of 1 in 20, so 95% of migrations are sent to a gateway that has never heard of the connection, and the migration is wasted.
So the load balancer has to read the connection ID and route by it: RFC 9308 section 11.1 requires a deployment to make sure packets always reach the machine holding the connection’s state, and one way is to encode routing information in the connection ID (the IETF’s QUIC-LB draft). Also, in QUIC’s first version only the client can migrate.
Three: NATNATNAT(地址转换)Equipment at the carrier or in a home router that swaps the phone’s private address for a public address and port and records the mapping; packets pushed down find the phone only through it. A mapping idle too long is deleted, which is why long-lived connections need heartbeats.See the glossary forgets UDP sooner, so heartbeats come more often. Chapter 23 assumed a NAT forgets an idle TCP connection after 5 minutes, and sent a heartbeatheartbeat心跳A small packet sent while a connection is idle. It finds connections that have died, and keeps carrier equipment from forgetting connections that stay silent too long.See the glossary every 4 minutes. The standard for UDP is already shorter: RFC 4787 requires a NAT’s UDP mapping (the entry it keeps pairing the phone’s address and port with an outside port) to last at least 2 minutes and recommends 5 or more. In practice it is shorter still: RFC 9308 cites a 2010 study saying UDP applications can assume a NAT entry may expire after 30 seconds idle.
At 30 seconds, a heartbeat every 25 seconds (RFC 8085 asks UDP keep-alives to be no more often than every 15 seconds). Push capacity is chapter 22’s assumption of 50,000 pushes a second per gateway:
| NAT forgets after | Heartbeat every | Heartbeats/s, 1,000,000 connections | Per gateway (share of push capacity) | Idle phone’s radio on per day | |
|---|---|---|---|---|---|
| TCP (chapter 23) | 5 min | 4 min | about 4,170 | 208 (0.4%) | 1 hour |
| UDP / QUIC | 30 s | 25 s | 40,000 | 2,000 (4%) | 9.6 hours |
Nearly ten times more on the server, which it can carry; the phone’s battery is another matter. The last column (chapter 23: the radio stays on for about 10 seconds each time it wakes): a day is 86,400 seconds; a heartbeat every 4 minutes is 360 a day, 3,600 seconds of radio, 1 hour; one every 25 seconds is 3,456 a day, about 9.6 hours.
The connection ID doesn’t save pushes here either. Once the NAT has forgotten the mapping, there are two cases. If the phone sends first, the NAT opens a new port for it; the server sees a different port but recognises the phone by its connection ID, and the connection carries on. If the server sends first (a new message has come in), it only knows the old port; the packet reaches the NAT, which no longer knows which phone that port belonged to, and drops it. Section 3.2 of RFC 9308 makes exactly this point: the connection ID only helps if the other side is the first to send after the timeout.
Four: on the web, WebSocket over HTTP/3 is still rare. RFC 9220 (2022) says how to open a WebSocket over HTTP/3: one WebSocket takes one HTTP/3 stream. So even with it, all messages are on one stream and head-of-line blocking is unchanged. Browser support is limited: Chrome posted an Intent to Prototype in November 2022, and at the time of writing I could not find a major browser that turns it on by default. A web chat connection is, for now, a WebSocket over TCP.
5. The decision
The trunk’s chat connection stays on TCP: a custom protocol over TLS in the apps, WebSocket on the web. The reason is the numbers in section 2: with chat traffic, head-of-line blocking makes under 1% of messages wait a few tenths of a second longer; moving to QUIC means running a second path over TCP, changing the load balancer and sending heartbeats nearly ten times as often.
Call audio and video go over UDP, and over TCP/TLS through a TURN relay where UDP is blocked.
For network changes the trunk doesn’t rely on connection migration but on a quick reconnect and sync (chapters 6 and 11; chapter 26 covers switching networks). When to revisit: live-room message streams (chapter 41), or the day a TCP fallback, load balancing by connection ID and browser support are all off the shelf.
Further reading
UDP brings back some of the trouble TCP hides from you, such as finding out at which layer packets are being lost. The author wrote up one such investigation: Troubleshooting a UDP Packet Loss Issue on Linux.
The main path goes on at chapter 23, “Heartbeats”: most of a million connections are silent most of the time, and a silent connection may have died long ago.
Comments
Comments live in GitHub Discussions and need a GitHub account.