Framing | IM Systems in Depth

Chapter 2 chose a long-lived connection: WebSocket in the browser, and often a custom TCP protocol over TLS in the apps. That chapter left one line: on native TCP, “where a message starts and where it ends” is yours to cut. This side trip is about that line.

TCP hands the receiver a stream of bytes, not a list of messages. It promises the bytes arrive in order, none lost, none repeated. It does not promise that one write on the sender is one read on the receiver. How many bytes one call to read() returns is up to the network and the kernel.

So the sender has to write the boundaries into the bytes. The usual way is two bytes in front of every message, saying how long it is. Ana sends “hi!” and then “ok”, and the bytes on the wire are:

00 03 68 69 21 00 02 6f 6b

To make it easier to read, printable bytes are written as characters from here on: 00 03 h i ! 00 02 o k. 00 03 says “the next 3 bytes are one message”, so h i ! is the first; then 00 02 says “the next 2”, and o k is the second.

Framing happens once per hop: the server cuts Ana’s messages out of her connection, then writes them to Ben’s connection, and Ben’s phone cuts them again. The scene below is the second hop.

1. Watch it break

Many people’s first receiver looks like this: read the 2-byte length in full, then make one read() for the body and take whatever it returns as the body. The mistake is in what read() promises: it returns “at most” the bytes you asked for, and may return fewer. The correct code keeps reading until it has enough (io.ReadFull in Go), or keeps what it has for next time. The first receiver below skips that step.

First, the example from the top: 00 03 h i ! 00 02 o k arrives in two pieces, 00 03 h and i ! 00 02 o k. The receiver reads the 2-byte length, then makes one read() for the body and takes what it gets. What does Ben see?

The server wrote 6 times to Ben’s connection

0003hi!

“hi!”

0002ok

“ok”

0003e5a5bd

“好”

0009on␣my␣way

“on my way”

000csee␣you␣at␣7

“see you at 7”

0006where?

“where?”

The bytes Ben’s phone got, in the pieces they arrived in

00length (2 bytes)abtext (neighbouring messages alternate shades)

One read for the body: right: –
Wrong bubbles: –
Never shown: –
Length prefix + buffer: right: –

First pick “Test on one machine”: the server wrote to Ben’s connection 6 times, each write arrives on its own, and both receivers are right. On one machine nothing is lost and the reader keeps up, so one write is almost always read in one read. This code passes every test.

Now pick “Mobile network”. The same 47 bytes arrive in 6 pieces:

Piece 4 holds only the length of “on my way” and its first 4 bytes, “on m”. The length says 9, the body read() returns 4 bytes, and the receiver shows “on m” as a whole message. It throws nothing away: the rest, “y way”, is still waiting on the connection.
The next length it reads is “y” and a space: “y” is 0x79 and a space is 0x20, so read as a 2-byte length, high byte first, that is 0x7920 = 31,008. The body read() takes the 17 bytes left in this piece and shows them as a garbled bubble. “see you at 7” never shows up on its own.
Piece 6 happens to start at the start of a message, so “where?” is right again.

Ben sees 4 of the 6 right, and it fails quietly: no error, just half a message and some garbage. “Another cut” shows other cuts; sometimes every cut falls inside a length or between two messages and the code is right all the way. That is what makes it hard to find.

Another bug that is just as common has no length at all: one JSON object per write, and whatever one read returns goes straight to the JSON parser. It fails on half a message, and it fails when two messages share one read.

The “Length prefix + buffer” receiver is right every time. It puts what it reads into a buffer and only takes messages out by their length; when a message is not complete yet, it keeps the bytes and waits for the next piece. In Go, a bufio.Reader with two io.ReadFull calls (one for the length, one for the body) does the same.

Sticky packets and half packets

Chinese engineers call these two cases 粘包, “sticky packets” (several messages in one read), and 半包, “half packets” (part of one message in one read). The names sound like TCP is broken. It is not. TCP is a byte stream; it never had packet boundaries to give you, and it never promised that writes and reads match up. The application lost the boundary: either it did not write one down when sending, or it wrote one and did not cut by it when reading. (UDP delivers each datagram on its own and keeps its boundary, so it has no such name.)

On a mobile network both happen every day:

Sticky: a packet is lost and TCP resends it; the bytes behind it have already arrived but wait in the kernel, and when the resent packet lands, several messages come up at once. An app waking from the background, or a main thread that stalls for a moment, also lets several pile up.
Half: a message is bigger than the data one TCP packet carries (about 1.4 KB), or the lost or late packet falls in the middle of a message, so it arrives in several reads.

2. Three ways to cut

A delimiter: a special byte after every message, such as a newline, one message per line (JSON Lines works like this). It is simple, and you can read it with telnet. It costs three things:

The delimiter must never appear inside a message, or it has to be escaped. JSON already writes a newline inside a string as \n, so “one JSON per line” works; binary data has to be escaped, or turned into base64 first, 33% more bytes.
The receiver has to look at every byte to find the delimiter.
A connection that never sends a newline makes the buffer grow forever, so a line needs a maximum length too. Go’s bufio.Scanner allows 64 KB per line by default; past that it reports “token too long” and stops.

Redis’s protocol, RESP, uses both: a short “simple string” ends with \r\n and may not contain \r or \n; a “bulk string” of any content starts with its length, as in $5\r\nhello\r\n, and its body is neither scanned nor escaped.

A length prefix: the “Length prefix + buffer” receiver above. Every message starts with a length of fixed size, and the receiver cuts like this:

Append whatever read() returned to this connection’s buffer.
Fewer than 2 bytes in the buffer (the length itself)? Wait for the next read.
Read the length n. Fewer than 2 + n bytes in the buffer? Wait for the next read.
Take those n bytes out: that is one whole message. Hand it on, and go back to step 2.

One read can give zero messages, one, or several; whatever is incomplete stays in the buffer. The body can hold any bytes, with no escaping and no scanning. Note that the length counts bytes, not characters: “好” is one character and 3 bytes in UTF-8, so its length in the scene is 00 03.

A fixed size: every message is exactly N bytes, so there is no length to write. But chat messages vary a lot: size N for the longest and short ones are mostly padding; size it for short ones and long ones do not fit. It only suits things that really are a fixed size.

The choice here is the length prefix.

3. Estimate: how many bytes for the length, and how long a message may be

The header’s cost can be ignored: with the book’s 200-byte message, a 2-byte length is 2 ÷ 200 = 1% and a 4-byte one 2%, both smaller than the 40 bytes of headers every TCP packet already carries.

The real choice is the maximum:

2 bytes go up to 65,535 bytes. A long text of 4,000 Chinese characters is 12,000 bytes in UTF-8, and fits; images and files do not travel on this connection anyway (chapter 8: images and files go through their own upload and download).
But once the length field is fixed it is hard to change (chapter 30: new servers have to work with apps two years old). So the common choice is 4 bytes, up to about 4 GB, with a cap of your own.

Why does the cap matter? The receiver collects as many bytes as someone else’s length says. An old app version that writes a wrong length, or a client that misbehaves on purpose, sends 7f ff ff ff: “2 GB follow”. A server that allocates a buffer of that size up front can be brought down by one connection. Even one that only grows the buffer as bytes arrive can be fed slowly, by a connection that never finishes. The worst case at the cap, with every connection holding one half-received message:

Buffer per connection, at most	v1: 10,000 online (one server)	v3: one gateway, 100,000 connections (16 GB of memory)
64 KiB	10,000 × 64 KiB ≈ 0.66 GB	100,000 × 64 KiB ≈ 6.6 GB
1 MiB	10,000 × 1 MiB ≈ 10.5 GB	100,000 × 1 MiB ≈ 105 GB

So:

Set the cap by what the protocol really needs. A phone sends single messages, and 64 KB is generous; a batch the server sends down (chapter 6: after a reconnect, the missed messages are fetched page by page) is paged by the server itself, so the cap need not grow for it.
Do not allocate by the announced length; grow the buffer as bytes arrive. And give every frame an in-frame deadline: once a frame’s first byte arrives, the whole frame must arrive within T, or the connection is closed. If any trickle of bytes counts as alive, a connection that drips one byte at a time is never caught. An idle connection, with no frame under way, is caught by chapter 22’s heartbeats (a small packet sent on a timer to show the connection is alive).
What if a length is over the cap? If the length is honest and just too big, the receiver can read that many bytes, throw them away and go on with the next message: Netty’s decoder does exactly that, dropping the long frame’s bytes, raising TooLongFrameException and decoding the next one. But the receiver cannot tell “honestly too big” from “the length itself is corrupt”, and after a corrupt one it can never find where the next message starts. Skipping also means reading all those bytes. So IM servers usually close the connection. The phone reconnects, and messages without an ACK are sent again (chapter 3: a message without an ACK is sent again).

4. WebSocket already frames for you

As chapter 2 said, a browser can only use WebSocket, and WebSocket is a framing protocol itself. By RFC 6455, every frame has a 2-byte header with 7 bits of length: 0–125 is the length; 126 means 2 more bytes of length follow; 127 means 8 more. A frame from client to server also carries a 4-byte mask. For a 200-byte message, the header from server to phone is 2 + 2 = 4 bytes; from phone to server it is 8 bytes, 4%.

The browser’s onmessage always hands you a whole message, and server-side WebSocket libraries put frames together for you. A message may even be split over several frames; the library joins them. So the web side never sees sticky or half packets.

The cap is still yours to set. Section 10.4 of RFC 6455 warns that a malicious endpoint can send one frame claiming 2^60 bytes, or split a message into an endless stream of small frames; implementations must protect themselves, and should limit both the frame size and the size of the joined message. When you use a WebSocket library, find its setting for frame and message size, and set it.

When the app speaks native TCP, the loop from section 2 is yours to write, or take a ready one: in Go, a bufio.Reader with io.ReadFull; in Netty, LengthFieldBasedFrameDecoder, which must be given a maxFrameLength and raises TooLongFrameException past it.

5. Other answers

Telegram’s MTProto over TCP has several framings: a 1-byte length (counted in 4-byte units, 4 bytes for long packets), a fixed 4-byte length, or 12 bytes (length, sequence number and a CRC32 checksum). One or four special bytes at the start of the connection say which; none at all means the 12-byte one.
HTTP/2 is frames on one TCP connection too: a 9-byte header on every frame, with 24 bits of length; by default a receiver only takes frames up to 16,384 bytes, and can announce a larger size, up to 2^24 − 1. gRPC, on top of it, gives every message its own 5-byte prefix (a 1-byte compression flag and a 4-byte length), because one gRPC message can span several HTTP/2 frames. Two layers of framing, each keeping its own boundaries.
Redis’s RESP: as above, delimiters and length prefixes together; a bulk string is at most 512 MB by default (proto-max-bulk-len), a cap of its own.

6. The decision

Decision card

Problem: TCP hands over a byte stream, and how much one read returns is up to the network. “Read the length, then one read for the body” passes on one machine; on a mobile network it shows half a message, then reads lengths from the middle of messages and everything after turns to garbage.
Choice: On the web, WebSocket: the protocol and its library cut the frames. In the app's native TCP protocol, a 4-byte length before every message; the receiver cuts by it through a buffer and hands a message on only when it is complete; the length has a cap, and going over it closes the connection; each frame must arrive within a deadline once it starts.
Cost: 4 more bytes per message (about 2% of a 200-byte message); a read buffer per connection; a length read wrong cannot be recovered from, only closed and reconnected.
Revisit when: When one connection has to carry big blobs and small messages together (chapter 8, where images and files take another path); when there are so many connections that read buffers matter (chapter 21, a million connections); when the header also needs a type, a version or a compression flag (side trip W2, encoding; chapter 30, old and new versions together).
Other answers: A delimiter (JSON Lines, RESP's simple strings); a fixed size; MTProto's 1-, 4- or 12-byte headers; HTTP/2 frames with gRPC's 5-byte prefix.

The lengths cut out runs of bytes; how those bytes are laid out is still open: the same message in JSON or in binary, how many bytes apart? That is side trip W2, “Encoding”. The main path goes on at chapter 3, “Acks and retries”.