v1: a reliable channel | IM Systems in Depth

At the start of chapter 3, the complaint was “I know I sent it, and they say it never arrived.” Each of the six chapters since then fixed one thing: messages get lost (chapter 3), repeated (chapter 4) and out of order (chapter 5); a phone coming back must catch up (chapter 6); a killed app must not lose messages (chapter 7); a photo must not block the text behind it (chapter 8).

This chapter makes no new decision. It does four things: it follows one message through v1 from end to end; it puts the six decisions and their costs in one table; it redoes the accounts for the whole server and for one phone, from v0 to v1; and it says plainly what v1 cannot do yet.

The scale has not changed: 100,000 daily users, 10,000 online at peak, 139 messages a second, one server, one database. v1 did not add a single user. Everything it added is a guarantee.

1. One message, end to end

7 pm. Ana is in a lift and sends Ben “I’m downstairs.” Ben is on the subway, with no signal.

Ana’s phone

Written to the local database first. The app makes a message ID for the message (a 128-bit random number, say) and, in one transaction, writes a row to the local messages table in state “sending”. A transaction is a group of writes that either all take effect or none do. The screen reads that row from the database and shows it at the bottom of the chat (chapters 4, 5, 7). The row is the outbox: if the app is killed now, it is still there next time.
Sent, then wait for the ACK. The message goes out on the long-lived connection with the conversation, the message ID and the text. In the lift the connection drops and the ACK never comes. The phone reconnects and resends with backoff (at 0, 1, 3, 7 and 15 s in the simulator; a real app counts time, keeps waiting while there is no network, and adds some random jitter), always with the same message ID (chapter 3).

The server: message layer

Repeats recognised. First it checks the message IDs seen in the last few minutes (32 s in the simulator), kept in memory. If it has seen this one, it writes nothing and sends the first ACK again, unchanged (chapter 4).
One transaction. If not, it does three things in one transaction: it takes the conversation’s next seq (chapter 5); it stores the message; and it adds an entry to the inbox of each member of the conversation (here Ana and Ben), numbered by that person’s own inbox seq (chapter 6). The message table has two unique keys: the database will not accept a second row with the same sender + message ID, or the same conversation + seq. Hitting the first one means a repeat the window missed, such as a message resent from the outbox an hour later after the app was killed; the server looks up the original and sends the same ACK (chapter 4).
The ACK only after the commit, carrying the message’s seq. Ana’s phone runs one UPDATE that marks the row “sent” and fills in the seq; the message moves to its place by seq (chapters 5, 7).

Dispatch layer

Push. The server finds Ben’s connection and pushes the message with its seq and Ben’s inbox seq. Ben is on the subway, his connection died long ago, and the push is lost. That is fine: a push is only a hint, and the message is already in Ben’s inbox (chapters 2, 6).

Ben’s phone

Out of the subway: sync. The phone keeps one inbox cursor, meaning “I have everything in my inbox up to this number.” Once reconnected, it asks with the cursor: “everything after 7,345.” Pages of 50; each page is committed in one transaction with the new cursor and each conversation’s last_seq (chapters 6, 7). If it is more than 10 pages behind, or the cursor is older than the 7 days the inbox keeps, the server answers with the conversation list only; the older messages are recorded locally as a gap, a marker that says “this stretch is still on the server, not pulled yet”, filled in when Ben scrolls up to it.
Later pushes. While connected, a push whose inbox seq is exactly cursor + 1 is committed together with the new cursor. If the number jumps, the phone stores the message, leaves the cursor where it is, waits 0.5 s, and if something is still missing pulls from the cursor on (chapters 5, 6).
The screen reads only the local database. Ben opens the chat with Ana: no request, and “I’m downstairs” is already there (chapter 7).

If it is a photo. Before step 1 there is one more stage: the phone compresses the photo, asks the server for an upload address, and uploads the file in parts to an upload service in front of object storage, in state “uploading”; after a drop it asks the upload service “how much do you have?” and goes on from there. Only then does it send a small message of about 300 bytes with the file id, size, hash, width and height and a placeholder. From step 2 on, that small message travels the same road as text. Ben sees the placeholder and a thumbnail first, and downloads the full photo from a CDN when he taps it (chapter 8).

Any of the nine steps can go wrong: the connection in the lift, the lost ACK, the push into the subway, the app killed halfway. Each of these is caught by a later step, and what catches it is not “be more careful” but a rule.

2. Switch one off, and what comes back

Each chapter’s simulator is still in its chapter, and you can go back and switch things off yourself. Below are the numbers each of them gives, all at the book’s 10% loss; as chapter 3 said, that is far more than real networks lose, chosen so the problems are visible.

Switch off	What comes back (in that chapter’s simulator)
ACKs and resends	Chapter 3: the phone shows “sent” but the server never stored it, 10.1% of 10,000 messages
Message ID	Chapter 4: Ben sees repeats, 1,113 in 10,000 (11.1%). With only the window of recent IDs and no unique key, a resend an hour later still shows twice
Conversation seq	Chapter 5: the two screens disagree on order, 19.6 pairs per 100 messages; a lost push goes unnoticed, 10.0 per 100 (0.01 with seq)
Inbox	Chapter 6: nothing is lost; with a cursor per conversation instead, the phone still catches up. What comes back is only cost: each sync looks at 200 conversations, about 13,900 reads/s at peak against 69 with the inbox, and uploads 2.4 KB instead of 8 bytes
One transaction	Chapter 7: the app is killed and the cursor gets ahead of the messages. Cursor first, messages committed one by one: killed at 1.0 s in the busy evening, 74 messages are missing and will not be pulled again; a kill at 85.3% of moments loses some (a commit per page: 4.7%; one transaction: 0)
Separate upload	Chapter 8: text waits behind the photo, the three lines wait 27.7, 23.7 and 20.7 s; on a link that drops every 8 s, not one line gets out in 60 s

The inbox row deserves a fair word: the inbox does not fix a bug, it is a trade. Each message writes one inbox row for each person it reaches, 6.67 on average (a message reaches 10 devices on average and each person has 1.5, so 10 ÷ 1.5 ≈ 6.67 people, chapter 6). In return, each sync reads only what actually arrived, and the inbox can also say things that belong to no single conversation, such as “Ben was added to a new group”. At v1’s scale, either way is defensible.

3. Two halves, built facing each other

Split the nine steps by where they happen, and v1 is two halves, each one the phone and the server built to match each other:

The sending half: the phone’s outbox, message ID and resends, facing the message layer’s dedup, seq and ACK. Together, each message Ana sends is stored exactly once on the server, with one definite place in its conversation.
The receiving half: the dispatch layer’s pushes with the inbox seq, facing the phone’s sync: the inbox cursor, pulls on a jump, and pages written into the local database one transaction each. Together, Ben’s phone ends up with every message.

Both halves use the same three rules, and each rule appears once at each end:

A unique key recognises repeats. On the server it is sender + message ID (chapter 4); the phone’s local database has the same unique key, so however many times a message arrives, by push, by sync or as an ACK, it lands on the same row (chapter 7).
A number is committed with what it stands for. The server takes the seq and the inbox seq inside the transaction that writes the message, so the order of the numbers is the order of the commits (chapters 5, 6); the phone moves its cursor inside the transaction that writes the messages, so the cursor never gets ahead of them (chapter 7).
A push is a hint; the pull is the truth. A push that is lost, late or out of order is made good by a pull by number (chapters 2, 5, 6).

What people call “exactly once” is, here, two things put together: at least once (resends, sync) plus recognising repeats (the unique key). A network cannot deliver exactly once; what you can do is make a repeat harmless.

4. The whole v1 map

The whole v1 map: still one server and one database, with a media path beside them (upload service, object storage, CDN). Most of what chapters 3–8 added are rules: the local database on each phone (outbox, inbox cursor), dedup, seq and the inbox in the message layer, pushes that carry the inbox seq, and the rule that a photo is uploaded before its small message is sent.

Compared with chapter 2’s v0, it is still one program on one server, with a media path beside it. Almost nothing in v1 changed the boxes; what changed are the rules between them, and the nine steps above are those rules.

5. v1’s six decisions

Chapter	Decision	Cost	Revisit when
3 Acks and retries	The message goes into an outbox; the server ACKs once it is stored; without an ACK, reconnect and resend with backoff (and jitter); mark it failed only when it keeps failing	A lost ACK stores a repeat (about 10% of messages at 10% loss); a resent message arrives later; the outbox must be on disk	What “stored” means (chapter 28); reconnect storms when the network comes back (chapter 23)
4 Duplicates	The phone makes a message ID before the first try; the server remembers recent IDs; a unique key on sender + message ID is the backstop	One more unique index, one more lookup on a repeat; the phone must keep IDs from colliding	Several message servers (chapter 27); ACK after writing a log (chapter 28)
5 Ordering	One counter per conversation, seq taken in the write’s transaction; phones order by seq, sending ones at the bottom, and fetch on a jump	One conversation’s writes take turns (at most 200/s at 5 ms a write); your own messages may move; a lost push of the last message waits for the next one	One server cannot hand out seqs for every conversation (chapter 27)
6 Offline sync	One inbox per person, an entry for each recipient in the message’s transaction; the phone keeps one inbox cursor; too far behind, only the conversation list	6.67 more rows per message (927/s at peak, 853 MB a day); a person’s messages queue on their row’s lock	Groups too big to write every inbox (chapter 10); several devices per person (chapter 14); whether inbox writes wait for the ACK (chapter 28)
7 The local database	SQLite on the phone; the screen reads only it; each page, push and ACK is committed in one transaction with the cursor or state it moves	Two copies of the truth, and the client must repair its own; about 34 MB of text a year; migrations on every upgrade; transaction discipline	How deep a new device syncs (chapter 14); the conversation list (chapter 17); the web client (chapter 26); one SDK for every platform (chapter 29)
8 Images and files	Files are uploaded in parts to an upload service and stored in object storage, then a small message of about 300 bytes is sent; the receiver sees a thumbnail first and downloads from a CDN on tap	Two paths that must agree (orphan files, images that will not open); a photo’s seq comes after text sent during the upload; whoever has a file’s address can open it; an upload service, object storage and a CDN to run, 43.8 TB of storage a year	A photo in a group of 500 is seen by 750 devices (chapter 18); new devices (chapter 14); a placeholder message updated later (chapter 16); end-to-end encryption (chapter 41)

Of the six costs, the one that will hurt first is chapter 6’s: inbox writes grow with the number of people in the conversation. A 1:1 chat writes two rows per message; v2’s hiking group has 500 members, so 500.

6. v1’s accounts

As in chapter 2, here are the accounts for the whole server and for one phone. Both columns use the same assumptions: v0 is the push design at the end of chapter 2, with phones that store no messages and ask the server for the latest 50 every time a chat is opened (the approach at the start of chapter 7); every number in the v1 column comes from the chapter in brackets.

The server (10,000 online, at peak)

One item is in both columns: a pull each time the app is opened, 100,000 people 20 times a day, at peak 2,000,000 ÷ 86,400 × 3 ≈ 69 a second. v0 has one more: 100,000 × 60 = 6 million chat opens a day, about 208 a second at peak, each answered with 50 × 200 bytes = 10 KB.

Resource	v0 (end of chapter 2)	v1
CPU	About 1,389 pushed frames/s; 69 pulls/s; 208 chat-open queries/s; a burst of handshakes on restart (about 5 s for 10,000 phones)	Pushes unchanged; 69 syncs/s (now by inbox cursor); the 208 chat-open queries are gone; sends received and ACKs sent, 139/s with no loss; at the book’s 10% loss, a message is sent 1.23 times on average and each try arrives 90% of the time, so the server receives 139 × 1.23 × 0.9 ≈ 154/s, about 15 of them repeats (chapters 3, 4), each ACKed; about 7 upload addresses/s (chapter 8); each message’s transaction writes the message, the conversation and 6.67 inbox entries instead of 1 row
Memory	About 30 KB per connection, about 300 MB for 10,000; user → connections	The same, plus the window of recent IDs: about 4,400 IDs, 142 KB for 32 s, and only 1.3 MB for 5 minutes (chapter 4)
Network bandwidth	Pushes 1,389 × 200 bytes ≈ 2.2 Mbit/s; chat opens 208 × 10 KB ≈ 16.7 Mbit/s; about 19 Mbit/s in all	About 2.2 Mbit/s, since a photo message is only about 300 bytes; the photos themselves skip the message server: 11.1 Mbit/s uploaded to the upload service, about 39 Mbit/s downloaded, mostly from the CDN (chapter 8)
NIC packets	A push frame plus its ACK, about 2 packets: about 2,800/s; chat opens: 208 requests/s, each 10 KB reply about 7 packets (up to about 1,460 bytes each, not counting TCP ACKs), about 1,700 more: about 4,400 in all	139 ACK frames/s more, about 280 packets, and no chat-open packets: about 3,100 in all with no loss
Disk capacity	About 0.8 GB of messages a day, about 292 GB a year	Messages unchanged, plus two unique indexes; the inbox 853 MB a day, about 6.0 GB kept for 7 days (chapter 6); photos in object storage, 40 GB a day, 14.6 TB a year, 43.8 TB with 3 copies (chapter 8)
Disk IO	Writes: about 139 inserts/s at peak, each one commit that waits for fsync (until the data is really on disk); reads: 69 pulls/s and 208 chat-open queries/s	Still about 139 commits/s (chapter 5), but each writes more rows: 927 more inbox rows/s (chapter 6); reads: 69 inbox range reads/s by cursor, plus a few gap fetches

File descriptors are as in v0: 10,000 connections are 10,000 descriptors; photo uploads connect to the upload service, not to the message server.

In one sentence: v1 makes the message server write more, 6.67 inbox rows per message, and read less: with a copy on each phone, the 208 chat-open queries a second and about 17 Mbit/s are gone. The big numbers are on the other path: photos are 40 GB a day, 50 times all the text, which is why they need a path of their own (chapter 8).

Ben’s phone (one day)

Most of v1’s changes only show over a day (a sync per app open, how many chats are opened a day), so this table counts a day rather than chapter 2’s hour. The v0 phone, as above, stores no messages; v0 had no photos either.

Resource	v0	v1
Requests	A pull per app open (20); for each chat opened, a request for the latest 50 (60); 40 messages sent	A sync per app open (20; 13.3 messages arrive between opens on average, one page, chapter 6); opening a chat sends no request; 40 messages sent, each waiting for an ACK; photos: 2 uploads, about 13 thumbnails and 4 full photos downloaded (chapter 8)
Data	Down: 53 KB of pushes plus 600 KB for opened chats, about 650 KB	Down: 53 KB of text, 935 KB of photos (thumbnails plus the 30% opened); up: 400 KB more for photos; a sync carries one 8-byte cursor
Battery	The radio wakes for each push; each chat opened waits for a round trip	Pushes unchanged; opening a chat uses no network; a few resends on a bad signal (1.23 tries per message at 10% loss); back after a week offline (1,869 messages, over 10 pages), the server sends only the conversation list: 4 requests, about 42 KB (chapter 6)
Background	The system closes the connection; no outbox, so written out counts as sent	The connection is still closed, but the outbox and upload progress are in the local database, and sending and uploading continue at the next open; nothing can wake the phone yet (chapters 15, 25)
Storage	Almost none	About 92 KB of text a day, about 34 MB a year (chapter 7); a photo cache of about 487 MB a year that can be cleared (chapter 8)

The phone spends more on storage and on photo data; what it saves is downloading the same messages again: v0’s 600 KB a day for opening chats is gone.

7. What v1 cannot do yet

v1 does one thing well: between two people, a message is not lost, repeated or reordered, and a killed app loses nothing. Here is what it cannot do yet:

A group of 500. Writing every inbox, one message is 500 rows; the hiking group’s 1,000 messages a day are 500,000. Write once or 500 times is chapter 10. Who is in the group, and who gets the messages at the moment someone joins or leaves, is chapter 11.
Who may message whom. v1’s business layer is still one check: “is Ana in this conversation?” Strangers, blocks, buyers and sellers are chapter 12.
Unread counts and read receipts. v1 knows how far Ben’s phone has received, not how far Ben has read (chapter 13).
Several devices. The book assumes 1.5 per person; Ana has a phone and a laptop. The inbox could keep a cursor per device, but “read up to here” has to be combined across devices, and how deep a new device should sync has no answer yet (chapter 14).
Waking a closed phone. Once the app goes to the background the connection is gone; the message waits in the inbox, and Ben does not know (chapter 15).
Recall and edit (chapter 16), and the conversation list on the first screen (chapter 17). v1’s local database keeps only each conversation’s last message.

All of this is v2 (chapters 10–18). Further on, v1 is still one server: with more connections it cannot hold them, and every deploy disconnects everyone. That is v3.

Next, chapter 10: Ana writes one line in the hiking group of 500. Should the server write it once, or 500 times? After v2’s eight chapters, chapter 18 walks through all of v2 the way this chapter did for v1.