# Chapter 33: Where did my message go?

> Ana says the message she sent at 8 o’clock never reached Ben. Every server’s dashboard is green. At which hop did it go wrong? And the harder question: right now, are everyone’s messages arriving?

IM Systems in Depth · https://im.liko.page/en/tracing-a-message/

At 8 p.m., Ana sends feedback through the app: “Ben says he never got the message I sent him.” On her phone, the message shows as sent.

By v3, a message goes through seven hops: Ana’s phone, the gateway she is on, the conversation’s message server, the message log, push, the online routes, and the gateway Ben is on, and only then Ben’s phone (chapter 32). This chapter first finds this one message, then answers a bigger question: **right now, are everyone’s messages arriving? How do we know?**

## 1. First, find this one

Every hop keeps logs, but each in its own terms: the gateway logs connection numbers, the message server request numbers, push the seq, the phone its local row numbers. They can’t be joined up.

To join them, you need a key every hop knows. It already exists: the message ID, which Ana’s phone generates and chapter 4 uses to spot duplicates; once a seq is assigned, the message also has “conversation + seq”. There is just one rule: **every hop logs the message ID and the seq.**

Joined up, it looks like this:

| Time | Hop | What it logged |
|---|---|---|
| 19:57:30 | Ben’s phone | last frame received from the server |
| 20:00:00 | gateway 7, message server, message log | received, seq 4,823 assigned, written to the message log, ACKed |
| 20:00:00 | push, gateway 12 | the routes say Ben’s phone is on gateway 12; written to that connection, success |
| 20:01:30 | Ben’s phone | 4 minutes of silence, sends a heartbeat (chapter 23) |
| 20:01:40 | Ben’s phone | no answer in 10 seconds, reconnects to gateway 4, syncs; gets 4,823 at 20:01:41 |

The message was not lost; it was **101 seconds** late. Between the phone and the server sits the carrier’s NAT: it swaps the phone’s address for a public one and keeps a “mapping” so that packets coming back can find the phone (chapter 23). Ben’s carrier’s NAT drops a mapping after 2 minutes idle, shorter than the 5 minutes chapter 23 assumed, so the connection had quietly died at 19:59:30, and nobody was told. Push wrote into a dead connection: a successful `write()` only means the bytes reached this machine’s buffer (chapter 2).

Along the way, no server made an error. On gateway 12’s dashboard, the push success rate is 100%.

## 2. Why the dashboards are all green

Each server measures its own step: did it receive, how long did it take, how many errors. But in the case above, the trouble was between steps: the gateway wrote it out, the phone did not get it. No server can see that.

Google’s SRE book splits monitoring in two: metrics from inside the system are white-box; “testing externally visible behavior as a user would see it” is black-box ([Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/)). It also says that not measuring on the client misses problems that “affect users” but “don’t affect server-side metrics” ([Service Level Objectives](https://sre.google/sre-book/service-level-objectives/)).

“Did it arrive” can only be measured from the user’s side.

## 3. One target

First, say clearly what is wanted. In the SRE book’s terms, the number measured is an **SLI** (service level indicator), and the target set for it is an **SLO** (service level objective).

v3’s main indicator: **every device that was online when a message was sent receives it within 5 seconds**, where receiving means it is in the phone’s local database. The target is 99.99% over 30 days. Offline devices are not counted here; they rely on OS push and the next sync (chapter 17), and are counted separately.

99.99% sounds high, but at 400 million messages a day (10 million users, 40 each) it still allows **40,000** to be late or lost. That is counted in messages because it is easy to remember; the instruments measure each delivery, about 10 per message, so about 4 billion a day, and the same 0.01% is 400,000 deliveries. Spent as whole-site downtime, it is just **4.3 minutes** in 30 days. The SRE book calls this margin the error budget: how much is left decides how much risk can still be taken, such as deploys ([Embracing Risk](https://sre.google/sre-book/embracing-risk/)).

Two more numbers sit beside it:

- **Latency**: from sending to the other side receiving, looked at as a distribution, not an average.
- **Real duplicates**: chapter 4’s resends, about 36 million a day, which the server recognises and drops, are normal work and do not count. What counts is the one that slips through: v3 writes the message log first and storage later (chapter 29), so a very late resend that is not recognised gets a second seq and is pushed, and the other side sees the same message twice; only when storage writes that second copy does the “sender + message ID” unique key (chapter 4) clash and give it away. Each clash is counted once.

## 4. Two instruments

**Probes.** A few dedicated test accounts send each other messages, one per gateway per second, all day. With 20 gateways that is 72,000 an hour, only 0.14% of peak traffic. If a whole gateway breaks, one message a second fails there, and it shows within seconds.

But probes are too few to measure 99.99%: if every path loses 0.01%, only 7 fail in an hour, and you cannot tell a failure from bad luck. And probes run inside the data centre, never crossing real users’ carrier networks, so they never meet a NAT like Ben’s.

**Sampled real messages.** A message is sampled when the hash of its message ID, divided by 1,024, leaves a remainder of 0. Each hop works this out for itself and gets the same answer, so no extra marker has to be passed along. A phone that receives a sampled message at once sends a tiny frame back on its own connection: “got 4,823”.

Timing uses only the servers’ clocks: from Ana’s gateway receiving the message to Ben’s gateway receiving the report. Phone clocks can’t be trusted (chapter 5). This counts one extra upstream leg, about 0.1 seconds, which is acceptable. If Ben’s phone is disconnected, the report only arrives once he reconnects, which is rightly counted as late.

At peak there are 139,000 deliveries a second; 136 are sampled, about 490,000 an hour.

At the target, about 49 an hour should be late. With counts this size, luck moves the number only a little: seven or eight either side of 49 is normal, and a few dozen extra is not luck. The probes, at 7 an hour, cannot do this.

An example. Suppose the load balancer in front of gateway 12 gets a config change, idle connections are now cut sooner than the heartbeat, and 0.5% of its pushes are written into dead connections:

- Gateway 12’s own dashboard: push success rate still 100%.
- Samples passing through it: about 24,000 an hour, about 125 late; by the target only 2 should be, so 122 are its doing.
- The whole site: the other deliveries are already 0.01% late at the target; add its 0.025% (0.5% ÷ 20 gateways) and it is 99.965%, burning the error budget at **3.5 times** the allowed rate.

## 5. Logs: record only some

Section 1 relied on logs. If every delivery had a line logged by each of the 6 services on the way (the gateways at both ends, the message server, the message log, push, the routes):

> 139,000 deliveries a second × 6 lines × 200 bytes ≈ **167 MB a second**, 14 TB a day at that rate (about 5 TB at the daily average, a third of the peak)

One gateway would write 6,950 lines a second. What often happens in practice is that application logs are the first thing to saturate disk IO. The logging sits on the push path, so when the disk slows (log rotation, another process, a full disk), pushes wait with it. Google’s Dapper paper measured it on a web search cluster: tracing every request added 16.3% to average latency; tracing one in 1,024 made no measurable difference ([Dapper, 2010](https://research.google.com/archive/papers/dapper-2010-1.pdf), table 2).

So three rules:

- **Sample.** Log every hop only for sampled messages, picked the same way as in section 4: 163 KB a second.
- **Buffer, don’t wait.** Logs go first into a bounded buffer in memory; when it is full, logs are dropped rather than making pushes wait.
- **Ship them off the machine.** A background job sends them in batches to a dedicated log service, not to the local disk.

Two things are logged in full: connections opening and closing (about 2.8 MB a second at peak), and the traces a message leaves by itself (position in the message log, seq, time stored). If the message in section 1 had not been sampled, the table would lack the push row, but the conclusion would be the same: the connection events say Ben’s connection on gateway 12 received nothing after 19:57:30, and Ben’s phone log says it only reconnected at 20:01:40.

The phone keeps a few days of its own logs, which the app uploads when the user sends feedback. The logs hold only IDs, seqs, times and states, never message content.

## 6. The numbers

- **Target**: 400 million messages a day × 0.01% = 40,000 may be late; a downtime budget of 4.3 minutes over 30 days.
- **Probes**: 20 a second, 72,000 an hour; a whole broken gateway shows within seconds, a 0.01% loss does not show.
- **Samples**: 1 delivery in every 1,024, 136 deliveries a second, about 490,000 an hour, enough to measure 99.99%.
- **Logs**: 167 MB a second logging everything; 163 KB a second sampled.

## 7. The cost

- **Probes are fake traffic.** Test accounts have to be removed from the business’s statistics, or they themselves push the success rate up.
- **Phones send a little more.** On average each phone sends fewer than one report a day; 136 frames a second across the site.
- **A single complaint may not have a full trail.** For an unsampled message, the server has only connection events and the message’s own traces; when that is not enough, it takes the logs uploaded from the phone.
- **Logs are data too.** Who may read them and how long they are kept need rules; message content is never logged.

## 8. Other answers

- **Trace everything, keep only the problems**: record everything first, and decide whether to keep it once it is known to be late or failed. The fullest trail, but every hop has to record everything first, so section 5’s bill is paid in full.
- **The receiver ACKs every message**: each push waits for the phone to confirm, and is pushed again if no confirmation comes. Delivery is the clearest it can be, but every delivery adds an upstream frame, 139,000 a second at peak; v3 fills gaps with seqs and sync (chapters 5, 6) and does not need it.
- **Probes only**: the simplest; it catches a whole machine breaking, but it cannot measure 99.99%, and it cannot see real users’ networks.
- **Alert on how fast the error budget burns**, rather than on every error: chapter 38.

## 9. This chapter’s decision

**Decision card**

- Problem: Messages get lost between two hops: the gateway wrote it out, the phone never got it. Each server measures only its own step, so the dashboards are all green; each hop’s logs are kept its own way and cannot be joined up.
- Choice: Measure from the user’s side: online devices receive within 5 seconds, 99.99% over 30 days. Two instruments: a probe through each gateway every second; 1 in 1,024 real messages, picked by message ID, reported back by the receiving phone and timed on the clocks of the gateways at both ends. Every hop’s logs carry the message ID and seq, record only sampled messages, and are buffered in memory, dropped when full, and shipped off the machine; connection events are logged in full; phone logs are uploaded on demand.
- Cost: Probes have to be removed from statistics; phones send a few extra report frames; investigating an unsampled message relies on connection events and phone logs; log access and retention need rules.
- Revisit when: When more comes in than the system can take (chapter 34); SLOs as the gate when deploying gateways (chapter 35); alerting on error-budget burn (chapter 38).
- Other answers: Record everything and keep only the problems afterwards; the receiver ACKs every message; probes only; watching every error.

Now we can see. The next time a dashboard turns red, it may be someone running a script that posts 100 messages a second into the hiking group. When more comes in than the system can take, what do we protect first, and what do we drop first? Next: overload.
