Images and files

By chapter 7, text messages on the phone are no longer lost, repeated or out of order, and a killed app loses none of them. But chats are not only text.

Ana climbs to the top of the mountain, sends Ben a photo, then three lines: “Made it!”, “So windy”, “Dinner when I’m down?”. Up there she has one bar of signal.

Every mechanism so far was built for 200-byte messages. This chapter asks: can a photo take the same road? If not, which one?

1. How big is a photo

Take a photo from the phone’s camera as 4 MB; before sending, apps usually shrink and recompress it, say to 200 KB. Both numbers are this chapter’s assumptions; phones and apps differ a lot, but the order of magnitude holds.

• The compressed photo is 200,000 ÷ 200 = 1,000 times a 200-byte text message; the original is 20,000 times.
• Upload time on a weak network like the mountain top, assumed 2 Mbit/s up, that is 250 KB a second: the original takes 16 s, the compressed one 0.8 s, a text under 1 ms. At a better 8 Mbit/s: 4 s and 0.2 s.
• Side trip W1 capped one frame on the message connection at 64 KiB. The original is 61 times that, the compressed photo 3.1 times. Sending the original on the message connection means raising the cap to about 4 MiB; in W1’s worst case, with each of v1’s 10,000 connections stuck holding half a message, that is 10,000 × 4 MiB ≈ 41.9 GB of buffers, against 0.66 GB at 64 KiB.

To be fair: with compressed photos and a good network, a photo holds the connection for 0.2 s, putting it in the message connection works, and chapters 3 and 4 give it the outbox, ACKs, resends and dedup for free. The trouble is that people also send originals, videos and files of tens of MB, and the signal is often poor. So look at the hard case: a 4 MB original, a weak network, one drop in the middle.

2. Watch it fail

Ana sends the photo (the 4 MB original), then a line at 3, 7 and 10 s. The uplink carries 250 KB a second. At 12 s the network drops for 2 s; with chapter 2’s 0.6 s handshake on top, the connection is back.

Ana sends a 4 MB original photo, then three texts while it is on its way. Three quarters of the way through, the link drops for 2 s. The photo and the texts share the message connection: when do the texts reach the server?

Each in about 0.1 sAs soon as the link is backOnly after the photo is through

↑ Guess first

bytes sentcut by a drop, sent again a text waitingdrop and reconnect

Longest a text waited

–

Photo message stored

–

Uploaded for the photo

–

Photo on the message connectionone big message, texts queue behindSeparate upload, one requesta drop starts it overSeparate upload, in parts512 KiB parts, resumes after a drop

No dropsOne dropFlaky

The default is “Photo on the message connection”: the photo is one 4 MB message sharing the connection with the texts.

• The texts queue behind the photo. The bytes on one connection go in order (side trip W1: TCP is one byte stream, and one frame must finish before the next), and the outbox sends in order too. The three lines were sent at 3, 7 and 10 s, but they wait for the photo. This is head-of-line blocking: one big item at the front holds up everyone behind it.
• One drop, and the photo starts over. By the drop, 250 KB/s × 12 s = 3 MB has gone out, but half a message is useless to the server; when the connection is back, the outbox sends the 4 MB again from the first byte. The texts wait all over again.
• The result: the photo is stored at 30.7 s, and the three lines wait 27.7, 23.7 and 20.7 s; the phone sends 3 + 4 = 7 MB for this photo, 3 MB of it for nothing.

Switch to “Flaky” (a drop every 8 s): each time the connection comes back, 5.4 s remain, not enough for a 16 s photo. The photo never finishes, not one line gets out, and in 60 s Ben receives nothing while the phone has sent 13.6 MB. On a train or in a lift, the user does not feel “the photo is slow” but “the whole chat is stuck”.

3. Estimate

A drop at 70%: how much is sent again. The original takes 16 s on the weak network; at 70%, 11.2 s in, with 2.8 MB sent, the link drops.

• One request: half a file is useless to the server; when the connection is back, the whole 4 MB, 16 s goes again, and the 2.8 MB already sent is wasted.
• In parts: parts of 512 KiB (524,288 bytes, Telegram’s largest part), each kept by the server once complete. At the drop, 5 parts (2.62 MB) are complete and the 6th loses 179 KB. Back online, the phone asks “how far did you get?” (one round trip, 0.2 s) and sends the remaining 1.38 MB, 5.51 s. Wherever the drop falls, it costs at most one part, 2.1 s of uploading.

When drops are frequent. Say drops come at random, on average every T seconds; each one costs R = 2.6 s with the reconnect, and then the upload starts over. A stretch of u seconds of sending gets through without a drop with probability e^(−u/T), so on average it takes e^(u/T) tries; counting the time each failed try wastes plus R, it takes (T + R)(e^(u/T) − 1) seconds on average (the standard result for a task that restarts on failure).

• T = 30 s: the 16 s original in one request takes 23.0 s on average (the chance of no drop in 16 s is e^(−16/30) ≈ 59%); in parts of about 2 s each, 18.0 s.
• T = 8 s: one request takes 67.7 s on average, because the chance of no drop is only e^(−2) ≈ 14%, so nearly every try is wasted; in parts, 24.2 s.

The shorter each piece, the likelier it gets through before the next drop.

The bytes the server gains. Assume 5% of v1’s 4,000,000 messages a day are photos (an assumption):

• 4,000,000 × 5% = 200,000 photos a day, × 200 KB = 40 GB. All the text in a day is 4,000,000 × 200 bytes = 0.8 GB. Photos are 5% of the messages and 50 times the bytes of the text.
• A year is 14.6 TB, 43.8 TB with the book’s 3 copies; text is 292 GB a year, 0.88 TB with 3 copies.
• At peak, 139 messages a second, 6.95 of them photos: 11.1 Mbit/s of uploads. Downloads are larger: a message reaches 10 devices on average, and if each downloads the full photo, that is 111 Mbit/s. Chapter 2 found that pushing all the text at peak takes 2.2 Mbit/s.

A message is small, must be fast and in order; a photo is large, written once and never changed, and can be cached: they belong in different places, rather than growing the message table 40 GB a day and keeping the message server’s network card busy moving photos.

4. The fix: the file takes its own road

Sending a photo becomes a few steps:

1. Compress. The phone shrinks and recompresses the photo (4 MB → 200 KB), computes its SHA-256 hash, width and height, and makes a tiny placeholder. If the user ticked “original”, it skips the compression.
2. Ask for an upload address. On the message connection the phone asks: “I want to upload a 4 MB file.” The server runs the business checks (may Ana send files, is it under the size limit), assigns a file id and returns an upload address. The address points to the upload service below and carries a time-limited token signed by the server: whoever holds it can upload this one file there, without any other credentials. It is the same idea as S3’s presigned URLs.
3. Upload in parts, resume after a drop. The phone opens another connection and sends the file part by part. What remembers “which parts have arrived” is an upload service in front of object storageobject storage对象存储A storage service that keeps whole files by name (S3, for example); images and files go here, not into the message table. The phone gets a short-lived upload address and uploads to it directly; downloads usually go through a CDN.See the glossary: once it has all the parts, it puts them together and stores the whole file. After a drop, the phone asks the upload service how far it got and carries on from there. This is a resumable uploadresumable upload断点续传A file is uploaded in parts and the server remembers which have arrived; after a drop the phone asks how far it got and sends only the rest instead of starting over.See the glossary; tus is a ready-made open protocol for it: POST creates an upload, PATCH with Upload-Offset writes into it, and after a drop a HEAD tells how much the server has.
4. Send a small message. With the file uploaded, the phone sends an ordinary message on the message connection, whose content is a reference to the file:

{ "type": "image", "file_id": "f_7Kq2…", "size": 4000000, "sha256": "9f2c…",
  "width": 4032, "height": 3024, "mime": "image/jpeg", "blurhash": "LKO2?U%2Tw=w]~RBVZRi};RPxuwH" }

About 300 bytes, the size of a line of text. It takes the road chapters 3 to 7 fixed: outbox, ACK, resend, dedup, seq, sync, all of it. The message server checks that the file id exists and was uploaded by Ana. The size and hash can be trusted only if the server computed them itself: the upload service computes them once it has every part; or the presigned URL signs the size and SHA-256, and S3 computes the checksum itself and rejects the upload if it does not match. Otherwise a client can lie about the hash. 5. Ben downloads on demand. The message reaches Ben’s phone. The bubble reserves space from the width and height and decodes the blurhash into a blurred placeholder (BlurHash describes an image’s rough colours in 20 to 30 characters); then it downloads a thumbnail of about 10 KB; only when Ben taps does the full photo come. Downloads go through a CDN: a finished file never changes, so the CDN and the phone can cache it freely.

In the simulator above, switch to the middle “Separate upload, one request”: each line takes only 0.1 s, no longer queued behind the photo; but the photo still starts over after the drop and is stored at 30.9 s. Switch to “Separate upload, in parts”: the photo is stored at 20.6 s, 10.3 s sooner, with 4.18 MB sent and only 0.18 MB wasted. On “Flaky”, the one-request photo never finishes in 60 s; the one in parts is through at 28.5 s.

With no drops, the separate upload is a little slower than the message connection: the photo is stored at 17.1 s instead of 16.1 s. The extra is the round trip to ask for the address (“address” in the figure), a handshake for the upload connection, and the final small message: the fixed price of two roads, about 1 s a photo.

What downloading on demand saves. Assume 30% of recipients tap to see the full photo (another assumption). The server’s downloads a day fall from 2,000,000 × 200 KB = 400 GB with every device downloading the full photo to 2,000,000 × (10 KB + 30% × 200 KB) = 140 GB, and at peak from 111 to 39 Mbit/s, most of it carried by the CDN. Ben receives 267 × 5% ≈ 13 photos a day; his downloads fall from 2,670 KB to 935 KB. All his text is 53 KB.

5. The outbox gains a step

Before, a message in the outbox had one step: send, wait for the ACK. A photo message has two: upload, then send. Ana’s screen gains states:

• Uploading 37%: the photo sits where she sent it, with a progress bar;
• Sending: the file is up, the small message waits for its ACK;
• ✓ Sent, or ! Failed. A failure must say which step: if the upload failed, a retry continues the file; if the send failed, the file is already on the server and a retry sends only the 300-byte message.

These states live in chapter 7’s local database: the messages row’s state gains 'uploading', and an uploads table keeps the local file’s path, its hash, the file id, the upload address and its expiry, and the number of parts confirmed so far (parts_done).

Chapter 7’s rule still holds: each state change is committed in the same transaction as the fact it stands for. When the file is up, setting state to 'sending' and recording file_id are one transaction. parts_done is only the phone’s own note; what counts is the upload service’s answer. If the app is killed halfway, on the next launch it finds the 'uploading' row, asks the upload service how far it got, and carries on. If the upload address has expired, it asks for a new one.

On the receiving side, photos live in the app’s cache folder and the database keeps only the file id and the path; photos come to about 487 MB a year (1.12 GB if every one were downloaded in full) against 33.6 MB of text, which is why chat apps have “clear cache”.

Upload the same file only once? A forward keeps the same file id; the server checks that the forwarder can see the file, and not a byte is uploaded. Deduplicating by hash across all users (“a file with this hash is already here, skip the upload”) also saves uploads, but it tells anyone whether someone else has uploaded a given file: a side channel that leaks information (Harnik et al., 2010). Whether and how to deduplicate safely is a separate decision, not this chapter’s.

6. The cost

• The two roads must agree. With the file and the message apart, one can succeed while the other does not:
  • The file is up but the message never goes (the app was deleted, the user cancelled): object storage holds a file nobody refers to. It must be cleaned up, say, deleted if no message refers to it within a few days.
  • The message arrives but the file does not: Ben sees an image that will not open. So the phone must wait for the upload to be confirmed before sending the message, and the server must check the file exists when the message comes.
• The order changes. The photo message goes out only after the upload, so its seq (chapter 5) comes after the three lines sent meanwhile. Ana’s screen shows [photo, Made it!, So windy, …]; Ben sees [Made it!, So windy, …, photo]. Either move Ana’s photo down when the ACK comes, to match Ben, or accept that the two see different orders. Another way is to send a placeholder message first (“Ana is sending a photo”), which takes its seq the moment she taps send, and turn it into the real photo when the upload is done; that needs “editing a sent message”, chapter 16.
• Who can download. A file id or download address is itself the permission: whoever holds it can download. The usual answer is a signed, time-limited download address: the CDN checks the signature and expiry at the edge, then serves the same cached file, as with CloudFront signed URLs. The cost is a coarse check: until it expires, anyone holding the address can download, and after Ben is removed from the conversation, the addresses he already has keep working until they expire. Matrix moved to authenticated downloads in version 1.11 (/_matrix/client/v1/media/download/…) and deprecated the unauthenticated /_matrix/media/v3/download; but it checks only that the user is logged in, not that they are in the room.
• More systems to run. The upload service, object storage, the CDN, signing upload addresses, cleaning up expired files. Storage grows by 43.8 TB a year, so someone must decide how long photos are kept.

7. Other answers

• Keep one connection, but cut the photo into small frames interleaved with the texts. HTTP/2 does this: one connection carries many streams whose frames can be interleaved, so a large file does not block small requests. That removes head-of-line blocking and allows resuming. The cost: every photo byte still passes through the message server, the 40 GB a day and 111 Mbit/s peak included.
• Telegram uploads files part by part with MTProto’s upload.saveFilePart: per its documentation, parts are at most 512 KiB, and such requests are best kept on separate sessions and connections that run nothing else. The same idea in its own protocol. A message carries an “extremely low-res thumbnail” (photoStrippedSize), which plays BlurHash’s role.
• Matrix: its content repository is a separate upload service. The client calls POST /_matrix/media/v3/upload and gets back an address of the form mxc://<server-name>/<media-id>; an m.image message carries only that address, plus width, height, type, size and a thumbnail in info.
• Straight to S3: S3’s multipart upload needs parts of at least 5 MiB, and its documentation suggests multipart from about 100 MB; so a 4 MB photo sent straight to S3 is one PUT that starts over after a drop. Small resumable parts need your own upload service in front.

8. This chapter’s decision

Every piece of v1 is now in place. The next chapter puts them together: it follows one message along the whole path, switches the mechanisms of chapters 3–8 off one at a time and shows, with each chapter’s own simulator, which failure comes back; then it redoes the server’s and the phone’s accounts from v0 to v1.