IM Systems in Depth

Chapter 14

Read position and unread counts

Ana read the hiking group on her laptop, yet her phone still shows a red 37 beside it. How should the server keep the number beside each conversation, so that it is exact and the same on every device?

This chapter is a first draft. The author is still checking it, and its text and numbers may change.

Ana read the new messages in the hiking group on her laptop and closed it. On her phone, the red dot beside the hiking group still says 37.

That number is the unread count: how many messages in the conversation she has not read. It has to be right, and what she read on the laptop has to clear on the phone too.

1. First, what does the product need?

Products answer “has she read it?” differently. Ordered by how much extra work the server takes on, there are four tiers:

Tier Cases What the server does
Up to the client Read when the chat opens, or only when a message is on screen; show a number, “99+”, or only a dot Nothing extra: the phone reports “read up to message N”
One number is enough All of a person’s devices agree; messages from before joining don’t count One read position per person per conversation (section 4)
One number plus a little An exact count, yet system messages and your own should not count (section 5); jumping to an @ while the messages in between stay unread; “mark as unread” (section 6) Something kept beside the read position
Another product For every message, who has read it and who has not (workplace chat) Per-message state, another design (chapter 39)

The trunk product is a social app with groups of up to 500. It chooses: per conversation, every device agrees; opening a chat lands on the first unread message; the count is exact; sending a message does not mean you have read everything above it.

2. Who counts: the phone or the server?

The phone counts The phone counts, starting from the server’s rough number The server counts
How Count the unread messages in the local database that should count On each resync, take the server’s rough number (latest seq minus read seq); while online, add one for each new message from others that counts The server keeps and works out the count, and sends it with the conversation record
Good The phone knows what each message is, so system messages and your own are easy to leave out Right while online; a simple server Every device agrees; the count exists while the app is closed
Hard Every unread message has to be on the phone; each device counts on its own Two ways of counting: the rough number includes system messages, so the count jumps at each resync The server has to know which messages count (section 5)
Example End-to-end encrypted apps (chapter 42): the server cannot see the content, cannot tell which messages count, and leaves it to the phone OpenIM’s SDK Telegram: each conversation carries an unread_count (dialog)

The trunk counts on the server, for two reasons:

  • The phone does not have every message. Chapter 6 settled that on return the phone pulls only the newest message of each conversation it is behind on. A week without opening the hiking group is 7,000 unread messages; to count them itself, the phone would first download them all, about 200 bytes each, 1.4 MB, just to show one number.
  • The count is needed while the app is closed. The badge on the app icon can only come from the server, inside an OS push (chapter 17).

3. The first idea: a counter

Counting on the server usually starts with a counter: one unread number per person per conversation, plus one for each message, zero when read. It goes wrong in two places:

  • A redone to-do adds twice. In chapter 11, a message is first written down as one “to-do” row, and a background job then works through it, updating each member’s conversation record. If the job crashes halfway, after the restart it does the whole to-do again, so the members it had already reached are updated a second time. Writing “latest seq = 4,812” twice still leaves 4,812, which is why that was fine. But “unread + 1” done twice turns 5 into 7: one message counted twice.
  • The reset arrives late. Ana reads on the laptop and closes the lid; the “read” report waits until the laptop is back online. Meanwhile 3 new messages arrive in the hiking group and the counter goes up by 3; when the laptop reconnects, the reset finally lands and the counter goes to 0, though she has seen none of the 3.

A counter knows how many are left, not where she has read to. Once it is wrong, nothing can recompute it; it stays wrong until the next reset.

4. One number: the read position

On each of Ana’s conversation recordsconversation record会话记录One row per person per conversation: pinned, muted, the conversation’s latest seq, and later how far they have read. Written on join, leave or a settings change, and from chapter 11 on also when a message arrives (with the latest seq). Each change bumps that person’s record version. The messages are not here.See the glossary (one row per person per conversation) the server adds a read position (read_seq, her read cursorread cursor已读游标How far a person has read in each conversation. It lives on the server, is shared by that person’s devices, and drives unread counts and read receipts. Not the same as the sync cursor.See the glossary): she has read everything up to this seq. Chapter 11 already keeps the latest seq on the same row, so:

unread = latest seq − read position

For now, assume the conversation holds only ordinary messages from other people. Ana’s own messages and system messages take a seq too; section 5 takes them out.

The rules:

  1. The phone reports what it has shown, at most once a second for an open chat.
  2. It only moves up. UPDATE … SET read_seq = N WHERE … AND read_seq < N. A late or repeated report changes nothing.
  3. When it moves, the record version moves. The record version is chapter 11’s per-person number that only goes up: whenever one of Ana’s conversation records changes, it gets a new number taken from her row in the users table, and each of her devices pulls only the records above the number it has. So once she reads on the laptop, the phone gets it at its next sync, and the dot clears.
  4. Counting starts at the join. A new member’s read position is set to the seq of the system message for the join. The messages chapter 12 lets newcomers see (the “last 100 before you joined” setting) all come before it: visible, but not unread.

One more thing to get right: a read can run ahead of the record. Ana replies in the group; her phone shows the reply at once and reports “read up to here”, but the to-do writes the latest seq into her record a little later. Say the reply is 4,823: the record still says latest 4,822, the read position is already 4,823, and unread = 4,822 − 4,823 = −1. The fix: the read’s transaction also writes latest_seq = GREATEST(latest_seq, N).

Whether Ana’s own message should move her read position forward is also the next section’s question.

5. Second step: count only what counts

This is the price of counting on the server: the phone knows what each message is, the server only knows its seq. Every message has a seq, including two kinds that should not count:

  • System messages: joins, leaves, a new group name; chapters 12 and 13 gave each a seq.
  • Ana’s own messages: her reply is not something she has to read.

A shortcut: when you send a message, move your read position to it. OpenIM’s server (separate from the SDK in section 2) does this: when it stores a message, it sets the sender’s read seq to that message’s seq. That is fine when Ana reads and then replies in the chat. But if she replies straight from the notification, everything above it that she never saw is marked as read.

So each message carries two more numbers, written in the same transaction that takes its seq (chapter 5):

  • The count so far: how many countable messages the conversation has, up to and including this one. A system message does not add one; it repeats the previous number. The count is shared by the whole conversation, so Ana’s own messages add one too, and are taken off when working out her unread count.
  • The sender’s running number: this is the sender’s Nth message in this conversation.
seq 4,821 4,822 4,823 4,824
What message Cara joined Ana’s message message
Count so far 101 101 102 103
Sender’s running number – – 7 –

Ana has read up to 4,820; subtracting seqs says 4 unread. The right answer takes four numbers, all kept on her conversation record:

  • the latest count and her latest running number: written by the to-do when it updates her record, only moving up. Here 103 and 7.
  • the count at her read position: looked up from message 4,820 when she reads. Here 100.
  • her running number at her read position: looked up from her last message at or before 4,820, with an index on (conversation, sender, seq), one query. Here 6.

unread = (103 − 100) − (7 − 6) = 2: messages 4,821 and 4,824.

The server works the count out from these four numbers whenever it sends the record. Section 4’s “read ahead of the record” applies here too: the read’s transaction also raises the latest count to the count at her read position if that is larger, and her latest running number likewise.

Here is one hour of the hiking group. Ana reads on her phone a few times, replies in the chat a few times, and twice replies from the notification; she reads twice on the laptop, and both reports arrive 4 minutes late:

MessagesAna readsUnread3800 min10 min20 min30 min40 min50 min60 min

others’ messagesystem message Ana replies in the chatAna replies from the notification phone readlaptop read, report late true unreadred dot showslaptop’s report on its wayminutes that disagree

Of 62 minutes checked this hour, the red dot disagreed with the truth in 46, by up to 3 (too many); it ends at 0, the truth is 0.

The dashed line is her true unread count, the solid line what the server shows. Minutes when a message or a report is still on its way don’t count; of the 62 minutes left:

  • A counter: 46 disagree. Redone to-dos count too many; late resets too few.
  • Read position, the last report wins: this is what it looks like without “only moves up”. The laptop’s old report arrives last and drags the position back; it ends at 7 when the truth is 0.
  • Read position, only up: the devices agree, but system messages and Ana’s own are counted.
  • Count what counts, with sending marking the chat as read: the two replies from the notification swallow 3 and 19 messages.
  • Minus her own: right in every minute.

6. Where one number is not enough

Jumping to an @. Someone @mentions Ana at 4,900; she has read up to 4,800. She taps the @ button and jumps straight there; the 99 messages in between she has not seen, so the read position must not move to 4,900. An @ is kept as a row of its own (who, which conversation, which message), deleted once seen; the @ badge counts only the rows above the read position, so once the position passes them they no longer count. Telegram does the same: each conversation has an unread_mentions_count, the @ button steps through the mentions, and they are cleared once the last one has been read (mentions).

“Mark as unread”. The read position cannot go back: the only-up rule forbids it, and another device would report it forward again. Keep a flag instead and clear it on the next read. That read may bring no new messages, so write it as SET read_seq = GREATEST(read_seq, N), marked_unread = false … WHERE read_seq < N OR marked_unread: the flag clears and the position still does not go back. Telegram also keeps a separate unread_mark.

Ordinary messages are harder: the stretch you skip when jumping to an @ cannot be held in one number. Matrix, an open chat protocol, uses two positions to split a room’s history into three parts: read, possibly partly read, and not yet seen (read markers). Anything finer means keeping state per message.

7. The numbers

  • The phone: a conversation record grows from chapter 11’s 28 bytes to 39: read position 4, unread count 4, unread @ count 2, “marked unread” 1.
  • Reports: a report covers every message before it, and a report that does not move the position writes nothing, so there are at most as many reports as messages received. At the v2 peak, 139 messages a second each reach 6.67 people on average (chapter 11): about 927 reports a second.
  • Rows written: each report changes two rows, Ana’s users row (for the record version) and the conversation record: 1,853 rows a second. With chapter 11’s 1,853, that is 3,707 rows a second, 18.5% of what one storage node can write (20,000 rows a second).
  • Storage: 8 more bytes per message; at 4 million messages a day, 32 MB.

The figure’s “Another hour” button picks a different hour. Over 200 of them, the counter disagrees in about half the minutes and in 108 of them it ends the hour on a wrong number; this chapter’s design is never wrong.

8. Cost

  • Reading is writing. Each move of the read position bumps the record version, and the other devices sync once more.
  • One more lookup to send, two to read. Sending takes the sender’s running number (in the same transaction that takes the seq, chapter 5); reading looks up two things.
  • @everyone is expensive. In a group of 500 it is 500 rows; a flag on each person’s conversation record (“someone mentioned everyone”) saves them.
  • An old report can clear “mark as unread”. A report of a read made long ago on the laptop arrives late and clears a mark just set.

9. Other answers

  • The phone counts, or counts starting from the server’s rough number (section 2). For apps whose phones download every message, counting on the phone is simplest: no count so far, no running numbers.
  • A counter, reconciled now and then. Simple; slightly off, corrected from “latest seq − read position” from time to time.
  • Sending marks the chat as read (section 5). One running number fewer; replying from the notification swallows what was not seen.
  • A dot, not a number. Nothing to count, subtracting seqs is enough; many apps do this for muted chats.
  • Per-message read state. A row per person per message: a 500-member group with 1,000 messages a day makes 500,000 rows a day, against 500 read positions. Worth it when a workplace app must know, for every message, who has not read it (chapter 39).
  • Read ranges. A few ranges per person per conversation, such as [1–40] and [45–50], which remember jumps too.

10. This chapter’s decision

Chapter 14: each person’s conversation record gains a read position that only goes up, one number for all their devices; each move bumps the record version, so the other devices get it with conversation sync. The server works out the unread count from a number given only to messages that count, minus the reader’s own, and sends it with the record; an @ is kept as a row of its own.
Anaoutbox (local)Ana’s laptopreads synced tooBenrecord versionmessage → ← ACKpush + seq;return withrecord version:only changesone programConnectionholds connections; user → connectionsMessageseqs for messages, joins, leavesDispatchto-do: members whose range holds the seqBusiness (beside)groups, members, rolesmember versionStorageunique keys, recordssender+msg IDconv.last_seqrecords+readto-do: member rowsphoto uploaded in parts (an upload service stores it)upload addressObject storage + CDNwhole files by file idBen sees a placeholder first, the full photo on tap

Ana’s own read position is now clear to the server and every device. But what about the messages she sends? Has Ben read them? How many in the group have? Next: read receipts.

Comments

Comments live in GitHub Discussions and need a GitHub account.