Chapter 25
Accounts: the app’s users and the IM’s
The IM is now a service of its own, and the app has long had its own users and login. When Ana’s phone connects to a gateway, how does the gateway know it is Ana, without seeing her password and without asking the app every time?
This chapter is a first draft. The author is still checking it, and its text and numbers may change.
Up to v2, the IM was part of the app: when Ana logged in to the app, she was logged in to chat too. Chapter 20 split out the gateways, and the IM became a service of its own, often run by another team.
Ana’s phone connects to a gateway and sends its first frame. The gateway has never seen the app’s login; how does it know this phone is Ana’s? This chapter answers how the IM recognises the app’s users without seeing passwords and without asking the app on every connection.
1. What the IM needs to know
The gateway needs to know three things: who this is, which of her devices this is, and whether that identity is still valid. Only then can it register “Ana’s phone is on this gateway” in the online routesonline routes在线路由表Which gateway each online device is connected to. Gateways register devices as they connect, and dispatch looks gateways up here.See the glossary (chapter 20).
The most direct way is to let the phone say it itself: “I am Ana”. A gateway that takes it at its word lets anyone pretend to be Ana. Nobody ships that; below are two ways that really do ship.
2. Ask the app every time
The IM does not handle login; the app does. Once Ana logs in to the app, her phone holds the app’s session credential. It brings it when connecting to the IM, and the gateway asks the app’s login service: “Is this credential valid? Whose is it?” The app says Ana, and the gateway registers her.
This is correct: only the app ever sees the password, and revocation lives in one place, the app. The trouble is the count: every connection is a call to another service.
- Normally: every app open is one connection. 10 million people open the app 20 times a day each, about 2,300 a second on average; at v3’s peak, 3 times that, about 6,900 a second. All of these calls land on the app’s login service, which was sized for the app’s own traffic.
- In a reconnect storm: chapter 24 said the admission cap (how many connections the gateways let in each second) is set by the slowest part behind the gateways. Now the slowest is the app’s login service. Suppose it is sized for 1.5 times the normal peak, about 10,400 a second. When a gateway restarts, its 50,000 phones are checked in 4.8 seconds; fine.
- When a zone stops: just checking 500,000 identities takes 48 seconds. Running chapter 24’s model (with admission still on), the time for everyone to connect goes from 43 seconds to 57; for that whole minute, the app’s login service is pinned at its limit.
- When it fails: if the app’s login service goes down, the IM itself is fine, and yet not one person who dropped can get back in.
3. A token the gateway can check by itself
The fix is to have the app’s server vouch for Ana in advance, and the gateway only checks that the proof is genuine:
- Ana is already logged in to the app. Her app asks its own server for an IM token.
- The app’s server signs a token: Ana’s IM user ID, the device, when it was issued and when it expires, signed with the app’s own private key.
- The phone brings it when connecting to a gateway. Every gateway has the matching public key; it checks the signature and the expiry, and if they are right it registers her, calling no one.
A common format for the token is JWT (RFC 7519): sub is who, iat the time it was issued, exp when it expires.
Why a private and a public key rather than one secret both sides share? The gateways only get the public key, and with it they can check but not sign. If a gateway’s configuration leaks one day, all that leaks is the public key, and nobody can use it to pretend to be Ana.
How expensive is checking a signature? An Ed25519 signature takes about 270,000 CPU cycles to verify, about 114 microseconds on a 2.4 GHz CPU from around 2010 (ed25519.cr.yp.to; today’s CPUs are faster). A gateway does at most 2,000 handshakes a second, so checking tokens takes about a quarter of a core, far cheaper than the TLS handshake itself.
4. The IM’s own user ID
Which ID goes in the token? The app’s user IDs are often strings, like ana_92, up to a few dozen bytes. The IM gives each app user a separate 8-byte numeric ID, matched through a mapping table:
- When it is created: when Ana signs up in the app, the app’s server calls the IM once; the IM creates a user and records (app,
ana_92) → 7301…; the app’s server stores this number and writes it into every token it signs from then on. - Why not use the app’s ID directly: every row in the IM carries a user ID. Chapter 12’s member row is 17 bytes, 8 of them the user ID; with a 32-byte string instead, it would be 41 bytes, 2.4 times as big. More important, the same IM will later serve several apps (chapter 44), and two apps might each have an
ana_92. - How big the mapping table is: at v3’s 10 million users, about 44 bytes a row (app number 4, the app’s user ID up to 32, the IM ID 8), about 440 MB.
5. Expiry, renewal, revocation
Expiry. The longer a token is valid, the longer a lost one can be misused; the shorter, the more often it must be renewed. With 15 million devices (10 million people, 1.5 devices each on average), a 1-hour lifetime means about 4,170 renewals a second on average; 1 day, about 174. The app this series follows takes 1 day: renewals land on the app’s own server, and there are few of them.
Renewal. The app gives the IM SDK a “get token” callback. When the token is about to expire, or a gateway says it is wrong, the SDK calls it to get a new one from the app’s server. A failed renewal is no emergency: the token in hand has not expired and still connects. In a reconnect storm almost no token has expired, so the storm does not touch the app. (A device not opened for more than a day is different: its token has expired, and it must renew once before connecting.)
Revocation. When Ana taps “log out of all devices” in the app, or the account is banned, waiting for the token to expire is not good enough. The IM keeps one timestamp per person: tokens issued before this are void. After checking the signature, the gateway checks that the token was issued later than that. Say Ana taps “log out of all devices” at 10:00, so her timestamp becomes 10:00. The token on her old tablet, signed at 9 last night, has a good signature and has not expired, but it was issued before 10:00, so it is void. She logs in again on her phone at 10:05; the new token was issued after 10:00 and connects as usual. The number lives in the IM’s own storage and is read once per connection, without asking the app; 8 bytes a person, 80 MB for 10 million. At the same time, the IM looks up Ana’s open connections in the online routes and closes them one by one.
6. Guests
Viewers in a live room who have not logged in also need to connect to receive messages (chapter 41). Give them a temporary identity: a temporary ID, and a token that says “guest, may only watch this live room”, valid for a short time and not in the mapping table. Once the viewer logs in, they swap it for a normal token.
7. The numbers
- Normally: at v3’s peak, about 6,900 connections a second; asking the app means about 6,900 calls a second; with tokens, 0.
- When a zone stops: asking the app, everyone is connected after 57 seconds (assuming the app’s service is sized for 1.5 times the peak); with tokens, 43 seconds, as in chapter 24.
- What tokens cost: about a quarter of a core per gateway; with a 1-day lifetime, about 174 renewals a second; an 80 MB revocation table.
- IDs: member rows of 17 bytes instead of 41; a 440 MB mapping table.
8. Cost
- The signing private key becomes the most important secret in the system. Whoever has it can be anyone. When the key is replaced, the old and new keys must both be valid for a while.
- Revocation is checked only on connect. Connections already open have to be closed by the IM; any it misses last until they next reconnect.
- One more path, for renewal, and it has to keep working when the app’s server is slow.
- Deleting and merging accounts must reach the IM. When Ana deletes her account, the messages she posted in groups are still in other people’s timelines; showing them as from a “deleted user” or removing them is a product and legal choice. When two app accounts are merged, it is hard for the IM to combine two IDs’ histories into one; usually the old app ID is pointed at the IM ID that stays, and the old IM ID is kept as an alias.
9. Other answers
- The app’s server signs with a shared secret: Tencent Cloud’s UserSig uses HMAC-SHA256 to compute a signature from the app ID, the user ID (up to 32 bytes) and the lifetime, with the app’s secret key; in production it must be generated on the server, and the docs’ example lifetime is 180 days (Tencent Cloud docs). Stream’s Chat also signs JWTs on the server with the API secret, by default never expiring; the SDK renews through a “get token” callback; revocation uses
revoke_tokens_issued_before, compared with the token’siat(Stream’s token and revocation docs), the same thing as section 5’s timestamp. A shared secret is faster to compute; the price is that the IM also holds a key that can sign tokens. - The IM issues tokens, and the app’s server fetches them: OpenIM’s app server calls
get_user_tokenwith an admin token, and the user must already be registered in the IM; it returns a token and its lifetime, 90 days in the example, and the IM keeps track of the token’s state itself (OpenIM docs). - The IM accepts a JWT signed by the app as a login: Synapse, a server for the open protocol Matrix, has a non-standard
org.matrix.login.jwtlogin, withsubnaming the user; the example in its docs verifies with a shared secret (HS256) (Synapse docs). - Use the app’s user ID directly: Tencent Cloud and OpenIM both let the app choose the user ID, saving a mapping table. When the IM serves just one app and the IDs are short, this is simpler.
- The IM manages accounts and passwords itself: users have to sign up a second time, and the IM has to store passwords. Pointless when the IM and the app belong to the same company. When the IM is the product itself, it has to manage accounts: Telegram signs users in with a phone number and a code sent to it (Telegram API: authorization); its public API can log sessions out one by one (chapter 16’s device table) or log out all but the current one (
auth.resetAuthorizations). That revokes session by session; section 5 uses one timestamp to void every token signed before it.
10. This chapter’s decision
The gateway now knows who has connected and where. But a phone is not online all the time: going to the background, switching from Wi-Fi to 4G, being killed by the OS all drop the connection. Next: app lifecycle.
Comments
Comments live in GitHub Discussions and need a GitHub account.