# Chapter 37: Losing a data centre

> v3’s 20 gateways are split across two data centres. One loses all power, and the other can hold everyone. Every message is stored 3 times, so why can half the conversations still not send when one data centre is lost?

IM Systems in Depth · https://im.liko.page/en/losing-a-data-centre/

In April 2023, water leaked into a data centre building in Google Cloud’s Paris region, a fire followed, and the whole building lost power. The region had three buildings, each with its own power, cooling and network. But the replicas of a key database in the region (Spanner) were not correctly spread over the three, so losing one building left no majority, and many of the region’s services went down together ([Google Cloud incident report](https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY)).

v3 is split across two data centres too. Chapter 22 put the 20 gateways in two **availability zones** in the same city (two groups of data centres with independent power and networks), 10 on each side, so that if one side is lost the other can hold everyone; chapter 24 worked out how 500,000 devices reconnect.

This chapter’s question: **the connections survive, but what about the log, the storage and the leases? When one data centre is lost, are they still there, and can they still be written?**

## 1. First, what exactly is lost

| What is lost | How often | Who handles it | Where |
|---|---|---|---|
| One machine | Often | The system itself, within seconds | Chapters 28 and 32 |
| One data centre (availability zone) | Rarely | The system itself, losing no acknowledged message | First half of this chapter |
| A whole city (region) | Very rarely | A person decides to switch over; the last second may be lost | Second half of this chapter |

## 2. Two data centres, and no majority

Everything in v3 that must be safely written before it counts is stored 3 times, and a write is answered only once a majority of the 3 (2 copies) have it:

- **The message log**: the ACK comes after the write reaches 3 machines (chapter 29).
- **Storage**: each piece of data has 3 replicas, and we assume it too counts as written once a majority have it.
- **Leases**: to own a conversation, a message server must keep renewing a time-limited permit (chapter 28). The permits are kept in a coordination service, which also works by majority vote.

Put 3 copies in two data centres, and one side always holds 2. If the side with 1 copy is lost, no harm done; if the side with 2 is lost, the 1 copy left is not a majority, and that replica group can no longer take writes.

The log has many partitions, and half of them have their 2 copies on one side, half on the other. So whichever data centre is lost, half the partitions cannot be written: at peak, 6,950 messages a second cannot be sent. The lease service is worse: it is a single group, and its majority sits on a fixed side. Lose that side and no lease can be renewed; an owner that cannot renew stops by itself when its lease runs out (chapter 28), so 10 seconds later every owner stops, and **all 13,900 messages a second fail to send**, including from the perfectly healthy phones in the other data centre.

More copies don’t help. With two data centres, however many copies there are, losing the side with more leaves less than half. The documentation of etcd (a widely used coordination service) has this table: a 3-member cluster tolerates 1 failure, and a 4-member cluster still only 1 ([etcd FAQ](https://etcd.io/docs/v3.5/faq/)).

So why not let the 1 remaining copy carry on writing by itself? Because it cannot tell “the other side is gone” from “the network between us is cut”. If the network is cut, each side thinks the other is gone and carries on writing, and one conversation ends up with two owners, and two different messages get the same seq (chapter 28). That is the point of a majority: at most one side can have one.

## 3. A third data centre, one copy in each

The fix is a third availability zone: for the log, storage and the lease service, each group’s 3 copies go in 3 data centres, one in each. Whichever is lost, 2 copies remain, still a majority, still writable.

The third data centre holds no gateways; users still connect only to the first two. It holds only the third copy of each group. There were always 3 replicas, so what is new is not more machines but a placement rule: **never two copies of the same group in the same data centre.**

The cost is on every ACK. At least one copy in the majority is in another data centre, so the ACK waits for one more round trip between data centres. For availability zones in the same region, AWS says they are up to about 100 km apart, with single-digit milliseconds of latency, close enough for synchronous replication ([AWS whitepaper](https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/availability-zones.html)). At 1 millisecond, against the phone’s 200 milliseconds to the server, that is 0.5% more.

## 4. What the system does by itself when a data centre is lost

Say that at peak we lose one of the data centres with gateways:

1. **Not one acknowledged message is lost.** Each is on a majority, with at least one copy in another data centre.
2. **Owners change.** Conversations whose owners were in this data centre, say half of them, wait for the leases to expire; after 10 seconds other message servers take over and carry on from the last seq in the log (chapter 28). In those 10 seconds about 69,500 messages (6,950 a second × 10 seconds) go unacknowledged, and the phones resend them, as in chapter 3.
3. **Devices reconnect.** This data centre’s 500,000 devices connect to the gateways in the other one (chapter 24).
4. **The routes correct themselves.** 500,000 online routes point at dead gateways; as each device reconnects it overwrites its own entry, with no copying and no repair. If the machines holding the routes were in that data centre too, the gateways on the other side register their connections all over again: a gateway knows best which connections it holds.

Nobody has to press a button at any point.

## 5. When a whole city is gone

Take a round trip between two cities as 30 milliseconds. With copies that far apart, things change.

First, each conversation gets a **home**: it belongs to one city, where it takes its seq and writes its log, and is then replicated to the other city.

**Replicate before the ACK**, and every message waits 30 milliseconds more, 15% of the phone’s round trip, paid on every message, every day.

**Replicate after the ACK**, and there is no extra wait normally, but the replica is always a little behind. Say it is 1 second behind and each city is home to half the conversations: if one city is lost at peak, the other is missing about **6,950** messages that were already acknowledged.

The social app this series follows replicates after the ACK: losing a whole city is very rare, and that second of messages is worth it compared with every message being 30 milliseconds slower. Products where no message may ever be lost, such as workplace chat (chapter 39), may choose the other way.

Switching to the other city takes **a person to decide**; it is not automatic. Seen from the other side, a cut network between the cities looks the same as a lost city; switch automatically, and both sides might act as home at once.

After the switch, **the new owner’s seq has to jump far ahead**. Take the hiking group:

- The lost city had already sent 4,824 to 4,826, and pushed them to Ben’s phone;
- The other city had replicated only up to 4,823.

If the new owner carried on from 4,824, Ben’s phone would have two different 4,824s, and he would never fetch the new one. So the new owner jumps 10,000 ahead of 4,823, and the next message is 14,824. The stretch in between counts as empty numbers: a phone that sees a big gap in the seqs asks to fill it (chapter 5), the server answers once, “4,824 to 14,823 do not exist”, and the phone skips over it. In chapters 5 and 28 an empty number is answered one at a time, “this number does not exist”; here it is a whole stretch.

How far to jump? More than the most messages a conversation can send during the replication lag. Chapter 34 has two limits: 5 messages a second per device, and at most 5,000 recipients a second per conversation. A group of m members can send at most 5,000 ÷ m a second under the conversation limit; under the device limit, even all its members’ devices (1.5 each) sending together make only m × 1.5 × 5. The lower of the two applies: small groups hit the device limit, big ones the conversation limit, and the most is where the two meet, at about 26 members, about 192 a second; a jump of 10,000 covers 52 seconds of lag. Replication lag is measured all the time; if it is further behind at the moment of the switch, jump further. The step is the same as the step WeChat’s seqsvr uses to pre-allocate ranges of numbers ([InfoQ](https://www.infoq.cn/article/wechat-serial-number-generator-architecture)), and it too jumps a stretch ahead like this when it restarts.

So chapter 28’s “carry on from the last seq” holds only within one region: there the log’s majority is written synchronously, and the last seq is sure to be complete. Across cities replication lags, the last seq cannot be trusted, and the only choice is to jump.

## 6. The cost

- **A third data centre.** One more site to rent; the log is replicated between data centres: 400 million messages a day × 200 bytes = 80 GB, and two of the three copies sit in other data centres, so about 160 GB a day; every ACK waits about 1 millisecond more.
- **Two cities, twice the capacity.** Each city has to hold everyone, so normally half sits idle.
- **A cross-city switch may lose the last second.** About 6,950 acknowledged messages, recoverable only when that city comes back.
- **Recovered messages land in the past.** They have their own seqs, right inside the skipped stretch, but phones that have moved on won’t go back to fetch them; they have to be told separately.
- **A person deciding is slow.** From noticing to deciding to switch may take minutes or tens of minutes. That has to be practised in drills until it is routine (chapter 38).

## 7. Other answers

- **Two data centres plus a “witness”**: the third site holds only a voting member, no data. Cheap, and fits the lease service; but where the log and storage need 3 copies of the data, they still need a third copy.
- **Primary and standby**: normally only one region is used; the other only receives replication and takes no users. Simple, but the standby has never really run, and the switch is where things most often go wrong.
- **Both cities take users, one home per conversation** (this chapter’s approach); some set the home per user instead, but when a conversation’s members live in two cities, sending crosses cities. Telegram’s API works this way: each user’s data lives in one of several data centres around the world, and a request sent to the wrong one gets `USER_MIGRATE_X`, where X is the data centre to go to ([Data centers](https://core.telegram.org/api/datacenter), [Error handling](https://core.telegram.org/api/errors)).
- **Synchronous replication across cities too**: 30 milliseconds more per message, not one lost; suits products such as finance and workplace apps.

## 8. This chapter’s decision

**Decision card**

- Problem: v3 runs in two data centres, and losing one side’s gateways is fine; but the log, storage and leases each need a majority of 3 copies, and one data centre always holds 2. Lose it and half the conversations cannot be written, and once the leases expire nobody can send.
- Choice: Within one region, use three availability zones with one copy of each group in each, and the ACK waits for a majority (synchronous): losing a data centre loses no acknowledged message and needs nobody to act. Across cities, replicate after the ACK; a person decides on the switch, and the new owner’s seq jumps 10,000 ahead.
- Cost: A third data centre, about 1 millisecond more per ACK; two cities, twice the capacity; a cross-city switch may lose about 6,950 acknowledged messages, and recovered ones land in the past; a person deciding takes time.
- Revisit when: Products where no message may be lost (chapter 39); whether the switch and recovery have really been practised (chapter 38).
- Other answers: Two data centres plus a witness; primary and standby; a home per user (Telegram); synchronous replication across cities.

By now, most of the fallbacks that chapters 33 to 37 added (circuit breakers, degraded modes, draining, switching data centres) run only on bad days; only chapter 33’s probes and sampling run all the time. Things that have never run will most likely not work when the day comes. Next: drills and postmortems.
