Replication, Read Replicas and Lag
Copies of the database that serve reads, and the window in which they are wrong.
Overview
The arrangement
One server — the primary — accepts every write. It records each change in a log, and streams that log to one or more replicas, which apply the changes to their own copies and serve read queries.
The appeal is that most workloads are overwhelmingly reads. Adding replicas multiplies read capacity without any change to the data model, and unlike [sharding](sharding_in_databases.html) it is reversible: a replica that is not helping can simply be removed.
Replicas also serve as warm standbys. If the primary fails, one is promoted.
Replication, Read Replicas and Lag
Step through a save-then-view. Choose the routing strategy and watch whether the user sees their own change.
Worth knowing
Replication, Read Replicas and Lag
How to serve more reads than one machine can, and the staleness that comes with it.
How far reads actually scale
"Multiplies read capacity" has a limit, and the limit is worth seeing because it decides when replicas stop being the answer. Every replica serves reads *and* must apply every write the primary takes — the write load is not shared out, it is copied to all of them. So if one machine handles 10,000 operations a second, a replica taking W writes per second has only 10,000 - W left for reads, and N replicas serve N × (10,000 - W):
| write rate | 1 replica | 3 replicas | 10 replicas |
|---|---|---|---|
| 500/s (5%) | 9,500 | 28,500 | 95,000 |
| 2,000/s (20%) | 8,000 | 24,000 | 80,000 |
| 5,000/s (50%) | 5,000 | 15,000 | 50,000 |
At a light write rate reads scale almost linearly — ten replicas give nearly ten times the read throughput, which is why read replicas are the standard answer for read-heavy workloads. But look down the columns rather than across: as writes climb, every replica loses that capacity to apply work, and the whole fleet's read ceiling drops with it. Once writes alone approach what one machine can do, 10,000 - W heads toward zero and adding replicas buys almost nothing — each new machine spends all its capacity just keeping up with the write stream.
That is the wall replicas hit and [sharding](sharding_in_databases.html) is built to get past: sharding splits the writes across machines, so it is the tool for a write-bound system exactly where more replicas is the tool for a read-bound one.
Lag
A change committed on the primary is not instantly present on a replica. It has to be written to the log, sent over a network, received, and applied. The gap is replication lag, and it is never zero.
Under normal conditions it is milliseconds. Under load it is not, and the reasons matter:
A large write. A single statement updating a million rows produces a great deal of log to ship and apply.
Single-threaded apply. Some engines apply the log serially even though the primary generated it with many concurrent connections. The replica simply cannot keep up with a busy primary.
A long query on the replica. Applying a change that conflicts with a running read forces a choice between cancelling the query and pausing replication. Both happen, depending on configuration.
Network. A cross-region replica has a floor set by the speed of light.
The bug this causes
The failure mode is specific and extremely common:
1. User updates their profile -> primary
2. Application redirects to profile ->
3. Profile page reads -> replica, 200ms behind
4. User sees their OLD profileNothing errored. The user changed something, was shown the previous value, and reasonably concluded the save failed.
The standard fixes, in rough order of preference:
Read-your-writes routing. After a write, send that user's reads to the primary for a short window. Simple and effective.
Route by criticality. Anything the user just affected reads from the primary; dashboards and search read from replicas.
Wait for the log position. Record the primary's log position at commit and have the replica wait until it has applied at least that far. Correct, and requires engine support.
Do not redirect to a read. Render the result from what was just written.
Synchronous replication
The lag can be removed. synchronous_commit in PostgreSQL, semi-sync in MySQL: the primary does not acknowledge a commit until at least one replica confirms it has the change.
The cost is unavoidable and appears on every write: a commit now includes a network round trip. Throughput falls, and latency rises by the distance to the replica. If the synchronous replica becomes unreachable, writes stall entirely unless a fallback is configured.
That "distance to the replica" is not a figure of speech — it has a floor set by physics, and the floor is worth knowing before choosing where a synchronous replica lives. Light in fibre travels at about two-thirds of c, so the round-trip *cannot* be faster than twice the distance over that speed:
| synchronous replica | round-trip floor | added to every commit |
|---|---|---|
| same datacentre | ~0 ms | negligible |
| same region (~100 km) | ~1 ms | tolerable |
| across a continent (~5,500 km) | ~56 ms | every write waits 56 ms+ |
Real fibre paths run about half again longer than the straight-line distance, so the cross-continent figure is nearer 70 ms in practice. That is per commit, and no amount of hardware removes it — it is the reason a synchronous replica is kept nearby and durability across regions is bought with asynchronous copies plus the accepted risk of the section above.
This is a direct instance of the [CAP](cap_theorem.html) trade: consistency across replicas costs availability and latency, and no configuration escapes it. The usual compromise is one synchronous replica nearby for durability, and asynchronous replicas further away for read capacity.
Failover and lost writes
If the primary fails while a replica is 200ms behind, promoting that replica loses 200ms of committed writes. They were acknowledged to clients and are gone.
Which is why durability requirements and replication mode are the same decision. Asynchronous replication means accepting that a failover can lose the most recent writes.
Where it goes wrong
Assuming replicas are current. They are not, and the window is where the bugs live.
Sending a write to a replica. It is read-only; the error is confusing when routing is implicit.
Monitoring lag in bytes only. Bytes behind does not translate to seconds behind. Track both.
Using replicas for locking or counters. Anything read-then-write must go to the primary, or two clients read the same stale value.
Check yourself
0 of …Answer without scrolling back up.
Why does a user sometimes not see their own change after saving?
What does synchronous replication cost?
What happens if a primary fails while a replica is 200ms behind?
Replication, Read Replicas and Lag
One server — the primary — accepts every write. It records each change in a log, and streams that log to one or more replicas, which apply the changes to their own copies and serve read queries.