Two primaries accepting writes is the worst thing a database can do
Split-brain is when two nodes in a Postgres cluster both believe they are the primary and both take writes. Now you have two divergent histories of the same table, and there is no clean automatic merge for that. A human picks a winner, and the losing partition's writes are gone. The entire job of a high-availability setup is to never reach that state. The machinery that prevents it is a consensus store holding a quorum vote, and if you get the node math wrong, you have built the exact failure you were trying to design out.
The standard answer is Patroni as the cluster manager plus etcd as the distributed configuration store. Patroni decides who is primary; etcd is the source of truth every node agrees to obey. This is roughly the topology under our own infrastructure, so it is written from the operator's chair rather than the textbook.
Why etcd holds the leader key, not Postgres
Postgres has no built-in notion of "which of my replicas is in charge." Streaming replication is one-directional and dumb on purpose: a primary ships WAL, replicas apply it. Nothing in that protocol stops a second node from also declaring itself primary. So the leadership decision is moved out of Postgres entirely, into etcd.
etcd stores a single leader key with a time-to-live. The node holding that key is the primary, and only that node. To stay primary it has to renew the lease before it expires. Patroni's defaults are ttl=30 seconds, loop_wait=10, and retry_timeout=10, bound by the constraint loop_wait + 2 * retry_timeout <= ttl. The primary's HA loop renews the key roughly every loop_wait seconds. If it cannot renew for longer than ttl, the key expires and a leader race begins. Whoever wins the race in etcd promotes; everyone else stays or becomes a replica. Because etcd hands the key to exactly one claimant, there can be exactly one primary.
That last sentence does all the work, and it only holds if etcd itself cannot be tricked into handing the key to two nodes during a network split. Which is where quorum earns its keep.
What is quorum, and why does N/2+1 prevent two leaders?
Quorum is the minimum number of nodes that must agree before the cluster commits anything: N/2 + 1, a strict majority. etcd uses the Raft consensus algorithm, and Raft will not commit a write or elect a leader without a majority of the cluster voting for it.
The reason a majority is the magic number is arithmetic, not policy. Two disjoint groups cannot both hold more than half the nodes. If group A has the majority, group B mathematically cannot, so group B is frozen out of committing anything. Cut a three-node cluster into a 2-node side and a 1-node side, and the 2-node side has quorum and keeps operating. The 1-node side knows it can reach only itself, recognizes it has lost majority, and refuses to elect a leader or accept a write. The isolated Patroni node sees etcd is unreachable, knows it cannot renew the leader key, and demotes itself to read-only. The minority partition disqualifies itself before two primaries can exist.
Why does an HA cluster want an odd number of nodes?
Run the same partition on four nodes and the flaw shows up. A clean 2-2 split leaves neither side with a majority, since you need 3 of 4. Both sides freeze. You paid for a fourth machine and gained zero fault tolerance over three nodes, while doubling your odds of a tie. That is the whole case for odd counts: an odd cluster tolerates the same number of failures as the next even size up, with one fewer box, and a partition always produces a clear majority side.
Fault tolerance is (N-1)/2 permanent failures:
| Nodes (N) | Quorum (N/2+1) | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
Three and four both survive a single failure; five and six both survive two. The even sizes are pure waste, and they add the tie scenario on top. Three nodes is the sane production floor. Go to five if you need to survive two simultaneous failures, or a maintenance window that overlaps an outage.
What actually happens during a partition, step by step
Say the primary lands on the minority side of a split. The real sequence:
- The primary can no longer reach the etcd majority, so it cannot renew the leader key.
- The watchdog notices the lease is running out. Patroni sets the hardware watchdog to fire 5 seconds before
ttlexpires (safety_margin=5), so with the defaults the kernel will reset the box if Patroni has not demoted in time. This is the fencing layer: it guarantees the old primary stops taking writes even if its own software hangs. - On the majority side, the leader key in etcd expires. The two reachable nodes hold quorum, run a Raft election, and one of them grabs the new leader key and promotes.
- The connection router (HAProxy, a VIP, or a leader-aware client) follows the new primary. Writes resume.
- The network heals. The old primary comes back, sees a newer leader already owns the key, and rejoins as a replica. It runs
pg_rewindto discard any local WAL that diverged and match the new timeline.
No moment in that sequence has two writable primaries. The minority side was read-only or fenced from the instant it lost quorum; the majority side only promoted because it could prove majority through etcd. The honest caveat: if etcd itself loses quorum, Patroni cannot fail over at all and may demote the primary rather than risk split-brain. Availability is sacrificed to preserve consistency. That is the correct trade, but it means your etcd cluster's health is now load-bearing for your database's uptime, and most teams under-monitor it.
The footguns that survive a correct node count
Odd nodes and a quorum store are necessary, not sufficient. The split-brain incidents I have actually seen come from the layers around the consensus, not the consensus math:
- etcd co-located with Postgres on the same boxes. A box dies and you lose a database node and an etcd member at the same time. Survivable at three nodes, but it eats into your margin. Give etcd its own failure domain where you can.
- Two data centers "for redundancy." A quorum cannot live across exactly two sites. Whichever site holds two of three etcd members is the only one that can ever have a primary; lose that site and the cluster freezes even though half your hardware is fine. Three sites, or a tie-breaker member in a third location, is the only honest two-region HA story.
- No fencing. Disable the watchdog and let a primary hang on a GC pause or a frozen disk without releasing the key, and a promoted replica plus a zombie old primary that later wakes is precisely your split-brain. Keep the watchdog on.
- A connection router that ignores the leader key. etcd can be perfectly correct while HAProxy still points writes at the demoted node. The routing layer has to read the same source of truth Patroni writes.
We run this class of architecture because durability and consistency are not optional for a zero-knowledge storage service that, by design, cannot reconstruct a customer's data from anything but the ciphertext they uploaded. A lost write is not a support ticket; it is gone. If you want to see where the platform is heading on top of it, the public roadmap tracks what is shipping next, and
The math is small enough to keep in your head. A strict majority cannot exist in two places at once, so the side that does not hold it goes quiet. The engineering is making sure that side can always go quiet cleanly: fence the old leader, keep etcd in its own failure domain, never straddle exactly two sites, and point your router at the same key Patroni does.