Leader election and failover
How a group of nodes agrees on one of them to be in charge, notices when that node fails, and moves the role to another without two nodes acting as leader at the same time.
What it is
A leader is the one node in a group that is allowed to do a job that must not be done twice at once: accept writes for a shard, fire scheduled jobs, assign work to the others. Leader election is how the group agrees on which node that is. Failover is what happens after the leader dies or is cut off: the group notices, picks a replacement, and the old leader is prevented from carrying on. In Raft, for example, the cluster has a single elected leader that decides where new log entries go and drives replication to the followers without asking them first; that single point of decision is what makes ordering simple.
Election by majority vote. Raft divides time into terms, and each term begins with an election. A follower that hears nothing from the leader for an election timeout becomes a candidate: it increments the term, votes for itself and asks the other servers for their votes. A server votes at most once per term, first come first served. A candidate that collects votes from a majority becomes leader and starts sending heartbeats; one that sees a higher term gives up and follows. If the vote splits, the term ends with no leader and a new election starts. To make splits rare, each server picks its timeout at random from a range (150 to 300 ms is typical), so usually one server times out first and wins before the others wake up. A voter also refuses a candidate whose log is less current than its own, which is how committed writes survive the change of leader.
Election through a coordination service. With ZooKeeper, each process creates a child node under a shared path with the sequential and ephemeral flags. The service appends an increasing number to each name; the process that owns the lowest number is the leader. Because the node is ephemeral, it disappears when the owner's session ends, which is how the service announces a dead leader. The recipe tells each process to watch the node just below its own rather than the leader's, so that one failure wakes one waiter instead of all of them at once (the herd effect). Distributed locks and leases covers the lease version of this idea, where the holder must keep renewing a key with a time to live.
| Who decides | How the leader is chosen | How failure is noticed | Example |
|---|---|---|---|
The peers themselves | Majority of votes in a numbered term | Missed heartbeats past a timeout | Raft; MongoDB replica sets |
A coordination service | Lowest sequence node, or the holder of a lease | Ephemeral node vanishes when the session ends; a lease lapses when renewals stop | ZooKeeper election recipe |
External monitors | Monitors vote among themselves for one to run the failover | A quorum of monitors agree the leader is unreachable | Redis Sentinel |
Detection is a timeout, and it sets your downtime. MongoDB 9.0 members ping each other at two-second intervals and treat a member as inaccessible when a ping goes unanswered for 10 seconds; the election timeout defaults to 10 seconds as well, and the median time to elect a new primary with default settings should typically stay under 12 seconds. During that window the replica set accepts no writes, though reads can continue if the application sends them to secondaries. Redis Sentinel separates two thresholds: the quorum is how many Sentinels must agree the master is unreachable (after down-after-milliseconds of silence) before it counts as down, but the failover itself needs a majority of all Sentinels to elect one of them to run it. The documented example is five Sentinels with a quorum of two: two can declare the master down, but at least three must be reachable for the failover to start.
The old leader has to be stopped, not just replaced. A missed heartbeat means the leader is dead, slow, paused or on the far side of a partition, and the protocol has no way to tell which. Two defenses are common. First, the leader polices itself: a MongoDB primary that can see no more than a minority of the voting members steps down to secondary, while a member on the majority side holds an election. Second, the resource polices the leader. Martin Kleppmann's fencing token is a number the lock or lease service increases on each acquisition; each write carries it, and the storage rejects any write whose token is lower than one it has already accepted. A leader that paused for longer than its lease, then woke and wrote, is turned away because a newer holder has already written with a higher number. ZooKeeper's zxid or a znode's version can serve as that number. The storage has to take part in the check; a lease that nobody verifies downstream does not protect correctness when the holder is paused, though it still works as best-effort coordination.
When to use it
Reach for a leader when one node must own a decision: the primary for a shard that serializes writes so replicas apply them in one order; a scheduler where a job must fire once; a coordinator that assigns partitions to workers. A single decider removes the need to reconcile conflicting writes afterwards, which is why Raft gives the leader sole charge of the replicated log. See Replication and read replicas for how the followers consume what the leader writes.
Automate the failover when the service can't wait for a person. With MongoDB's defaults, detection plus election has a documented median under 12 seconds; a manual promotion depends on someone noticing and acting. The conditions: an odd number of voters, three at least, placed so they fail independently. Sentinel's own minimum is three processes on separate machines or availability zones, and its two-box example shows why: when one box dies, the survivor has no majority to authorize anything. The same majority rule is what stops a minority partition from electing its own leader.
When not to
Skip the leader when the work can safely run twice. Kleppmann separates locks taken for efficiency (two nodes doing the same expensive job costs a little money or a duplicate email) from locks taken for correctness (two nodes writing the same data corrupts it). If your case is the first, a best-effort lease is fine and losing it occasionally is acceptable; you don't need a consensus protocol for a cron job whose duplicate run is harmless. Make the job idempotent instead, with the ideas in Idempotency keys.
Partition the work instead of electing an owner for all of it when each key can be served by a node chosen by hashing, as in Consistent hashing: no node needs to be in charge of the whole, so there is nothing to elect. And don't promise a single leader where writes must be accepted on both sides of a partition, such as two regions that each keep serving during a network split; that calls for multiple writers and Conflict resolution, not a faster election.
Be cautious with automatic failover on a small or rarely exercised system. The Sentinel documentation states that no high-availability setup is safe unless it is tested from time to time, and that Sentinel with asynchronous replication does not keep all acknowledged writes across a failover. If you won't rehearse the failover, a single primary with a clear manual promotion runbook and a tested backup can be the more honest design: slower to recover, but without a promotion that fires on a false alarm and throws away writes.
Compared
| Approach | Chosen by | Stale leader is stopped by | Pick it when | Pick something else when |
|---|---|---|---|---|
Majority vote among the peers (Raft, MongoDB) | Votes from a majority in a numbered term | A candidate that sees a higher term stands down; a MongoDB primary that sees no more than a minority steps down | The leader also owns the replicated data, so votes and log state travel together | You have two nodes, or the nodes can't afford a consensus library; use a service or a monitor |
Coordination service (ZooKeeper, or a lease store) | Lowest sequence node or the current lease holder | Ephemeral node or lease expires; storage checks a fencing token | Stateless workers need one of them to act, and a shared service already exists | The resource can't check tokens and a paused holder could still write; the lease is then safe for efficiency only, not for correctness |
External monitors (Redis Sentinel) | Quorum of monitors agree it is down; majority of monitors elect the one to act | The old master is reconfigured as a replica when the partition heals; | The data store has no election of its own and clients can ask the monitors for the current address | Losing acknowledged writes is unacceptable; asynchronous replication can drop them |
Manual failover | A person promotes a replica | The person shuts the old primary down first | The system is small, failures are rare, and a false alarm would cost more than minutes of downtime | Recovery must happen in seconds, or nobody is on call |
In a picture
Step 1 of 4
Steady state
The leader of term 5 sends heartbeats to both followers. Each heartbeat resets a follower's election timer.