Skip to content

Distributed locks and leases

A distributed lock lets one process at a time act on a shared resource; a lease is that right with an expiry, so a dead holder frees it. The hard part is a live holder that outlasts its lease.

On this page

What it is

A distributed lock is an agreement among processes on different machines that one of them at a time may act on a shared resource: a file, a row, an external API call, a batch job. A plain lock has a weakness in a distributed setting: if the holder crashes or is cut off from the network, nobody releases it and the resource stays blocked. A lease fixes that by attaching an expiry. The holder gets the lock for a fixed window, renews it while it still needs it, and when renewals stop the lease lapses and the next process can acquire it.

The Redis pattern page states the three properties a lock service is built for: mutual exclusion (one holder at a time), freedom from deadlock (the lock becomes available again even if the holder crashed or was partitioned away), and fault tolerance (locks keep working while a majority of the lock nodes are up). The expiry buys the second one. The third depends on where the lock lives.

On one Redis node. The lock is a key set with NX (create it if it doesn't exist) and PX (expire it after a given number of milliseconds), and its value is a random string unique to this client and this attempt. Release runs a small script that deletes the key if its value still matches the one you set; a plain delete would let a client that overran its window remove a lock that another client now holds. A client that fails to acquire the lock should wait a random delay before trying again, so contenders stop retrying in step. A primary with an asynchronous replica is not a safe lock store either: if the primary dies before the key replicates, the promoted replica hands the same lock to a second client.

With a coordination service. The ZooKeeper lock recipe creates an ephemeral, sequential node under a lock directory. The client whose node carries the lowest sequence number holds the lock; everyone else sets a watch on the node just below their own and waits. Release is deleting your node, and because each node is watched by one client, a release wakes one waiter rather than the whole crowd. The ephemeral flag is what makes it a lease: when the holder's session dies, the service deletes the node once it detects the failure.

As a managed feature. Azure Blob Storage grants a lease on a blob for a chosen duration of 15 to 60 seconds or indefinitely, returns a lease ID, and refuses writes and deletes that don't carry the active ID; the holder can renew, release or change it, and a break ends the holder's lease while keeping newcomers off for a break period. Kubernetes represents the same idea as Lease objects in the coordination.k8s.io group, used for component leader election and for node heartbeats: each kubelet heartbeat updates the spec.renewTime field of its node's Lease, and the control plane reads that timestamp to judge whether the node is still alive.

The holder that outlives its lease. An expiry fixes the dead holder but not the slow one. A process can stop for longer than its lease without knowing it: a garbage collection pause, a swapped-out page, a stalled disk, a delayed packet that arrives after the lease passed to someone else. When it resumes, it still believes it holds the lock and writes. Martin Kleppmann's analysis of this case concludes that checking the expiry just before writing doesn't help either, because the pause can land between the check and the write. The fix is a fencing token: a number the lock service increases on each acquisition and that the client sends with each write. The resource keeps the highest token it has seen and rejects anything lower, so the late write from the old holder is turned away. This asks the storage side to take part; with the coordination service above, the zxid or a node version works as the token.

When to use it

Start with the question Kleppmann puts first, with a nod to the Chubby lock service: what happens if the lock fails and two processes act at once? If the result is wasted work or a minor annoyance (two nodes build the same expensive report, two crawlers fetch the same page, a user gets a duplicate email), the lock is for efficiency, and a single Redis node with an expiring key is a fair trade: cheap, fast, and wrong once in a while at a cost you can live with. If the result is corrupted data, a double payment or two leaders both accepting writes, the lock is for correctness, and you want a store that agrees on the holder through consensus, plus a fencing token the resource actually checks.

Leases also fit the jobs where one process should be active and others stand by: a scheduler that fires cron-style jobs, a single consumer that drains a queue in order, or a controller that must have one leader. The holder renews at a fraction of the lease length so a missed renewal or two doesn't cost it the lease, and the standbys poll or watch the lease to take over when the renew time goes stale. When the protected work is a chain of short steps, keep the lease short and extend it between steps rather than taking one long lease up front. Pair this with Leader election and failover for the hand-off and Heartbeats and presence for the renewal mechanics.

When not to

Skip the lock when the resource can enforce the rule itself. A unique constraint on the job's key, a conditional write that applies the change when the row still holds the version you read, or an idempotency key that makes the second attempt a no-op all remove the duplicate without anything to renew or to time out. The fallback for correctness-grade work is a database with proper transactions, and that is a good first answer in a round too. See Optimistic and pessimistic locking for the version-check version of this.

Don't build a five-node Redlock cluster for an efficiency lock; one Redis node does that job. And be careful proposing Redlock for a correctness lock: the critique of it holds that it rests on timing assumptions and issues no fencing tokens, and the Redis pattern page links both that critique and a counterpoint from Redis's author. In an interview, say that this is a known debate and which side you'd take for this system, rather than presenting Redlock as settled.

Skip the lock when the platform already hands each unit of work to one worker, such as a partitioned log that assigns each partition to a single consumer in a group; a lock on top adds a second source of truth about who owns what. And don't lease work that runs for hours as one step and can't be split: a lease that long makes failover slow, and a short one invites the overrun you just read about. Break the work into steps that can be redone, and lease each one.

Compared

ApproachHow exclusivity is enforcedWhen the holder diesPick it when

One Redis node, key with expiry

Set-if-absent; release compares a random value

Key expires after its time to live

Efficiency locks, where a rare double run is harmless

Redlock (majority of independent Redis nodes)

Lock held when a majority of nodes granted it within the validity window

Keys expire on each node

You want to survive a lost Redis node and accept the timing debate

ZooKeeper-style ephemeral node

Lowest sequence number wins; waiters watch their predecessor

Session loss deletes the node

Correctness locks; the zxid or version doubles as a fencing token

Database row or conditional write

Unique constraint or compare-and-set in a transaction

Nothing to clean up if the write is the lock

The data store is already the source of truth and can check a version

Managed lease (Azure Blob, Kubernetes Lease)

Writes must carry the active lease ID; renew time read by others

Azure: lapses after a chosen 15 to 60 seconds, unless set to never expire; Kubernetes: the renew time goes stale

The resource or platform already offers it and you'd rather not run a lock service

In a picture

acquire33Client ALock service

Step 1 of 3

Client A takes the lease

Client A asks the lock service for the lease and gets it with token 33. It starts its work.

How to say it in an interview

The common mistake

Quiz

A worker holds a 30-second lease, pauses for 45 seconds during garbage collection, then writes. What keeps that write from clobbering the new holder's work?
Which situation calls for a correctness lock rather than an efficiency lock?
In the ZooKeeper lock recipe, why does a waiting client watch the node just below its own instead of the lock directory?
Why does the single-node Redis release compare the key's value before deleting it?

Questions that use this

Related

Sources

  1. Redis documentation: Distributed Locks with Redis
  2. Martin Kleppmann: How to do distributed locking
  3. Apache ZooKeeper: ZooKeeper Recipes and Solutions
  4. Azure Storage REST API: Lease Blob
  5. Kubernetes documentation: Leases
Sign in to report a mistake