How to design peer-to-peer file distribution
The first sections are free. The deep dives, level expectations and quiz are part of Pro.
On this page
- The problem
- Clarifying questions
- The requirements menu
- Non-functional requirements
- Estimates
- Core entities
- The API
- The high-level design
- What breaks first
- The deep dives
- Entity fields
- The fastest it can go
- Who sends which piece
- Stragglers, failures and the switch
- Variants
- What each level is expected to show
- The 30-second answer
- Quiz
The problem
Peer-to-peer file distribution copies one very large file, such as a model's weights, from one machine to a fleet of hundreds or thousands as fast as their links allow. The answer turns on arithmetic: a receiver on a 10 Gbps link needs 400 s or more to take in 500 GB, and a single source sending that file to 1,000 machines one after another would take about 4.6 days. The design closes that gap by cutting the file into hashed pieces and letting each machine forward what it holds, so the fleet's total upload grows with each host that joins, while a tracker off the data path tells machines who to trade with, and each host switches to the new version only after its whole copy checks out.
Clarifying questions
| Ask | Why it matters | A typical answer |
|---|---|---|
Is the 10 Gbps per machine one cap shared by sending and receiving, or 10 Gbps each way? | It changes the fastest possible finish by a factor of two. With a cap per direction, a host can receive at full speed while forwarding at full speed; with one combined cap, every byte it forwards comes out of its own download rate. | Treat it as one link shared between upload and download, and say out loud which reading you take. Per direction the bound is 400 s; one combined cap pushes it to about 800 s. |
How large is the file and how many machines need it? | The file size, with the piece size you choose, sets the piece count and the manifest length; the host count sets how many peers a tracker must track and how long one source would take on its own. | A 500 GB file to 100 to 1,000 GPU workers. Some versions of the question use 10 GB and thousands of hosts; the shape of the design is the same, the numbers differ. |
Are the target machines serving live traffic while the copy runs? | If they are, distribution has to leave room on the link for that traffic, which means a rate cap per host and a slower finish than the raw bound. | Yes. Each host's link also carries live work, so distribution runs under a per-host cap and must stay below it. |
Are the hosts trusted, inside our own data centers? | Trusted hosts let you skip the defenses BitTorrent needs on the public internet, such as incentives against free-riders, and focus on speed. You still hash every piece, because corruption happens without malice. | Yes, trusted machines in our data centers. Access control on who may publish is out of scope. |
How does a host move to the new version: all hosts at one moment, or each when it is ready? | A fleet-wide switch is gated on the slowest machine; a per-host switch means two versions serve at once for a while. Either way, design the activation step rather than assume it. | Design for per-host activation once a host holds and has checked the whole file, and say what a fleet-wide switch would cost instead. |
What do we know about the network between hosts: flat, or racks with scarcer links between them? | If links between racks are oversubscribed, the design should keep most piece traffic inside a rack. On a flat network, random peer choice is fine. | Assume a flat network for the main design. Rack-aware peer choice is the curveball, so have a sentence ready about it. |
Non-functional requirements
| Requirement | What it forces |
|---|---|
Finish close to the lower bound the links allow | Every host's upload link must be busy nearly the whole time, which means pieces small enough to forward before the file is complete, several requests in flight per peer, and a piece order that leaves each host something its neighbors want. A store-and-forward tree fails this: a fan-out of ten takes about 12,000 s against a 400 s bound. |
Stay under each machine’s link cap, shared between sending and receiving, and leave room for live traffic | A rate limiter on each host's agent, covering upload and download together, set below the link speed. Capping distribution at 8 Gbps of the 10 Gbps link leaves room for serving and moves the per-direction bound from 400 s to 500 s, about 1,000 s with one combined cap; the design should state that cost rather than hide it. |
No piece is used or forwarded before its hash checks out | A manifest of piece hashes published before any piece moves, and an agent that checks each piece on arrival, announces it to peers only after the check, and drops a failed piece and asks another peer. A whole-file hash before activation adds a check on the assembled file, for boundary or storage errors that happen after the pieces were verified. |
No single coordinator on the data path | The tracker hands out peer lists and nothing else: bytes flow host to host. The tracker can then run as several stateless instances behind one address, and losing one slows peer discovery without stopping transfers in progress. |
Scale to thousands of hosts: Uber’s Kraken runs at least 8,000 per cluster | Tracker state per host stays small (the hosts in each version's swarm, not the pieces each one holds), announce intervals are spread so a thousand hosts joining at once do not all return at the same second, and seeders form a group rather than a single machine so late joiners have a full copy to reach. |
Estimates
| Quantity | Working | Result | What it sizes |
|---|---|---|---|
Pieces in the file at two piece sizes | 500 GB ÷ 256 KiB ≈ 500 × 10^9 ÷ 262,144 ≈ 1.9 million, call it 2 million; at 4 MiB pieces, 500 × 10^9 ÷ 4,194,304 ≈ 119,000, call it 125,000 | About 2 million pieces at 256 KiB; about 125,000 at 4 MiB | The manifest length, the bitmap a host keeps, and how many requests a full copy takes: a reason to pick pieces larger than the 256 KiB most common in BitTorrent for a 500 GB file. |
Manifest size, one hash per piece | 2 million pieces × 20 bytes ( | 40 MB to 64 MB at 256 KiB pieces, about 4 MB at 4 MiB pieces | What every host downloads before its first piece, and what the tracker or origin serves a thousand times at the start of a rollout. |
Store-and-forward tree, fan-out ten | 1,000 hosts = 10^3, so three levels. At each level a parent sends the whole 4,000 Gb file to ten children over one 10 Gbps link: 10 × 400 s = 4,000 s. Three levels in sequence: 3 × 4,000 s = 12,000 s | About 12,000 s, roughly 3.3 hours | Why relaying the whole file through a tree is not close to the bound: each level waits for the one above to finish, and leaves send nothing. |
Lower bound with one combined cap | Each byte is sent once and received once. 1,000 receivers × 4,000 Gb × 2 directions = 8,000,000 Gb of link time, spread over (1,000 + 1) × 10 Gbps = 10,010 Gbps: 8,000,000 ÷ 10,010 ≈ 799 s | About 800 s, twice the per-direction bound | The finish time to promise when sending and receiving share a single 10 Gbps cap: a host sends roughly as much as it takes in. |
Bound under a cap that leaves room for live traffic | Cap distribution at 80% of the link: 8 Gbps. 4,000 Gb ÷ 8 Gbps = 500 s per direction, about 1,000 s with one combined cap | 500 s per direction, about 1,000 s combined | The cost of the rate limiter: a quarter more time in exchange for serving traffic that keeps flowing. |
A production agent cap in link terms | 300 MB/s × 8 bits = 2,400 Mbps = 2.4 Gbps, about a quarter of a 10 Gbps link | 2.4 Gbps per agent | What one production system chose to leave for everything else on the host, and why its 3 GB benchmark sits at the cap: 3 GB ÷ 300 MB/s = 10 s, the p50 reported. |
Bitmap of pieces held | 2 million pieces = 2 million bits ÷ 8 = 250 KB per host; 1,000 hosts × 250 KB = 250 MB if a tracker kept all of them; at 125,000 pieces, about 16 KB per host | 250 KB per host at 256 KiB pieces, 16 KB at 4 MiB | What each announce carries, and what a tracker that kept the latest bitmap of each host would hold in memory: about 250 MB for a thousand hosts at 256 KiB pieces. Larger pieces shrink it, and peers trading bitmaps directly between announces keeps the tracker's copy from needing to be fresh. |
Announce load on the tracker | 1,000 hosts announcing every 30 s = 33 announces a second in steady state; 1,000 in a single second when the rollout starts and every host joins at once | About 33 a second steady, a burst of 1,000 at the start | The tracker's request rate: small, but the start-up burst is why announce intervals should be spread and the tracker should run as several instances. |
Core entities
| Entity | What it holds |
|---|---|
File version | One published file: its size, the hash of each piece and the hash of the whole, which every host trusts. |
Piece | A fixed-size slice of the file, the unit a host checks, keeps and forwards. |
Peer | One host taking part in a version's swarm and what it currently holds. |
Swarm | The set of peers holding or fetching one version, which the tracker hands out slices of. |
Activation | A rollout of one version to a set of hosts, with where each host has got to. |
The API
| Endpoint | Notes |
|---|---|
| Publishes the manifest before any piece moves. The caller also uploads the file to the seeders, which check it against the same hashes. The piece size is fixed per version, so a client reading the manifest knows the offset of every index. |
| Called on join and then on an interval, so the tracker can drop hosts that go quiet. The response is a slice of the swarm, not all of it, and includes a seeder so a host can find a missing piece. The tracker does not forward bytes. |
| The data path. Offset and length let a host fetch a block of a piece from one peer and the rest from another. A peer that does not hold the piece answers with HTTP 404 and the requester moves on. Each host rate-limits what it serves here. |
| Tells each listed host to switch its serving process to the version. A host refuses unless all pieces and then the full-file hash have passed, and reports itself under failed. Calling it with the previous version is the rollback. |
The high-level design
Step 1 of 4
One source and an agent on every host
Start with the simplest thing that works: a publisher uploads the file to an origin machine, and an agent on each target host pulls it from there and writes it to local disk. This is the version to put on the board first so the arithmetic can be done against it. A receiver on a 10 Gbps link needs 400 s for 500 GB, and the origin needs that same 400 s per host, so 1,000 hosts take 400,000 s, roughly 4.6 days. The origin's upload link is where this design breaks, and nothing on the receiving side can help it.
What breaks first
The deep dives
The fastest it can go. One machine holds a 500 GB file and 1,000 machines need it, each on a 10 Gbps link. What is the least time any design could take, what limits the simple designs, and which link do you hit first?
A concept that helps here: Peer-to-peer file distribution
Who sends which piece. A thousand hosts each hold a different subset of the pieces. How does a host find its peers, learn what they hold, choose what to ask for next and from whom, and what happens when a piece arrives corrupted?
A concept that helps here: Peer-to-peer file distribution
A concept that helps here: Content addressing and deduplicationA concept that helps here: Rate limiting algorithmsStragglers, failures and the switch. Most of the fleet holds the whole file, a few hosts are slow, and one crashed with most of the file on disk. How does every host end up serving the new version, and when does each one switch?
Concepts this guide uses
Sources
- BitTorrent.org: BEP 3, the BitTorrent protocol specification
- adrian.idv.hk: reading notes on Bram Cohen's BitTorrent incentives paper (IPTPS)
- Uber Engineering: Introducing Kraken, an open source peer-to-peer Docker registry
- uber/kraken README on GitHub
- Engineering at Meta: Owl, distributing content at Meta scale