Skip to content

How to design peer-to-peer file distribution

The first sections are free. The deep dives, level expectations and quiz are part of Pro.

On this page
Practice this question

The problem

Peer-to-peer file distribution copies one very large file, such as a model's weights, from one machine to a fleet of hundreds or thousands as fast as their links allow. The answer turns on arithmetic: a receiver on a 10 Gbps link needs 400 s or more to take in 500 GB, and a single source sending that file to 1,000 machines one after another would take about 4.6 days. The design closes that gap by cutting the file into hashed pieces and letting each machine forward what it holds, so the fleet's total upload grows with each host that joins, while a tracker off the data path tells machines who to trade with, and each host switches to the new version only after its whole copy checks out.

Clarifying questions

AskWhy it mattersA typical answer

Is the 10 Gbps per machine one cap shared by sending and receiving, or 10 Gbps each way?

It changes the fastest possible finish by a factor of two. With a cap per direction, a host can receive at full speed while forwarding at full speed; with one combined cap, every byte it forwards comes out of its own download rate.

Treat it as one link shared between upload and download, and say out loud which reading you take. Per direction the bound is 400 s; one combined cap pushes it to about 800 s.

How large is the file and how many machines need it?

The file size, with the piece size you choose, sets the piece count and the manifest length; the host count sets how many peers a tracker must track and how long one source would take on its own.

A 500 GB file to 100 to 1,000 GPU workers. Some versions of the question use 10 GB and thousands of hosts; the shape of the design is the same, the numbers differ.

Are the target machines serving live traffic while the copy runs?

If they are, distribution has to leave room on the link for that traffic, which means a rate cap per host and a slower finish than the raw bound.

Yes. Each host's link also carries live work, so distribution runs under a per-host cap and must stay below it.

Are the hosts trusted, inside our own data centers?

Trusted hosts let you skip the defenses BitTorrent needs on the public internet, such as incentives against free-riders, and focus on speed. You still hash every piece, because corruption happens without malice.

Yes, trusted machines in our data centers. Access control on who may publish is out of scope.

How does a host move to the new version: all hosts at one moment, or each when it is ready?

A fleet-wide switch is gated on the slowest machine; a per-host switch means two versions serve at once for a while. Either way, design the activation step rather than assume it.

Design for per-host activation once a host holds and has checked the whole file, and say what a fleet-wide switch would cost instead.

What do we know about the network between hosts: flat, or racks with scarcer links between them?

If links between racks are oversubscribed, the design should keep most piece traffic inside a rack. On a flat network, random peer choice is fine.

Assume a flat network for the main design. Rack-aware peer choice is the curveball, so have a sentence ready about it.

The requirements menu

RequirementScopeWhy

Publish a new version of a large file for a set of target machines

Core

The entry point. Publishing produces the manifest of piece hashes that every later step trusts, so it has to exist before any piece moves.

Every target ends up with a complete copy, checked against hashes

Core

Without a check per piece, a corrupted piece would be found late, if at all. The hash per piece is what lets a host accept data from any peer.

Targets fetch pieces from each other, not only from the source

Core

This is the requirement that makes the bound reachable: the source alone needs about 4.6 days for 1,000 hosts.

A target that crashes or joins late catches up without starting over

Core

With 500 GB per host, a restart from zero costs another 400 s at best. Pieces already checked and on disk should count.

Report progress, and switch each target to the new version once it holds all of it

Core

Distribution is not done when bytes land; it is done when the serving process on each host runs the new version. Progress reports are also how you find the stragglers.

Prefer peers in the same rack so links between racks carry less traffic

Optional

Take it if the interviewer raises network topology: the tracker already knows each host, so adding its rack to the peer list is cheap. Leave it on a flat network, where it adds nothing.

Reuse pieces unchanged since the previous version instead of fetching them again

Optional

Tempting when versions differ little, since pieces are already hashed and a host can compare. Take it if the interviewer says successive versions share most of their bytes: comparing the two manifests is cheap when the old piece hashes were kept, but if the versions share little, the saving is small.

Keep the previous version on disk so a host can roll back

Optional

Worth taking: it costs disk, not network, and it is the answer to a bad model reaching the fleet. The activation call can switch a host back the same way it switched it forward.

Pre-stage the file on several seeder machines before the fleet starts

Optional

Useful when a rollout is scheduled: several seeders remove the one-source start-up and give late joiners a sure place to find any piece. Leave it if the file must go out the moment it is published.

Distribute across several data centers with one seeder group per site

Out of scope

A real need, but it is a replication problem between sites layered on this one. Say one seeder group per site would pull once over the wide-area link and park it.

Encrypt piece transfers between peers

Out of scope

Our advice: leave it out here. The prompt places the hosts in our own data centers, hashes already catch corruption, and wire encryption would cost CPU on every host.

Training the model and loading weights into GPU memory

Out of scope

Both sit on either side of this system. The copy ends when a checked file is on local disk and the serving process is told to load it.

Peers on the public internet: assume trusted machines in our own data centers

Out of scope

Public peers bring churn, free-riders and hostile data, which is what BitTorrent's incentive machinery exists for. Inside a data center you can drop most of it and keep the hashing.

Access control on who may publish a file

Out of scope

Ordinary authorization on one endpoint. Note that the publish call should sit behind it, then move on.

Non-functional requirements

RequirementWhat it forces

Finish close to the lower bound the links allow

Every host's upload link must be busy nearly the whole time, which means pieces small enough to forward before the file is complete, several requests in flight per peer, and a piece order that leaves each host something its neighbors want. A store-and-forward tree fails this: a fan-out of ten takes about 12,000 s against a 400 s bound.

Stay under each machine’s link cap, shared between sending and receiving, and leave room for live traffic

A rate limiter on each host's agent, covering upload and download together, set below the link speed. Capping distribution at 8 Gbps of the 10 Gbps link leaves room for serving and moves the per-direction bound from 400 s to 500 s, about 1,000 s with one combined cap; the design should state that cost rather than hide it.

No piece is used or forwarded before its hash checks out

A manifest of piece hashes published before any piece moves, and an agent that checks each piece on arrival, announces it to peers only after the check, and drops a failed piece and asks another peer. A whole-file hash before activation adds a check on the assembled file, for boundary or storage errors that happen after the pieces were verified.

No single coordinator on the data path

The tracker hands out peer lists and nothing else: bytes flow host to host. The tracker can then run as several stateless instances behind one address, and losing one slows peer discovery without stopping transfers in progress.

Scale to thousands of hosts: Uber’s Kraken runs at least 8,000 per cluster

Tracker state per host stays small (the hosts in each version's swarm, not the pieces each one holds), announce intervals are spread so a thousand hosts joining at once do not all return at the same second, and seeders form a group rather than a single machine so late joiners have a full copy to reach.

Estimates

QuantityWorkingResultWhat it sizes

Pieces in the file at two piece sizes

500 GB ÷ 256 KiB ≈ 500 × 10^9 ÷ 262,144 ≈ 1.9 million, call it 2 million; at 4 MiB pieces, 500 × 10^9 ÷ 4,194,304 ≈ 119,000, call it 125,000

About 2 million pieces at 256 KiB; about 125,000 at 4 MiB

The manifest length, the bitmap a host keeps, and how many requests a full copy takes: a reason to pick pieces larger than the 256 KiB most common in BitTorrent for a 500 GB file.

Manifest size, one hash per piece

2 million pieces × 20 bytes (SHA-1) = 40 MB; × 32 bytes (SHA-256) = 64 MB; at 125,000 pieces × 32 bytes = 4 MB

40 MB to 64 MB at 256 KiB pieces, about 4 MB at 4 MiB pieces

What every host downloads before its first piece, and what the tracker or origin serves a thousand times at the start of a rollout.

Store-and-forward tree, fan-out ten

1,000 hosts = 10^3, so three levels. At each level a parent sends the whole 4,000 Gb file to ten children over one 10 Gbps link: 10 × 400 s = 4,000 s. Three levels in sequence: 3 × 4,000 s = 12,000 s

About 12,000 s, roughly 3.3 hours

Why relaying the whole file through a tree is not close to the bound: each level waits for the one above to finish, and leaves send nothing.

Lower bound with one combined cap

Each byte is sent once and received once. 1,000 receivers × 4,000 Gb × 2 directions = 8,000,000 Gb of link time, spread over (1,000 + 1) × 10 Gbps = 10,010 Gbps: 8,000,000 ÷ 10,010 ≈ 799 s

About 800 s, twice the per-direction bound

The finish time to promise when sending and receiving share a single 10 Gbps cap: a host sends roughly as much as it takes in.

Bound under a cap that leaves room for live traffic

Cap distribution at 80% of the link: 8 Gbps. 4,000 Gb ÷ 8 Gbps = 500 s per direction, about 1,000 s with one combined cap

500 s per direction, about 1,000 s combined

The cost of the rate limiter: a quarter more time in exchange for serving traffic that keeps flowing.

A production agent cap in link terms

300 MB/s × 8 bits = 2,400 Mbps = 2.4 Gbps, about a quarter of a 10 Gbps link

2.4 Gbps per agent

What one production system chose to leave for everything else on the host, and why its 3 GB benchmark sits at the cap: 3 GB ÷ 300 MB/s = 10 s, the p50 reported.

Bitmap of pieces held

2 million pieces = 2 million bits ÷ 8 = 250 KB per host; 1,000 hosts × 250 KB = 250 MB if a tracker kept all of them; at 125,000 pieces, about 16 KB per host

250 KB per host at 256 KiB pieces, 16 KB at 4 MiB

What each announce carries, and what a tracker that kept the latest bitmap of each host would hold in memory: about 250 MB for a thousand hosts at 256 KiB pieces. Larger pieces shrink it, and peers trading bitmaps directly between announces keeps the tracker's copy from needing to be fresh.

Announce load on the tracker

1,000 hosts announcing every 30 s = 33 announces a second in steady state; 1,000 in a single second when the rollout starts and every host joins at once

About 33 a second steady, a burst of 1,000 at the start

The tracker's request rate: small, but the start-up burst is why announce intervals should be spread and the tracker should run as several instances.

Core entities

EntityWhat it holds

File version

One published file: its size, the hash of each piece and the hash of the whole, which every host trusts.

Piece

A fixed-size slice of the file, the unit a host checks, keeps and forwards.

Peer

One host taking part in a version's swarm and what it currently holds.

Swarm

The set of peers holding or fetching one version, which the tracker hands out slices of.

Activation

A rollout of one version to a set of hosts, with where each host has got to.

The API

EndpointNotes

POST /versions { size, piece_hashes, file_hash } → { version_id }

Publishes the manifest before any piece moves. The caller also uploads the file to the seeders, which check it against the same hashes. The piece size is fixed per version, so a client reading the manifest knows the offset of every index.

POST /swarms/{version}/announce { host, pieces_held } → { peers }

Called on join and then on an interval, so the tracker can drop hosts that go quiet. The response is a slice of the swarm, not all of it, and includes a seeder so a host can find a missing piece. The tracker does not forward bytes.

GET /pieces/{version}/{index}?offset=&length= → bytes, served by every peer

The data path. Offset and length let a host fetch a block of a piece from one peer and the rest from another. A peer that does not hold the piece answers with HTTP 404 and the requester moves on. Each host rate-limits what it serves here.

POST /versions/{id}/activate { hosts } → { activated, failed }

Tells each listed host to switch its serving process to the version. A host refuses unless all pieces and then the full-file hash have passed, and reports itself under failed. Calling it with the previous version is the rollback.

The high-level design

filepiecespiecesPublisherOrigin seedersAgent per hostPieces on disk

Step 1 of 4

One source and an agent on every host

Start with the simplest thing that works: a publisher uploads the file to an origin machine, and an agent on each target host pulls it from there and writes it to local disk. This is the version to put on the board first so the arithmetic can be done against it. A receiver on a 10 Gbps link needs 400 s for 500 GB, and the origin needs that same 400 s per host, so 1,000 hosts take 400,000 s, roughly 4.6 days. The origin's upload link is where this design breaks, and nothing on the receiving side can help it.

filehashesannounceversionpeersactivatepieceshashespiecespiecesswitchPublisherOrigin seedersManifestTrackerRolloutcoordinatorAgent per hostOther agentsPieces on diskServing process
The whole design

What breaks first

The deep dives

The fastest it can go. One machine holds a 500 GB file and 1,000 machines need it, each on a 10 Gbps link. What is the least time any design could take, what limits the simple designs, and which link do you hit first?

A concept that helps here: Back-of-the-envelope estimationA concept that helps here: Naming the bottleneck first

Who sends which piece. A thousand hosts each hold a different subset of the pieces. How does a host find its peers, learn what they hold, choose what to ask for next and from whom, and what happens when a piece arrives corrupted?

A concept that helps here: Content addressing and deduplicationA concept that helps here: Rate limiting algorithms

Stragglers, failures and the switch. Most of the fleet holds the whole file, a few hosts are slow, and one crashed with most of the file on disk. How does every host end up serving the new version, and when does each one switch?

A concept that helps here: Chunked and resumable uploadsA concept that helps here: Heartbeats and presenceA concept that helps here: Failing open or failing closed

Concepts this guide uses

Sources

  1. BitTorrent.org: BEP 3, the BitTorrent protocol specification
  2. adrian.idv.hk: reading notes on Bram Cohen's BitTorrent incentives paper (IPTPS)
  3. Uber Engineering: Introducing Kraken, an open source peer-to-peer Docker registry
  4. uber/kraken README on GitHub
  5. Engineering at Meta: Owl, distributing content at Meta scale
Sign in to report a mistake