Chunked and resumable uploads
Splitting an upload into parts a client sends independently, retries one at a time and resumes after a dropped connection, with the server recording what arrived. Worth knowing for any design that takes large files from phones or browsers.
What it is
A chunked upload cuts a file into pieces and sends each piece as its own request. A resumable upload adds server-side state: the server remembers which pieces or how many bytes it has stored, so a client that loses its connection asks what arrived and continues from there instead of starting over. Two practical reasons drive chunking: browsers and other clients cap the size of one request, and chunking bounds how much data a client has to buffer in case the transfer fails. The recovery argument is the third: with parts, a network error means re-sending the interrupted parts, not the whole object.
The parts model. In S3 multipart upload, an initiate call returns an upload ID. The client then uploads parts under that ID, each tagged with a part number from 1 to 10,000; the numbers need not be consecutive unless the upload uses multipart checksums, which require consecutive numbers starting at 1. Parts can go in any order and in parallel, and uploading a new part with an existing number replaces the earlier one. Each part comes back with an ETag the client records. A final complete request lists the part numbers with their ETags, and S3 builds the object by joining the parts in ascending part number; after that the individual parts no longer exist. Parts are 5 MiB to 5 GiB each, except the last one, which has no minimum. Azure block blobs work the same way: Put Block stores a block under a block ID, Put Block List commits an ordered list of IDs, and a block isn't part of the blob until a list that names it is committed. In service version 2019-12-12 and later a block can be up to 4000 MiB and a blob can hold 50,000 blocks.
The offset model. The tus protocol (version 1.0.0) treats the upload as one resource with a byte offset. A HEAD on the upload URL returns Upload-Offset, the number of bytes stored so far. The client then sends a PATCH whose Upload-Offset header matches that value and whose body is the next bytes; the server answers 204 No Content with the new offset. If the client's offset doesn't match the server's, the server replies 409 Conflict and leaves the upload untouched. Google Cloud Storage's resumable upload follows the same idea with different headers: the client PUTs chunks to a session URI with a Content-Range, the server replies 308 Resume Incomplete with a Range header saying which bytes it has persisted, and the client starts the next chunk at the byte after that header's upper value rather than assuming its last chunk landed in full. Chunks are multiples of 256 KiB except the last, at least 8 MiB is recommended, and a client that doesn't know the total size writes * in place of it. The IETF's httpbis working group is standardizing this shape as Resumable Uploads for HTTP, an active draft (revision 15 in November 2026) that defines creating, appending to and canceling an upload.
| System | Unit and limits | How a client resumes | Unfinished upload |
|---|---|---|---|
S3 multipart | Numbered parts, 5 MiB to 5 GiB, up to 10,000; last part any size | List the parts the server has, re-send the rest, then complete | Kept and billed with no expiry until completed or aborted; a lifecycle rule can abort it after N days |
Azure block blob (2019-12-12 and later) | Blocks with IDs, up to 4000 MiB, up to 50,000 | Re-send uncommitted blocks, then commit the block list | Uncommitted blocks are discarded after a week |
Google Cloud Storage resumable | Byte ranges in multiples of 256 KiB, except the final chunk | Query the session, read the Range header, continue from there | Must finish within a week of being started; can be canceled any time |
tus 1.0.0 | Byte offset on one upload URL | HEAD for Upload-Offset, PATCH from that offset | Depends on the server implementation |
Integrity. The parts model checks each piece: S3 returns an ETag per part, and a checksum when checksums are in use, and the complete request lists the ETags; an Azure block can carry an MD5 so a corrupted block is re-sent on its own. tus offers a Checksum extension that puts an Upload-Checksum header on a PATCH. Google Cloud Storage's resumable upload is the exception: it doesn't verify intermediate chunks, so the whole-object check happens at the end. Our advice: whichever model you pick, hash the whole file on the client, send that hash with the final commit, and compare it with what the server assembled.
Visibility and concurrency. In all three stores the object shows up when the upload completes, not as parts arrive: Google Cloud Storage dates the object from completion, Azure keeps blocks outside the blob until committed, and S3 creates the object on the complete call. Two clients uploading to the same S3 key at once are both allowed to finish; in a versioned bucket, the upload that was initiated most recently becomes the current version even if it completes first, and in an unversioned bucket another write between initiate and complete can take precedence. A design that needs to prevent that can use a conditional write or give each upload its own key.
When to use it
Reach for parts or offsets when redoing a failed upload costs more than the extra requests. S3 suggests multipart at 100 MB and up; Google's client libraries switch to resumable above 8 MiB in Python and, by default, above 16 MiB in Go. Those figures differ by an order of magnitude, so there is no universal size. Our advice: set the threshold for the links your clients use, because the expected cost of a failed upload rises with the failure rate. A 20 MB photo over a train's mobile connection benefits more than a 200 MB file on a data-center link.
Pick the parts model when you want throughput on a good link, since parts upload in parallel, or when a browser should upload straight to the object store and your server should just initiate and complete; see Blob storage and presigned uploads. Pick the offset model when the data is a stream of unknown length, such as a file compressed while it's sent, when client memory is tight and you want to buffer one chunk at a time, or when your own upload server sits in front of storage and tus gives you a ready protocol. Both models let an upload begin before the final size is known.
When not to
Below a few megabytes, send one request. A multipart upload adds an initiate call, a complete call, state to track and an abandoned-upload problem to clean up; for a small image on a decent connection that overhead usually buys little, and Azure accepts a single write of up to 5000 MiB in service version 2019-12-12 and later, so the single-request path reaches well past the 100 MB rule of thumb.
Don't start a multipart upload you can't clean up. S3 keeps uploaded parts, and bills for them, until the upload is completed or aborted, with no expiry of its own; its advice is a lifecycle rule with the AbortIncompleteMultipartUpload action set to a number of days. Azure and Google Cloud Storage discard unfinished work after a week. If your design has no sweeper and no lifecycle rule, orphaned parts grow without bound.
Don't choose the offset model when parallelism is the goal. A tus client appends from one offset at a time, and parallel transfer needs the separate Concatenation extension. And don't build your own chunk-assembly server when an object store already does the assembly, and does the cleanup once you set a lifecycle rule or rely on its expiry: the three stores above each offer the parts or offset protocol directly, so your service usually needs to handle issuing credentials, recording the upload, and reacting when it completes, not holding bytes.
Compared
| Approach | How it resumes | Parallel pieces | Pick it when |
|---|---|---|---|
One request (a single PUT) | It doesn't: a failure restarts the transfer | No | The file is small for the link in use and you want no cleanup job; S3 suggests the single call below 100 MB, while Google's Python library switches to resumable above 8 MiB |
Parts, then one commit (S3 multipart, Azure block blobs) | List what the server holds, re-send what's missing, commit the ordered list | Yes, in any order | Large files on good bandwidth, or browser-to-store uploads where your server stays out of the data path |
Offset append (tus, Google Cloud Storage) | Ask the server for its offset, continue from there | Sequential by default | Streams of unknown length, tight client memory, slow mobile links, or an upload server you run yourself |
In a picture
Step 1 of 3
Initiate and send parts
The client asks the store to start an upload and gets an ID. It cuts the file into parts and uploads them under that ID, several at a time, recording each part's number and ETag.
How to say it in an interview
The common mistake
Quiz
Questions that use this
Related
Sources
- Amazon S3 User Guide: Uploading and copying objects using multipart upload
- Amazon S3 User Guide: Amazon S3 multipart upload limits
- tus resumable upload protocol 1.0.0
- Google Cloud Storage documentation: Resumable uploads
- Google Cloud Storage documentation: Perform resumable uploads
- Azure Storage REST API: Understanding block blobs, append blobs, and page blobs
- IETF Datatracker: Resumable Uploads for HTTP (draft-ietf-httpbis-resumable-upload)