Skip to content

Blob storage and presigned uploads

An object store holds files as blobs behind a key; a presigned URL lets a client upload or fetch one directly, within limits your server signs, so large bytes skip your API tier.

On this page

What it is

An object store keeps a file as an object: the bytes plus some metadata, addressed by a key that is unique within its bucket. Clients read and write objects over HTTP, so bucket plus key is what your application needs to record. Objects can be big: S3 takes up to 5 GB in a single PUT and up to 50 TB when the object is sent in parts. Your database keeps the key and the metadata (owner, size, content type, state) while the bytes stay in the store.

A presigned URL moves the authorization into the URL itself. Your server, which holds credentials, signs the bucket, the key, one HTTP method and an expiry, then hands the result to the client. Whoever holds that URL can make that one kind of request until it expires, with no account of their own. The URL can do no more than its signer: if the signing identity lacks permission to write that key, the upload fails, and a URL signed with temporary credentials stops working when those credentials expire, even if its own expiry is later. Google Cloud calls the same thing a signed URL; Azure uses a shared access signature (SAS), a token in the query string that scopes the resource, the permissions and the validity window.

ProviderUpload tokenLongest validityNote

Amazon S3

Presigned URL (SigV4)

7 days from the CLI or SDKs; 12 hours from the console

Shorter when the signing credentials are temporary

Google Cloud Storage

Signed URL (V4)

7 days

The holder needs no Google account

Azure Blob Storage

Shared access signature

Set by the signer; near-term expiry is recommended

A user delegation SAS, signed with Entra credentials, is the recommended kind

The upload has three steps, and the middle one is the one your servers don't see. First, the client asks your API to upload; the API checks that this user may add a file here, writes a metadata row in a pending state, and returns a signed URL for a key it chose. Second, the client sends the bytes to the store with that URL. Third, something confirms the bytes landed: the client calls back and the API checks that the object exists, or the store emits an object-created event to a queue or a function. S3 event notifications are designed for at-least-once delivery and usually arrive within seconds, though sometimes a minute or more, so the handler must tolerate a duplicate and the row may lag the upload a little. Then the row flips to ready. Treat anything written through a signed URL as unvalidated until your own code has checked it, since a client can send content other than what you expected.

A POST policy upload, the HTML form variant in the S3-compatible API family, embeds a signed policy in the form. The policy carries an expiration and a list of conditions the form fields must meet: an exact match on the bucket, a starts-with prefix on the key (so a user can write under their own folder and nowhere else), a prefix or exact value for Content-Type, and a content-length-range with a minimum and a maximum body size. If you need a hard bound on upload size, that condition is the way to put it into the signature. When the uploader is a page in a browser, the bucket also needs a CORS configuration that lists your origin and the allowed methods before the page can call it.

When to use it

Use the store plus a signed URL when users send files of meaningful size (photos, video, documents, exports) and your API would otherwise spend its bandwidth and memory relaying bytes. The store scales the byte path; your servers keep the small, fast work: authorization, key generation, metadata, confirmation. Mobile and browser clients fit well here, because you don't want to ship cloud credentials inside them and a signed URL gives them time-limited access without any.

Use the same trick for downloads of private files: a signed GET with a short expiry lets the browser fetch straight from the store, and for public or widely shared files a CDN in front of the store does better (see CDNs and edge caching). At 100 MB or more, or on connections that drop, send the object in parts: parts travel independently and in any order, and a failed part is retried on its own; that is covered in Chunked and resumable uploads. Parts of an upload that isn't completed still occupy storage and are billed, so set a lifecycle rule that aborts incomplete multipart uploads after a few days.

When not to

Keep small blobs that are read together with their row (a tiny thumbnail, a settings document of a few kilobytes) in the database or inline in the response; a second round trip and a second system to keep consistent cost more than they save. Relay the upload through your API when the content must be inspected or transformed before it may exist at all (a strict compliance rule, a synchronous conversion), when traffic is low enough that the relay is cheap, or when you want one place for validation logic and don't yet have a scale problem. Accepting first and validating afterward also works, but the pending state and the cleanup are extra machinery.

Skip signed URLs between your own services: a backend with its own cloud identity calls the store directly through the SDK. And don't lean on a signed URL for access you may need to withdraw quickly: Azure's guidance is to have a revocation plan before you issue one, and a stored access policy, which a service SAS can be tied to, is the lever it offers for pulling permissions back. A short expiry and a narrow key are the practical limits.

Compared

ApproachPath of the bytesWhat you controlWhen it's the right choice

Upload through your API

Client to API to store

All of it: you see the bytes before they're stored

Small files, low traffic, or content that must be checked or transformed before it may be stored

Presigned PUT

Client straight to store

Key, method, expiry, signed headers

A trusted client app uploading one file at a time, where size needs no hard bound

POST policy (HTML form)

Client straight to store, via a form

Plus a size range, a key prefix, a content type match

Browser uploads where you need a size bound or a plain HTML form

Multipart upload

Client to store, in parts

Parts sent independently, in any order, each retried on its own

Objects of 100 MB or more, or flaky connections; more bookkeeping and a cleanup rule

In a picture

requestpendingURLbyteseventreadyClientAPI serverMetadata DBObject storeUpload worker
The API signs and records; the bytes go straight to the store; an event marks the row ready.

How to say it in an interview

The common mistake

Quiz

A presigned PUT was created with a 24-hour expiry but stopped working after about an hour. What's the most likely cause?
A browser uploads straight to the bucket and you must cap uploads at 10 MB. Which fits?
Which case argues for relaying the upload through your own API instead of a signed URL?
Why keep the metadata row in a pending state until the upload is confirmed?

Questions that use this

Related

Sources

  1. Amazon S3 User Guide: What is Amazon S3?
  2. Amazon S3 User Guide: Uploading objects
  3. Amazon S3 User Guide: Download and upload objects with presigned URLs
  4. Oracle Cloud Infrastructure documentation: Uploads Using POST with Amazon S3 Compatibility API for Object Storage
  5. Amazon S3 User Guide: Amazon S3 Event Notifications
  6. Amazon S3 User Guide: Using cross-origin resource sharing (CORS)
  7. Amazon S3 User Guide: Uploading and copying objects using multipart upload
  8. Google Cloud Storage documentation: Signed URLs
  9. Azure Storage documentation: Grant limited access to Azure Storage resources using shared access signatures (SAS)
Sign in to report a mistake