Skip to content

How to design an over-the-air update system

The first sections are free. The deep dives, level expectations and quiz are part of Pro.

On this page
Practice this question

The problem

An over-the-air update system carries a new software image from a car maker's build pipeline to each car in the fleet and installs it on a computer the owner depends on to get home. Two ideas carry the answer: a release reaches the fleet in stages that stop themselves when installs fail, and a car keeps a bootable copy of its old system until the new one has proven it can start. Everything else, the CDN, the byte-range downloads, the deltas and the signatures, serves those two.

Clarifying questions

AskWhy it mattersA typical answer

How many cars are in the fleet, and how large is an update?

The product of the two is the number of bytes one release moves, and that number decides whether the cars can pull from your servers directly or need a CDN and a slow rollout.

The fleet size is your call; a sensible assumption is 1 million cars, which is where one cloud update service caps a single instance. Updates run up to 2 GB.

What link do the cars have, and who pays for it?

A metered cellular link that drops shapes the download: it has to resume, and it should move as few bytes as it can. Home Wi-Fi changes the cost but not the mechanism.

Patchy cellular, metered, with some cars on home Wi-Fi part of the day. Design for the cellular case.

Does the car have room for two copies of its system?

Two slots are the simplest way to keep a car bootable through a failed install, and they cost twice the storage. If the answer is no, you are designing around a recovery image instead.

Yes. Assume one main computer with two slots. The smaller controllers inside the car are out of scope.

Does the owner approve an install, or does it happen on its own?

An install reboots the computer, so it has to wait for a parked car either way. Whether the owner also has to consent changes how long a car can sit between downloaded and installed, which the rollout's timers must tolerate.

Installs must not interrupt a driver. Treat consent as a product setting: the car schedules the install for a parked window, and the owner can move it.

How fast must a release reach the whole fleet, and how fast must a security fix?

The answer sets the growth of the stages. A feature can take a week; a fix for a remote exploit has days at most, and it may need to overtake a rollout already running.

Rollouts start at 50 cars a minute and grow from there. A security fix should be able to pause a running rollout and go ahead of it.

What counts as a failed install?

The stop rule is as good as the signal feeding it. If a car that silently fell back to its old version counts as a success, the rule fires too late.

A download that gives up, a verification failure, a timeout, and a car that reboots back into its old version. Success is a confirmed boot of the new version, reported by the car.

Are cars spread across many running versions?

A delta is built from one starting version to one target. A fleet on many versions needs many deltas or a full image for the stragglers.

Yes. Most cars sit on the last release or two, with a long tail of cars that were parked for months.

The requirements menu

RequirementScopeWhy

Publish a new software version and choose which cars get it

Core

The entry point. A release is an image plus signed metadata, and a rollout names the cars it is for, for example by hardware variant and region.

Roll it out in stages, and pause or stop a rollout

Core

The control loop of the whole system. A bad release reaching the first few thousand cars is a bad day; reaching the fleet is a recall.

Cars download the update in the background and resume after a dropped link

Core

Cars sit on cellular links that drop; a download that restarts from the first byte on each drop may not finish for some cars.

Cars check the update is genuine before installing, and fall back to the old version if it fails

Core

The car's own safety net. The server can be wrong about what reached the car, so the car verifies the bytes and keeps its old system until the new one boots.

Report each car’s version and update state

Core

The status feed is what lets the rollout controller stop. Without it, stages grow on a timer and nobody learns about a failed model until owners call.

Ship a patch built from the version a car already runs, rather than the whole image

Optional

Worth taking when the interviewer stresses metered links: it cuts the bytes per car by a large factor. Leave it if the fleet sits on many versions, because each starting version needs its own delta and a full image is still needed for the rest.

Let the owner schedule the install for a time the car is parked

Optional

Needed in some form, because the install reboots the computer. Design the car agent to pick a parked window on its own and let the owner move it; a full scheduling UI is product work.

Prefer Wi-Fi for the download when the car has it

Optional

Cheap to add on the car and it saves most of the cellular bill for cars that park at home. Take it as a policy on the agent, not as a requirement, since a car may have no link but cellular.

Push a message to cars so they check for an update now

Optional

Tempting for speed, but a scheduled check-in with a random offset already reaches a parked car within its interval, and a car that is off for days misses the push anyway. Mention it for the security-fix case and keep check-ins as the base.

A dashboard of rollout progress and failures by hardware variant

Optional

The release engineer needs it, and it falls out of the status store. Say what it reads and move on; the storage question is more interesting than the charts.

Building and testing the software itself

Out of scope

The build pipeline hands you a signed image. Say where it drops the image and the signature and stop there.

Updating each small controller inside the car: assume one main computer

Out of scope

Uptane's model has secondary controllers behind a primary that fetches and verifies on their behalf; name it and keep one computer.

Paid features and map data

Out of scope

Entitlements and map tiles have their own delivery and billing. They would reuse the download path but not the install path.

Non-functional requirements

RequirementWhat it forces

An update never leaves a car unable to start

Two copies of the system on the car. The new image is written into the copy that is not running, the running copy stays untouched, and the bootloader goes back to it unless the new copy has been confirmed healthy after booting. It also means the installer may not change anything the running system or the bootloader needs.

Only software signed by the company installs

The car checks signatures with keys it already holds before it writes a byte, and treats the encrypted connection and the CDN as untrusted carriers. The signing keys live outside the servers that hand out updates, and a single stolen key should not be enough to install an image.

A dropped link costs one byte range, about 1 MB, not the whole download

Downloads go as HTTP range requests of about 1 MB against storage that supports them, the car keeps a record of how far it got, and each range is checked against a hash before it is kept, so a resume does not splice in bad bytes.

Installs never interrupt a driver

The download runs in the background while the car drives; the install, which ends in a reboot, waits for a parked car and a window the owner accepts. The rollout's timers have to allow days between downloaded and installed without treating that gap as a failure.

Send as few bytes as possible over metered cellular links

A patch from the car's current version when one exists, a full image when it does not, Wi-Fi preferred when the car has it, and a download that finishes once rather than restarting. Check-in responses stay tiny so a million cars polling costs little.

Estimates

QuantityWorkingResultWhat it sizes

Bytes one full rollout moves

Assume a fleet of 1 million cars and an image at the 2 GB file ceiling: 1,000,000 × 2 GB = 2,000,000 GB = 2 PB

Up to 2 PB per release before any delta

Why the cars should not pull from one origin: a CDN takes the edge traffic and the stages spread it over days.

Origin egress if the fleet pulled that in one day

2 PB ÷ 86,400 seconds ≈ 23 GB a second ≈ 185 Gbit a second, sustained all day

About 185 Gbit a second

The load a staged rollout plus a CDN exists to avoid; it is the number behind what breaks first.

Range requests per car and per rollout

2 GB ÷ 1 MB per range = about 2,000 requests per car; × 1 million cars = about 2 billion requests per rollout

About 2,000 per car, about 2 billion for the fleet

Request volume at the CDN edge, and how much a dropped link costs: one request in about 2,000.

Time for one car to download a full image

Assume 10 Mbit a second on a decent cellular link: 2 GB = 16,000 Mbit, so 16,000 ÷ 10 = 1,600 seconds ≈ 27 minutes. At 1 Mbit a second, 16,000 seconds ≈ 4.4 hours

About 27 minutes on a good link, about 4.4 hours on a poor one

Why a download must survive many drops and why installs lag notification by hours: the controller's hold between stages has to be at least this long.

How fast doubling stages reach the fleet

At 50 cars a minute the first 1,000 cars take 20 minutes; the rate doubles once another 1,000 cars have been notified, so the next 1,000 take 10 minutes, the next 5, and the total converges toward 40 minutes however large the fleet

Under 40 minutes to notify a million cars if nothing caps the rate

Why doubling on its own is not a staged rollout: without a maximum rate or a hold for installs to finish, the fleet is notified before the first stage has rebooted.

Failures that trip the stop rule in the first stage

Example thresholds, not targets: first stage of 1,000 cars, stop when failures pass 5% of cars tried, judged after at least 100 cars have finished. 5% × 1,000 = 50 failed installs

Across the whole first stage of 1,000 cars, more than 50 failures cancel the queue; the rule is checked on the share so far as soon as 100 cars have finished

The abort rule's sensitivity; the minimum of 100 tried keeps one early failure from canceling a healthy release.

Check-in load from scheduled polling

1 million cars checking once a day = 1,000,000 ÷ 86,400 ≈ 12 a second; once an hour ≈ 278 a second

About 12 to 278 check-ins a second

Why scheduled check-ins beat a held-open push connection: the update service sees a trickle, and a random offset on each car keeps it a trickle.

Bytes saved by a delta

Example, not a target: if a delta is a tenth of the image, 2 GB ÷ 10 = 200 MB per car, and 1 million × 200 MB = 200 TB per rollout instead of 2 PB. Ten starting versions need ten deltas, about 2 GB of extra storage, well inside a 100 GB update store

About 200 MB per car, about 200 TB per rollout in that example

The cellular bill and the rollout's duration; also the storage cost of keeping a delta per starting version.

Core entities

EntityWhat it holds

Release

One published software version: the image, its hash and size, and the metadata the company signed over them.

Delta

A patch from one running version to one target, so a car on that version downloads a fraction of the image.

Rollout

One release sent to a set of cars under a growth plan and a stop rule.

Car

One vehicle: what hardware it is, what it runs now, and when it last called in.

Update state

Where one car is with one rollout: queued, downloading, installed, booted, failed, and how many bytes it has.

The API

EndpointNotes

POST /releases { version, image, signed_metadata } → { release_id }

Called by the build pipeline. The image goes to object storage; the server checks that the signed metadata's hash matches the bytes it received and refuses the release otherwise. In this design the server does not sign anything here: the metadata arrives already signed from the release pipeline.

POST /rollouts { release_id, targets, stages, abort_threshold } → { rollout_id }

Creates a rollout in the created state. stages carries the base rate, growth factor, step size, maximum rate and hold time; abort_threshold carries the failure share, the minimum cars tried, and which outcomes count. A security fix sets a higher priority than the feature rollout it must overtake.

POST /rollouts/{id}/pause

Stops adding cars to the queue. Cars that are mid-download or mid-install finish; the controller does not try to pull an update back from a car. In this design an abort is a pause that is not resumed.

GET /cars/{id}/update?running_version= → { release, delta_url?, full_url, metadata } or nothing

The car's check-in. The service looks up whether this car is in an open stage of a rollout it qualifies for, picks the highest-priority one, and returns signed metadata plus a delta URL when a delta exists from running_version. Keep the empty answer cheap. The car calls on a schedule with a random offset, not on a fixed minute.

POST /cars/{id}/status { rollout_id, state, bytes_done, error? }

The car reports each transition: downloading, verified, installed, booted, failed, rolled back. The payload carries no version, so the server learns which version actually came up from the running_version on the car's next check-in after a booted or rolled-back report, and that check-in is the one input that moves a car's recorded version forward. Authenticated with the car's own key so a spoofed report does not mark a car healthy.

The high-level design

imagereleasestatusreleasebytesRelease engineerObject storageUpdate serviceCar agentUpdate state

Step 1 of 4

Cars pull a release and report back

Start with the smallest thing that works. The release engineer publishes a release: the image lands in object storage and the update service learns there is a new version. Each car agent calls the update service on a schedule, gets back the release meant for it and a URL, downloads the image from object storage, installs it and reports its state. A status store keeps one row per car and rollout. This is the whole product in miniature, and it has two faults the next steps fix: it tells the whole fleet at once, and the whole fleet downloads from one origin.

rolloutimagetargetsmetadataoriginreleaserangesstatusinstallfailuresRelease engineerRollout controllerObject storageDirector signerUpdate serviceCDN edgeCar agentUpdate stateTwo slots +bootloader
The whole design

What breaks first

The deep dives

Rolling out in stages. A new version is ready for a fleet of a million cars. Which cars get it first, how fast does the group grow, what makes the controller stop, and what happens to cars that are halfway through when it does?

A concept that helps here: WebSockets, server-sent events and pollingA concept that helps here: Retries and dead-letter queues

Downloads over bad links. A car on a weak cellular link is partway through a two gigabyte download when the signal drops. What happens to the bytes it already has, where do the next bytes come from, and how many bytes does the whole rollout move?

A concept that helps here: CDNs and edge cachingA concept that helps here: Chunked and resumable uploadsA concept that helps here: Content addressing and deduplication

Verifying and rolling back. The bytes are on the car. How does it know they came from the company and were not changed on the way, how does it install them without risking a car that will not start, and how does the server learn what the car runs afterwards?

A concept that helps here: Failing open or failing closedA concept that helps here: Consistency models

Concepts this guide uses

Sources

  1. AWS IoT Core developer guide: How job configurations work
  2. AWS IoT Core developer guide: Create and manage job configurations with the console
  3. Microsoft Learn: Device Update for IoT Hub limits
  4. Microsoft Learn: Device Update for IoT Hub delta updates
  5. Android Open Source Project: A/B system updates
  6. Uptane Standard for Design and Implementation
  7. MDN: HTTP range requests
  8. Marc Brooker: Jitter: Making Things Better With Randomness
  9. AWS IoT Core developer guide: Create and manage jobs by using the AWS CLI
Sign in to report a mistake