How to design an over-the-air update system
The first sections are free. The deep dives, level expectations and quiz are part of Pro.
On this page
- The problem
- Clarifying questions
- The requirements menu
- Non-functional requirements
- Estimates
- Core entities
- The API
- The high-level design
- What breaks first
- The deep dives
- Entity fields
- Rolling out in stages
- Downloads over bad links
- Verifying and rolling back
- Variants
- What each level is expected to show
- The 30-second answer
- Quiz
The problem
An over-the-air update system carries a new software image from a car maker's build pipeline to each car in the fleet and installs it on a computer the owner depends on to get home. Two ideas carry the answer: a release reaches the fleet in stages that stop themselves when installs fail, and a car keeps a bootable copy of its old system until the new one has proven it can start. Everything else, the CDN, the byte-range downloads, the deltas and the signatures, serves those two.
Clarifying questions
| Ask | Why it matters | A typical answer |
|---|---|---|
How many cars are in the fleet, and how large is an update? | The product of the two is the number of bytes one release moves, and that number decides whether the cars can pull from your servers directly or need a CDN and a slow rollout. | The fleet size is your call; a sensible assumption is 1 million cars, which is where one cloud update service caps a single instance. Updates run up to 2 GB. |
What link do the cars have, and who pays for it? | A metered cellular link that drops shapes the download: it has to resume, and it should move as few bytes as it can. Home Wi-Fi changes the cost but not the mechanism. | Patchy cellular, metered, with some cars on home Wi-Fi part of the day. Design for the cellular case. |
Does the car have room for two copies of its system? | Two slots are the simplest way to keep a car bootable through a failed install, and they cost twice the storage. If the answer is no, you are designing around a recovery image instead. | Yes. Assume one main computer with two slots. The smaller controllers inside the car are out of scope. |
Does the owner approve an install, or does it happen on its own? | An install reboots the computer, so it has to wait for a parked car either way. Whether the owner also has to consent changes how long a car can sit between downloaded and installed, which the rollout's timers must tolerate. | Installs must not interrupt a driver. Treat consent as a product setting: the car schedules the install for a parked window, and the owner can move it. |
How fast must a release reach the whole fleet, and how fast must a security fix? | The answer sets the growth of the stages. A feature can take a week; a fix for a remote exploit has days at most, and it may need to overtake a rollout already running. | Rollouts start at 50 cars a minute and grow from there. A security fix should be able to pause a running rollout and go ahead of it. |
What counts as a failed install? | The stop rule is as good as the signal feeding it. If a car that silently fell back to its old version counts as a success, the rule fires too late. | A download that gives up, a verification failure, a timeout, and a car that reboots back into its old version. Success is a confirmed boot of the new version, reported by the car. |
Are cars spread across many running versions? | A delta is built from one starting version to one target. A fleet on many versions needs many deltas or a full image for the stragglers. | Yes. Most cars sit on the last release or two, with a long tail of cars that were parked for months. |
Non-functional requirements
| Requirement | What it forces |
|---|---|
An update never leaves a car unable to start | Two copies of the system on the car. The new image is written into the copy that is not running, the running copy stays untouched, and the bootloader goes back to it unless the new copy has been confirmed healthy after booting. It also means the installer may not change anything the running system or the bootloader needs. |
Only software signed by the company installs | The car checks signatures with keys it already holds before it writes a byte, and treats the encrypted connection and the CDN as untrusted carriers. The signing keys live outside the servers that hand out updates, and a single stolen key should not be enough to install an image. |
A dropped link costs one byte range, about 1 MB, not the whole download | Downloads go as HTTP range requests of about 1 MB against storage that supports them, the car keeps a record of how far it got, and each range is checked against a hash before it is kept, so a resume does not splice in bad bytes. |
Installs never interrupt a driver | The download runs in the background while the car drives; the install, which ends in a reboot, waits for a parked car and a window the owner accepts. The rollout's timers have to allow days between downloaded and installed without treating that gap as a failure. |
Send as few bytes as possible over metered cellular links | A patch from the car's current version when one exists, a full image when it does not, Wi-Fi preferred when the car has it, and a download that finishes once rather than restarting. Check-in responses stay tiny so a million cars polling costs little. |
Estimates
| Quantity | Working | Result | What it sizes |
|---|---|---|---|
Bytes one full rollout moves | Assume a fleet of 1 million cars and an image at the 2 GB file ceiling: 1,000,000 × 2 GB = 2,000,000 GB = 2 PB | Up to 2 PB per release before any delta | Why the cars should not pull from one origin: a CDN takes the edge traffic and the stages spread it over days. |
Origin egress if the fleet pulled that in one day | 2 PB ÷ 86,400 seconds ≈ 23 GB a second ≈ 185 Gbit a second, sustained all day | About 185 Gbit a second | The load a staged rollout plus a CDN exists to avoid; it is the number behind what breaks first. |
Range requests per car and per rollout | 2 GB ÷ 1 MB per range = about 2,000 requests per car; × 1 million cars = about 2 billion requests per rollout | About 2,000 per car, about 2 billion for the fleet | Request volume at the CDN edge, and how much a dropped link costs: one request in about 2,000. |
Time for one car to download a full image | Assume 10 Mbit a second on a decent cellular link: 2 GB = 16,000 Mbit, so 16,000 ÷ 10 = 1,600 seconds ≈ 27 minutes. At 1 Mbit a second, 16,000 seconds ≈ 4.4 hours | About 27 minutes on a good link, about 4.4 hours on a poor one | Why a download must survive many drops and why installs lag notification by hours: the controller's hold between stages has to be at least this long. |
How fast doubling stages reach the fleet | At 50 cars a minute the first 1,000 cars take 20 minutes; the rate doubles once another 1,000 cars have been notified, so the next 1,000 take 10 minutes, the next 5, and the total converges toward 40 minutes however large the fleet | Under 40 minutes to notify a million cars if nothing caps the rate | Why doubling on its own is not a staged rollout: without a maximum rate or a hold for installs to finish, the fleet is notified before the first stage has rebooted. |
Failures that trip the stop rule in the first stage | Example thresholds, not targets: first stage of 1,000 cars, stop when failures pass 5% of cars tried, judged after at least 100 cars have finished. 5% × 1,000 = 50 failed installs | Across the whole first stage of 1,000 cars, more than 50 failures cancel the queue; the rule is checked on the share so far as soon as 100 cars have finished | The abort rule's sensitivity; the minimum of 100 tried keeps one early failure from canceling a healthy release. |
Check-in load from scheduled polling | 1 million cars checking once a day = 1,000,000 ÷ 86,400 ≈ 12 a second; once an hour ≈ 278 a second | About 12 to 278 check-ins a second | Why scheduled check-ins beat a held-open push connection: the update service sees a trickle, and a random offset on each car keeps it a trickle. |
Bytes saved by a delta | Example, not a target: if a delta is a tenth of the image, 2 GB ÷ 10 = 200 MB per car, and 1 million × 200 MB = 200 TB per rollout instead of 2 PB. Ten starting versions need ten deltas, about 2 GB of extra storage, well inside a 100 GB update store | About 200 MB per car, about 200 TB per rollout in that example | The cellular bill and the rollout's duration; also the storage cost of keeping a delta per starting version. |
Core entities
| Entity | What it holds |
|---|---|
Release | One published software version: the image, its hash and size, and the metadata the company signed over them. |
Delta | A patch from one running version to one target, so a car on that version downloads a fraction of the image. |
Rollout | One release sent to a set of cars under a growth plan and a stop rule. |
Car | One vehicle: what hardware it is, what it runs now, and when it last called in. |
Update state | Where one car is with one rollout: queued, downloading, installed, booted, failed, and how many bytes it has. |
The API
| Endpoint | Notes |
|---|---|
| Called by the build pipeline. The image goes to object storage; the server checks that the signed metadata's hash matches the bytes it received and refuses the release otherwise. In this design the server does not sign anything here: the metadata arrives already signed from the release pipeline. |
| Creates a rollout in the |
| Stops adding cars to the queue. Cars that are mid-download or mid-install finish; the controller does not try to pull an update back from a car. In this design an abort is a pause that is not resumed. |
| The car's check-in. The service looks up whether this car is in an open stage of a rollout it qualifies for, picks the highest-priority one, and returns signed metadata plus a delta URL when a delta exists from |
| The car reports each transition: downloading, verified, installed, booted, failed, rolled back. The payload carries no version, so the server learns which version actually came up from the |
The high-level design
Step 1 of 4
Cars pull a release and report back
Start with the smallest thing that works. The release engineer publishes a release: the image lands in object storage and the update service learns there is a new version. Each car agent calls the update service on a schedule, gets back the release meant for it and a URL, downloads the image from object storage, installs it and reports its state. A status store keeps one row per car and rollout. This is the whole product in miniature, and it has two faults the next steps fix: it tells the whole fleet at once, and the whole fleet downloads from one origin.
What breaks first
The deep dives
Rolling out in stages. A new version is ready for a fleet of a million cars. Which cars get it first, how fast does the group grow, what makes the controller stop, and what happens to cars that are halfway through when it does?
A concept that helps here: Staged rollouts and canaries
A concept that helps here: WebSockets, server-sent events and pollingA concept that helps here: Retries and dead-letter queuesDownloads over bad links. A car on a weak cellular link is partway through a two gigabyte download when the signal drops. What happens to the bytes it already has, where do the next bytes come from, and how many bytes does the whole rollout move?
Verifying and rolling back. The bytes are on the car. How does it know they came from the company and were not changed on the way, how does it install them without risking a car that will not start, and how does the server learn what the car runs afterwards?
A concept that helps here: Signed updates and A/B slots
A concept that helps here: Failing open or failing closedA concept that helps here: Consistency modelsConcepts this guide uses
- Staged rollouts and canaries
- WebSockets, server-sent events and polling
- Retries and dead-letter queues
- CDNs and edge caching
- Chunked and resumable uploads
- Content addressing and deduplication
- Signed updates and A/B slots
- Failing open or failing closed
- Consistency models
Sources
- AWS IoT Core developer guide: How job configurations work
- AWS IoT Core developer guide: Create and manage job configurations with the console
- Microsoft Learn: Device Update for IoT Hub limits
- Microsoft Learn: Device Update for IoT Hub delta updates
- Android Open Source Project: A/B system updates
- Uptane Standard for Design and Implementation
- MDN: HTTP range requests
- Marc Brooker: Jitter: Making Things Better With Randomness
- AWS IoT Core developer guide: Create and manage jobs by using the AWS CLI