How to design GitHub Actions
The first sections are free. The deep dives, level expectations and quiz are part of Pro.
On this page
- The problem
- Clarifying questions
- The requirements menu
- Non-functional requirements
- Estimates
- Core entities
- The API
- The high-level design
- What breaks first
- The deep dives
- Entity fields
- Queueing jobs onto runners
- A runner that dies mid-job
- Logs, artifacts and caches
- Running untrusted code
- Variants
- What each level is expected to show
- The 30-second answer
- Quiz
The problem
GitHub Actions turns a push into workflow runs, splits each run into jobs, and hands those jobs to a shared pool of machines while developers watch the logs. The answer turns on keeping three things apart because they scale differently: taking events in durably, deciding which job runs next when thousands of tenants share one pool, and moving logs, artifacts and caches around the scheduler rather than through it. GitHub's published figures anchor the arithmetic: 71 million jobs each day, and from its own internal fleet an average of about 8.3 minutes per job, which is the one job-length figure this guide carries through its estimates.
Clarifying questions
| Ask | Why it matters | A typical answer |
|---|---|---|
How do we learn about a push: does the source host call us, or do we poll it? | A webhook decides the shape of intake: a receiver that must acknowledge fast, deal with a repeated delivery, and survive a host that resends. | The source host sends a webhook per event with a delivery id. It may send the same delivery again, so one push must start one run. |
What scale: repositories, pushes a second, jobs a day, and how long a job runs? | Pushes a second sizes the intake; jobs a day and job length together size the runner pool, which is the expensive part. | About 10 million repositories, about 10 pushes a second with bursts, and GitHub's own figure of 71 million jobs each day. Jobs on hosted runners can run up to 6 hours; no average is given, so estimate one and say where it comes from. |
Are the machines ours, the customer's, or both? | Customer-owned runners sit behind firewalls and change how jobs reach a machine; hosted runners make isolation and warm capacity our problem. | Both. Hosted runners by default, and customers may register their own machines, matched to jobs by labels. |
Do public repositories run workflows for pull requests from strangers? | If yes, the runner executes code written by anyone, and isolation and secret handling move from nice-to-have to a core requirement. | Yes. Assume a public repository accepts pull requests from anyone and runs their workflow, perhaps after a maintainer approves the first run from a new contributor. |
How uneven are tenants? What does the worst burst look like? | Fairness is the center of this question. One tenant's burst size tells you whether a single queue is acceptable. | One organization can queue 10,000 jobs at merge time while a small team queues 5. The small team should not wait behind the big one. |
Can jobs have side effects, such as deploying, and may the system rerun them on its own? | A retried build is harmless; a retried deploy may not be. The answer decides whether a lost runner's job goes straight back in the queue. | Yes, jobs deploy. Retries are expected after failures, so the design has to make a repeated deploy step safe or stop it. |
Non-functional requirements
| Requirement | What it forces |
|---|---|
Tens to hundreds of thousands of runs at once across many tenants | The runner pool, not the control plane, is the big thing to size: one job per machine means hundreds of thousands of machines or VMs in flight. Job state has to be in a store that many scheduler instances can read, and the pool has to grow and shrink with demand. |
Fair scheduling: one organization queuing 10,000 jobs does not starve one queuing 5 | Ready jobs can't sit in one global line. Queues are kept per tenant, each tenant has a cap on jobs running at once, and idle runners are shared across the tenants that are waiting. |
Strong isolation: one workflow cannot read another tenant’s secrets or files | A fresh machine or VM per job that is destroyed afterward, secrets issued per job and scoped to one repository and one environment, and no secrets at all for runs triggered from a fork. |
A typical pull request reaches a deploy decision in tens of minutes | Queue wait and machine start both count against this budget, so the design keeps warm capacity ready and caches dependencies and build outputs instead of rebuilding them. |
A push event is never lost, and never starts the same run twice | The receiver writes the event durably, keyed by the host's delivery id with a unique constraint, before it acknowledges, and the workflow service picks events up from that record rather than from the request. |
Estimates
| Quantity | Working | Result | What it sizes |
|---|---|---|---|
Average job length, from a figure GitHub states | GitHub's own CI on larger runners at peak: 125,000 build minutes an hour across about 15,000 jobs an hour (GitHub blog, 2023). 125,000 ÷ 15,000 ≈ 8.3 minutes per job | About 8.3 minutes per job, for GitHub's internal mix in 2023 | The one job-length figure the later rows use. It describes GitHub's own repositories, so treat it as an informed assumption for the whole platform and say so. |
How busy GitHub's peak fleet was | 4,500 concurrent runners × 60 minutes = 270,000 runner-minutes an hour available; 125,000 build minutes used; 125,000 ÷ 270,000 ≈ 46% | Under half of the fleet's minutes were used in the busiest hour | Headroom: a pool sized to its average load has no room for a burst, and a fleet that looks big is often waiting. Don't assume all runners are busy when you derive job length. |
Jobs in flight at GitHub's scale | 820 jobs per second (the daily average) × 8.3 minutes × 60 seconds ≈ 410,000 jobs running at once, if the platform mix resembles GitHub's own. GitHub gives no peak; an assumed peak of 3 × average gives about 1.2 million. Don't divide these by the interviewer's 10 pushes a second: that figure comes from a different scenario | About 410,000 runners busy on average, around 1.2 million at an assumed peak of three times average | The runner pool, one job per machine: the largest and most expensive part of the design. |
Draining one tenant's 10,000-job burst | 10,000 jobs × 8.3 minutes ≈ 83,000 runner-minutes. Under a cap of 1,000 jobs running at once (the ceiling GitHub lists as usual for one Linux larger-runner type), that is 10 waves × 8.3 minutes ≈ 83 minutes. In a single shared line with a pool of that size, a five-job team queued behind it starts after about 9 waves, roughly 75 minutes; with fair sharing between tenants it starts in the first wave | About 83 minutes for the burst under a 1,000-job cap; the five-job team waits about 75 minutes in one shared line, or about 8 minutes with fair sharing | Why per-tenant caps and fair sharing are the fix, and what the small team gains from them. |
Lease and heartbeat timing | GitHub re-queues a job whose assigned runner hasn't started it inside 60 seconds. Using the same order for a running job: heartbeats at 10-second intervals, a lease that lapses after 60 seconds, so a dead runner is noticed in about a minute. A 6-hour job restarted from scratch repeats up to 6 hours of work, so a lapse must be rare and a false alarm expensive | A design choice, not a sourced figure: heartbeats at 10-second intervals and a 60-second lease; a wrong requeue can cost up to 6 hours | How long a silent runner's job waits before another machine takes it, and why deploy steps need a fencing token rather than a shorter lease. |
Log volume a day | Assume 1 MB of log per job (an assumption; no source gives it). 71 million jobs × 1 MB ≈ 71 TB a day; ÷ 86,400 seconds ≈ 820 MB a second of appends on average. At an assumed retention of 90 days, 71 TB × 90 ≈ 6.4 PB. If each line were a 100-byte row, 1 MB is 10,000 rows a job, about 8.2 million row inserts a second | About 71 TB a day, 820 MB a second, roughly 6.4 PB kept at 90 days of retention | Why log lines belong in object storage, fed by a log service, and not in the main database as rows. |
Cache storage ceiling | GitHub's default cap is 10 GB per repository. 10 GB × 10 million repositories = 100 PB if all 10 million repositories filled the default cap; the realized total depends on how much of the cap repositories actually use, which no source gives | About 100 PB if all repositories sat at the default cap; a repository can raise its cap, so the true ceiling is higher | Object storage with per-repository caps and eviction of entries nobody has used recently, not a database. |
What a prebuilt machine image saves | GitHub reports its workflows took about 50 minutes without custom images and about 12 minutes with them: 50 − 12 = 38 minutes saved, 38 ÷ 50 ≈ 76% | About 38 minutes, roughly three quarters of the workflow | The value of warm images that already contain the source tree and compiled outputs, against the cost of rebuilding them several times a day. |
Core entities
| Entity | What it holds |
|---|---|
Event delivery | One row per webhook delivery from the source host, written before the host hears back. |
Workflow | A YAML file in the repository, read at the triggering commit, that says when to run and what jobs to create. |
Run | One execution of a workflow for one event: the unit developers see as a check on a commit. |
Job | A node in the run's dependency graph; the thing a runner leases and executes. |
Runner | A machine or VM that polls for work and runs one job at a time. |
Log chunk | A slice of one step's output, addressed by its byte offset so a viewer can resume. |
Blob entry | An artifact or a cache entry: metadata in the index, bytes in object storage. |
The API
| Endpoint | Notes |
|---|---|
| Verify the signature, insert the delivery with its id under a unique constraint, and answer with HTTP 202 once the row is committed. A duplicate id gets the same 202 and nothing else happens. The handler doesn't read the YAML or start jobs; that work belongs to the workflow service. |
| A long poll opened by the runner, so a customer's machine behind a firewall needs no inbound port. The scheduler answers with a job, a lease and its expiry, a per-job token for secrets and uploads, and the job's attempt number, our own counter (not a rerun count such as GitHub's run_attempt) that in this design increases on each lease grant (a requeue after a lapsed lease as well as a rerun) so it can serve as the fencing token. |
| Logs carry a step number and byte offset so a retried upload is harmless. Status posts also act as heartbeats and extend the lease; the server rejects them if the attempt number is stale. |
| The offset lets a browser resume after a reconnect without a gap or a repeat. While the job runs, lines come from the log service; once it finishes, the same call reads from object storage. |
| Cancel flips the job rows and the runner learns of it on its next heartbeat response. Rerun creates a new attempt for the failed jobs, which also invalidates any lease the old attempt held. |
The high-level design
Step 1 of 4
Take the event, expand the workflow, record the run
Start with the path from a push to a set of jobs. The source host posts a webhook to an event receiver, which checks the signature and hands the event to a workflow service. That service fetches the YAML files at the triggering commit, works out which workflows the event matches, expands each into a run with its jobs and their dependencies, and writes all of that to a runs-and-jobs database. The receiver does no build work itself, so a slow workflow can't block intake. The gap: the receiver has acknowledged before anything durable exists, so a crash between the acknowledgment and the write loses the push, and a host that resends the same delivery starts a second run.
What breaks first
The deep dives
Queueing jobs onto runners. Pushes arrive in bursts. One organization's merge-time burst is 10,000 jobs, and a small team has 5 waiting behind it. Who runs first, how does a job reach a machine, and how many machines does the pool need?
A concept that helps here: Fair queuing across tenants
A concept that helps here: Job scheduling and delayed workA concept that helps here: Backpressure and load sheddingA runner that dies mid-job. A machine running a deploy job goes silent halfway through. What notices, what runs again, and what stops the same machine from finishing the deploy later under a lease that has already lapsed?
Logs, artifacts and caches. A developer watches a build's log live while a thousand other jobs write theirs. How does a line get from the runner to the screen, where does it end up, and how do caches decide a saved result is safe to reuse?
Running untrusted code. A stranger's pull request against a public repository triggers its workflow. How does that job run without reaching another tenant's secrets or files, and how does a legitimate job get the credentials it needs?
Concepts this guide uses
- Fair queuing across tenants
- Job scheduling and delayed work
- Backpressure and load shedding
- Distributed locks and leases
- Heartbeats and presence
- Idempotency keys
- Blob storage and presigned uploads
- Content addressing and deduplication
- WebSockets, server-sent events and polling
- Sandboxing untrusted code
- Multi-tenant isolation
- Authorization models
Sources
- GitHub docs: Actions limits
- GitHub docs: Self-hosted runners reference
- GitHub blog: How GitHub runs its own CI on Actions larger runners
- GitHub blog: Let's talk about GitHub Actions
- Bazel docs: Remote caching
- GitHub docs: Using GitHub-hosted runners
- GitHub docs: Choosing the runner for a job
- GitHub docs: Understanding GitHub Actions
- GitHub docs: Events that trigger workflows
- GitHub docs: OpenID Connect
- GitHub docs: Secure use reference
- GitHub docs: Best practices for using webhooks
- GitHub docs: Webhook events and payloads
- Martin Kleppmann: How to do distributed locking
- GitHub docs: Dependency caching reference