How to design an A/B testing platform
The first sections are free. The deep dives, level expectations and quiz are part of Pro.
On this page
- The problem
- Clarifying questions
- The requirements menu
- Non-functional requirements
- Estimates
- Core entities
- The API
- The high-level design
- What breaks first
- The deep dives
- Entity fields
- Stable assignment at scale
- Exposures and metrics
- Guardrails and trustworthy results
- Variants
- What each level is expected to show
- The 30-second answer
- Quiz
The problem
An A/B testing platform lets product teams define an experiment, hands each user a variant that stays put across sessions and devices, and reports which variant moved the metrics. The design turns on one idea: a variant is computed, not stored. Hashing a user's id together with a salt unique to the experiment gives the same answer on any device and any server, so the hot path makes no network call and keeps working while the platform's backend is down. The rest is a pipeline: exposure and outcome events flow to a warehouse, get rolled up to one row per user and then compared per variant, and pass through a stats engine that checks the split, adjusts for repeated looks, and pauses a treatment that hurts a guardrail.
Clarifying questions
| Ask | Why it matters | A typical answer |
|---|---|---|
How many variant evaluations happen, and how many experiments run at once? | The evaluation rate decides whether assignment can afford a network call at all, and the experiment count decides whether a nightly join over raw events is realistic. | Up to 23 trillion evaluations a day, about 266 million a second, across about 35,000 running experiments, with about 50,000 assignment log writes a second. |
Does a variant have to be the same across a user's devices, or is per-device enough? | Cross-device stability rules out anything keyed on a device or cookie and pushes you to a deterministic function of the user id. | The same variant across sessions and devices for a signed-in user. Logged-out traffic is a variant you can name and park. |
Can a team change the traffic split while the experiment is running? | Ramping is normal practice, and it tests whether your assignment scheme keeps users in treatment when the split changes. | Yes. Teams ramp from a small share to a large one, and anyone already in treatment should stay there. |
How fresh do results need to be? | Daily results allow a batch pipeline; a crash-rate guardrail that has to fire within hours needs a faster path for a few metrics. | Most results are read about once a day. Guardrail metrics need to raise an alert or pause the experiment well before the next daily run. |
Do some experiments have to keep their users apart from each other? | Two experiments on the same screen can change each other's results, so you need a way to make them mutually exclusive without breaking randomization. | Yes, for experiments on one surface. Most other experiments can overlap freely. |
What should a product service do when it can't reach the platform? | It decides whether assignment lives inside the caller or behind a call, and what a user sees during an outage. | Keep serving variants from the last configuration it received, and fall back to the default experience for an experiment it doesn't recognize. |
Non-functional requirements
| Requirement | What it forces |
|---|---|
Assignment fits inside callers’ latency budgets: LinkedIn evaluates in 700 ns on average, 3 μs at p99 | Sub-microsecond is in-process territory: no network hop and no disk on the path, and nothing that waits on another request. In this design the variant comes from a hash computed inside the calling service against a configuration already in memory, and targeting that needs a remote attribute lookup is the exception, not the rule. |
Thousands of experiments at once: over 1,000 at Uber, about 35,000 at LinkedIn | Assignments across experiments must be independent, so each experiment needs its own salt rather than one shared hash. On the analysis side, a pipeline that joins raw exposures against raw outcomes once per experiment grows with the experiment count times the event volume; the design we take rolls events up to one row per user first and shares that work across experiments. |
Assignment keeps working when the platform’s backend is down | The SDK holds the last configuration it received and evaluates from it; nothing on the request path depends on the control plane or a store being reachable. Unknown experiments fall back to the default experience, and exposure logging is fire-and-forget to a stream so a slow sink can't block the caller. |
Results stay trustworthy: split checks, and no peeking without a sequential test | Each report runs a chi-squared check of observed counts against the configured split before it shows a number, and anything a team reads before the planned end date comes from a sequential test, not a fixed-horizon one. A report with many metrics, or several variants, gets a false-discovery correction before it shows a win. |
Estimates
| Quantity | Working | Result | What it sizes |
|---|---|---|---|
Assignment evaluations a second | 23 trillion a day ÷ 86,400 seconds ≈ 266 million a second | About 266 million evaluations a second | The request rate the assignment path has to serve, which rules out a network call per evaluation. |
How much heavier a remote call per evaluation would be | 266 million evaluations a second ÷ 800,000 network requests a second ≈ 330 | About 330 times the network load the platform's backend actually serves | Why evaluation moves into the caller's process: a central assignment service would carry hundreds of times the traffic that the real backend handles. |
CPU spent on local evaluation across the company | 266 million a second × 700 ns ≈ 186 core-seconds per second | About 190 cores' worth of compute, spread across all product services | The CPU cost of evaluating locally, which you can compare with the size of your own fleet; it is also the reason a 700 ns evaluation can sit on the request path. |
Assignment log rows a day | 50,000 writes a second × 86,400 seconds ≈ 4.3 billion | About 4.3 billion rows a day | The daily input to the exposure side of the metrics pipeline, and the reason it batches rather than joins per experiment. |
Savings from logging selected assignments, not each evaluation | 23 trillion evaluations a day ÷ 4.3 billion logged rows a day ≈ 5,300 | About 5,300 times fewer rows than logging each evaluation | Why the SDK logs exposures and selected assignments rather than each hash it computes. |
False wins expected across many metrics | Assuming the conventional 5% significance level: 1,000 metrics × 5% = 50 | About 50 metrics look significant by chance when nothing changed | Why a report with many metrics needs a false-discovery correction before anyone calls a win. |
Core entities
| Entity | What it holds |
|---|---|
Experiment | The definition a team writes: who is eligible, how traffic splits, what to measure, and the salt that makes assignment reproducible. |
Exclusive group | A set of experiments on one surface that must not share users: one salt, a fixed bucket space, and a ledger of which buckets belong to whom. |
Assignment | A user's variant in an experiment. Computed from the user id and the salt on demand, and not stored anywhere on the request path. |
Exposure event | A record that a user actually received a variant: the row the analysis starts from. |
Metric definition | How to turn raw events into one number per user, written once and reused across experiments. |
Result | One row per experiment, metric and variant, written by the stats engine and read by the dashboard. |
The API
| Endpoint | Notes |
|---|---|
| In this design we'd have creation check the definition (shares that add up, metrics that exist), mint the experiment's salt, and reserve bucket ranges if the experiment joins an exclusive group, then store the definition. From there the control plane pushes the config to SDKs; starting a test doesn't involve an assignment endpoint. |
| We'd keep the HTTP form for cases where embedding the SDK is awkward, such as a thin client or a one-off lookup while investigating a report. The SDK form is the hot path: in this design both forms are meant to read the same pushed config and apply the same hash, so a lookup over HTTP is for convenience, not a separate source of truth. |
| Fire-and-forget from the SDK to the event stream; we'd buffer and send in groups rather than one request per exposure. It carries the variant the SDK actually applied rather than one recomputed later, so a config that lagged still produces an honest record. Duplicates are expected and removed in the pipeline. |
| Reads rows the stats engine wrote to the results store, so a dashboard load is a lookup rather than a query over events. If the split check failed, we'd have the response carry the warning and hide the estimates. |
| Widens the treatment range; in this design each ramp also stamps a new config version so an exposure can record which step was in force. Ramping up keeps existing treatment users in treatment; ramping down moves some users to control, which the analysis has to treat as switching. |
The high-level design
Step 1 of 4
A control plane that pushes configs to an SDK
Start with the two ends that have to exist: a UI where a team defines variants, splits, targeting and metrics, and the product services that need a variant for each request. A control plane stores the definition and pushes it to an SDK embedded in each service. The SDK computes the variant from a hash of the user id and the experiment's salt against the ranges in its in-memory config. At about 266 million evaluations a second, a call to a central assignment service would carry roughly 330 times the traffic that the backend of the largest reported platform actually handles, so the hash runs in the caller's process and nothing on the request path depends on the platform being up.
What breaks first
The deep dives
Stable assignment at scale. A user opens the app on a phone and a laptop while the team ramps the experiment from 1% to 50%. How does each request arrive at a variant, where does that computation run, and what keeps both devices and both ramp steps consistent?
A concept that helps here: Deterministic bucketing with hashes and salts
A concept that helps here: Failing open or failing closedA concept that helps here: Cache invalidation and TTLsExposures and metrics. An experiment has run for a week and the product manager wants to know whether treatment won. Trace one exposure and one purchase from the SDK to the number on the dashboard, and say what the pipeline does with repeated, late and conflicting exposures.
Guardrails and trustworthy results. One treatment crashes the app for part of its users, and another shows a 2% lift that might be noise. What does the platform do about each, how does it notice a harmful treatment within hours, and how does it keep daily reading from inflating false wins?
A concept that helps here: Experiment statistics: split checks, peeking and corrections
A concept that helps here: Stream processing and windowsConcepts this guide uses
- Deterministic bucketing with hashes and salts
- Failing open or failing closed
- Cache invalidation and TTLs
- Stream processing and windows
- Delivery semantics: at most, at least, exactly once
- Queues and logs
- Experiment statistics: split checks, peeking and corrections
Sources
- LinkedIn Engineering: Making the LinkedIn experimentation engine 20x faster
- LinkedIn Engineering: A/B testing at LinkedIn, assigning variants at scale
- LinkedIn Engineering: Our evolution towards T-REX
- Spotify Engineering: Spotify's New Experimentation Platform, part one
- Spotify Engineering: Spotify's New Experimentation Platform, part two
- Spotify Engineering: Spotify's New Experimentation Coordination Strategy
- Uber Engineering: Under the Hood of Uber's Experimentation Platform
- Kohavi et al.: Online Controlled Experiments at Large Scale
- ExP Platform: CUPED (Deng, Xu, Kohavi and Walker)
- Wikipedia: False discovery rate