Skip to content

How to design an A/B testing platform

The first sections are free. The deep dives, level expectations and quiz are part of Pro.

On this page
Practice this question

The problem

An A/B testing platform lets product teams define an experiment, hands each user a variant that stays put across sessions and devices, and reports which variant moved the metrics. The design turns on one idea: a variant is computed, not stored. Hashing a user's id together with a salt unique to the experiment gives the same answer on any device and any server, so the hot path makes no network call and keeps working while the platform's backend is down. The rest is a pipeline: exposure and outcome events flow to a warehouse, get rolled up to one row per user and then compared per variant, and pass through a stats engine that checks the split, adjusts for repeated looks, and pauses a treatment that hurts a guardrail.

Clarifying questions

AskWhy it mattersA typical answer

How many variant evaluations happen, and how many experiments run at once?

The evaluation rate decides whether assignment can afford a network call at all, and the experiment count decides whether a nightly join over raw events is realistic.

Up to 23 trillion evaluations a day, about 266 million a second, across about 35,000 running experiments, with about 50,000 assignment log writes a second.

Does a variant have to be the same across a user's devices, or is per-device enough?

Cross-device stability rules out anything keyed on a device or cookie and pushes you to a deterministic function of the user id.

The same variant across sessions and devices for a signed-in user. Logged-out traffic is a variant you can name and park.

Can a team change the traffic split while the experiment is running?

Ramping is normal practice, and it tests whether your assignment scheme keeps users in treatment when the split changes.

Yes. Teams ramp from a small share to a large one, and anyone already in treatment should stay there.

How fresh do results need to be?

Daily results allow a batch pipeline; a crash-rate guardrail that has to fire within hours needs a faster path for a few metrics.

Most results are read about once a day. Guardrail metrics need to raise an alert or pause the experiment well before the next daily run.

Do some experiments have to keep their users apart from each other?

Two experiments on the same screen can change each other's results, so you need a way to make them mutually exclusive without breaking randomization.

Yes, for experiments on one surface. Most other experiments can overlap freely.

What should a product service do when it can't reach the platform?

It decides whether assignment lives inside the caller or behind a call, and what a user sees during an outage.

Keep serving variants from the last configuration it received, and fall back to the default experience for an experiment it doesn't recognize.

The requirements menu

RequirementScopeWhy

Create an experiment: variants, traffic split, targeting and metrics

Core

The control plane's job. Everything downstream reads this definition: the SDK for the split, the pipeline for the metrics.

Give each user a variant that stays the same across sessions and devices

Core

Without it the experiment measures nothing: a user who flips between variants pollutes both groups. It is also what makes assignment a pure function of the user id attractive: a stored sticky assignment would work too, but at this evaluation rate a lookup per call is the expensive way to get the same stability.

Log an exposure when a user actually gets the variant

Core

Results compare people who experienced the change, not people the hash happened to place in a group. The trigger point is a design decision in its own right.

Compute each metric per variant, with confidence intervals

Core

A point estimate without an interval can't tell a win from noise. This is the pipeline plus the stats engine.

Ramp traffic up, and alert or pause when a guardrail metric gets worse

Core

Ramping limits the blast radius of a bad change, and the guardrail check is how the platform notices harm before a human reads a report.

Mutually exclusive experiments on one surface

Optional

Worth taking if the interviewer asks about two tests on the same screen. It costs you a shared bucket space and some wasted traffic; skip it if the round is already deep in the metrics pipeline.

Holdbacks that keep a slice of users out of new experiments for a period

Optional

Lets a team measure the combined effect of a quarter's launches. Take it if exclusivity is already on the board, since it uses the same bucket machinery; otherwise it is a sentence and a pointer.

Gradual rollouts of a shipped change under the same guardrails

Optional

The assignment and monitoring pieces already exist, so this is cheap to add and shows you see the platform as the company's release safety net. Leave it if time is short.

Pre-experiment checks: an A/A comparison, or a look at group activity before the start

Optional

Catches a broken randomizer before anyone trusts a result. Name it when you discuss trust; building it is a small addition to the stats engine.

Experiments on logged-out visitors keyed by a device or cookie id

Out of scope

Real, but a different identity problem: the same person looks like several users. Say you would hash on whatever id the caller passes, and move on.

Multi-armed bandits

Out of scope

Bandits shift traffic toward the leader while the test runs, which changes the statistics and the assignment scheme. Name them as a later mode, not part of this design.

Choosing which metrics matter to the business

Out of scope

A product and data science question. The platform stores metric definitions and computes them; which ones a team picks is theirs.

Feature flags beyond experiments

Out of scope

The same config push can carry plain flags, and it is tempting to design the flag service too. Mention the overlap and keep the round on experiments.

Causal inference without randomization

Out of scope

Synthetic controls and difference-in-differences serve cases where you can't randomize. Different math, different pipeline; park it.

Non-functional requirements

RequirementWhat it forces

Assignment fits inside callers’ latency budgets: LinkedIn evaluates in 700 ns on average, 3 μs at p99

Sub-microsecond is in-process territory: no network hop and no disk on the path, and nothing that waits on another request. In this design the variant comes from a hash computed inside the calling service against a configuration already in memory, and targeting that needs a remote attribute lookup is the exception, not the rule.

Thousands of experiments at once: over 1,000 at Uber, about 35,000 at LinkedIn

Assignments across experiments must be independent, so each experiment needs its own salt rather than one shared hash. On the analysis side, a pipeline that joins raw exposures against raw outcomes once per experiment grows with the experiment count times the event volume; the design we take rolls events up to one row per user first and shares that work across experiments.

Assignment keeps working when the platform’s backend is down

The SDK holds the last configuration it received and evaluates from it; nothing on the request path depends on the control plane or a store being reachable. Unknown experiments fall back to the default experience, and exposure logging is fire-and-forget to a stream so a slow sink can't block the caller.

Results stay trustworthy: split checks, and no peeking without a sequential test

Each report runs a chi-squared check of observed counts against the configured split before it shows a number, and anything a team reads before the planned end date comes from a sequential test, not a fixed-horizon one. A report with many metrics, or several variants, gets a false-discovery correction before it shows a win.

Estimates

QuantityWorkingResultWhat it sizes

Assignment evaluations a second

23 trillion a day ÷ 86,400 seconds ≈ 266 million a second

About 266 million evaluations a second

The request rate the assignment path has to serve, which rules out a network call per evaluation.

How much heavier a remote call per evaluation would be

266 million evaluations a second ÷ 800,000 network requests a second ≈ 330

About 330 times the network load the platform's backend actually serves

Why evaluation moves into the caller's process: a central assignment service would carry hundreds of times the traffic that the real backend handles.

CPU spent on local evaluation across the company

266 million a second × 700 ns ≈ 186 core-seconds per second

About 190 cores' worth of compute, spread across all product services

The CPU cost of evaluating locally, which you can compare with the size of your own fleet; it is also the reason a 700 ns evaluation can sit on the request path.

Assignment log rows a day

50,000 writes a second × 86,400 seconds ≈ 4.3 billion

About 4.3 billion rows a day

The daily input to the exposure side of the metrics pipeline, and the reason it batches rather than joins per experiment.

Savings from logging selected assignments, not each evaluation

23 trillion evaluations a day ÷ 4.3 billion logged rows a day ≈ 5,300

About 5,300 times fewer rows than logging each evaluation

Why the SDK logs exposures and selected assignments rather than each hash it computes.

False wins expected across many metrics

Assuming the conventional 5% significance level: 1,000 metrics × 5% = 50

About 50 metrics look significant by chance when nothing changed

Why a report with many metrics needs a false-discovery correction before anyone calls a win.

Core entities

EntityWhat it holds

Experiment

The definition a team writes: who is eligible, how traffic splits, what to measure, and the salt that makes assignment reproducible.

Exclusive group

A set of experiments on one surface that must not share users: one salt, a fixed bucket space, and a ledger of which buckets belong to whom.

Assignment

A user's variant in an experiment. Computed from the user id and the salt on demand, and not stored anywhere on the request path.

Exposure event

A record that a user actually received a variant: the row the analysis starts from.

Metric definition

How to turn raw events into one number per user, written once and reused across experiments.

Result

One row per experiment, metric and variant, written by the stats engine and read by the dashboard.

The API

EndpointNotes

POST /v1/experiments { variants, splits, targeting, metrics } → { experiment_id }

In this design we'd have creation check the definition (shares that add up, metrics that exist), mint the experiment's salt, and reserve bucket ranges if the experiment joins an exclusive group, then store the definition. From there the control plane pushes the config to SDKs; starting a test doesn't involve an assignment endpoint.

GET /v1/assignments?user_id&experiment → { variant }, also evaluated inside an SDK

We'd keep the HTTP form for cases where embedding the SDK is awkward, such as a thin client or a one-off lookup while investigating a report. The SDK form is the hot path: in this design both forms are meant to read the same pushed config and apply the same hash, so a lookup over HTTP is for convenience, not a separate source of truth.

POST /v1/exposures { user_id, experiment, variant, at }

Fire-and-forget from the SDK to the event stream; we'd buffer and send in groups rather than one request per exposure. It carries the variant the SDK actually applied rather than one recomputed later, so a config that lagged still produces an honest record. Duplicates are expected and removed in the pipeline.

GET /v1/experiments/{id}/results → [{ metric, variant, estimate, interval }]

Reads rows the stats engine wrote to the results store, so a dashboard load is a lookup rather than a query over events. If the split check failed, we'd have the response carry the warning and hide the estimates.

POST /v1/experiments/{id}/ramp { percent }

Widens the treatment range; in this design each ramp also stamps a new config version so an exposure can record which step was in force. Ramping up keeps existing treatment users in treatment; ramping down moves some users to control, which the analysis has to treat as switching.

The high-level design

push configExperiment UIControl planeSDK in eachservice

Step 1 of 4

A control plane that pushes configs to an SDK

Start with the two ends that have to exist: a UI where a team defines variants, splits, targeting and metrics, and the product services that need a variant for each request. A control plane stores the definition and pushes it to an SDK embedded in each service. The SDK computes the variant from a hash of the user id and the experiment's salt against the ranges in its in-memory config. At about 266 million evaluations a second, a call to a central assignment service would carry roughly 330 times the traffic that the backend of the largest reported platform actually handles, so the hash runs in the caller's process and nothing on the request path depends on the platform being up.

pushvariantexposuresoutcomesalert or pausereadExperiment UIControl planeExperimentconfigsSDK in eachserviceApp requestEvent streamProduct servicesWarehouseGuardrail monitorMetrics pipelineStats engineResults store
The whole design

What breaks first

The deep dives

Stable assignment at scale. A user opens the app on a phone and a laptop while the team ramps the experiment from 1% to 50%. How does each request arrive at a variant, where does that computation run, and what keeps both devices and both ramp steps consistent?

A concept that helps here: Failing open or failing closedA concept that helps here: Cache invalidation and TTLs

Exposures and metrics. An experiment has run for a week and the product manager wants to know whether treatment won. Trace one exposure and one purchase from the SDK to the number on the dashboard, and say what the pipeline does with repeated, late and conflicting exposures.

A concept that helps here: Stream processing and windowsA concept that helps here: Delivery semantics: at most, at least, exactly onceA concept that helps here: Queues and logs

Guardrails and trustworthy results. One treatment crashes the app for part of its users, and another shows a 2% lift that might be noise. What does the platform do about each, how does it notice a harmful treatment within hours, and how does it keep daily reading from inflating false wins?

A concept that helps here: Stream processing and windows

Concepts this guide uses

Sources

  1. LinkedIn Engineering: Making the LinkedIn experimentation engine 20x faster
  2. LinkedIn Engineering: A/B testing at LinkedIn, assigning variants at scale
  3. LinkedIn Engineering: Our evolution towards T-REX
  4. Spotify Engineering: Spotify's New Experimentation Platform, part one
  5. Spotify Engineering: Spotify's New Experimentation Platform, part two
  6. Spotify Engineering: Spotify's New Experimentation Coordination Strategy
  7. Uber Engineering: Under the Hood of Uber's Experimentation Platform
  8. Kohavi et al.: Online Controlled Experiments at Large Scale
  9. ExP Platform: CUPED (Deng, Xu, Kohavi and Walker)
  10. Wikipedia: False discovery rate
Sign in to report a mistake