How to design real-time fraud detection
The first sections are free. The deep dives, level expectations and quiz are part of Pro.
On this page
- The problem
- Clarifying questions
- The requirements menu
- Non-functional requirements
- Estimates
- Core entities
- The API
- The high-level design
- What breaks first
- The deep dives
- Entity fields
- Scoring inside the latency budget
- Fresh features and late labels
- Changing rules and models safely
- Variants
- What each level is expected to show
- The 30-second answer
- Quiz
The problem
A real-time fraud check sits on the payment path: before a card payment is sent for authorization, it gets a decision to approve, block or send to review, and the payment waits for that decision. The answer turns on a hard deadline of under 100 ms that the check has to meet at about 2,300 payments a second, which argues against any lookup whose cost grows with a card's history. Counts for the card, the account and the device are kept ready in a key-value store, read in parallel, scored by a model, then passed through rules that analysts can change without a deploy. The second half of the design is the learning loop: the truth about a payment arrives as a chargeback 30 to 90 days later, so the system records, for each decision, the feature values behind it, and labels are joined back to that record.
Clarifying questions
| Ask | Why it matters | A typical answer |
|---|---|---|
Where in the payment flow does the check run, and how long may it take? | A check before authorization has a hard deadline and blocks the customer's purchase; a check after authorization loses that deadline but has to reverse the payments it later judges fraudulent. The two are different systems. | Before authorization, on the payment path, with a budget of under 100 ms for the whole decision. |
How many payments at peak, and how rare is fraud? | The peak rate decides how many lookups a decision can afford, and the fraud rate decides how to judge the model and how much manual review a given block threshold produces. | About 2,300 payments a second at peak, about 1,350 on average over the busiest weekend, and roughly one payment per thousand fraudulent. |
What are the possible outcomes, and who acts on a review? | Three outcomes instead of two add a manual review queue, an analyst tool and a label source; they also give you a softer action than block when a rule is new. | Approve, block or review. Flagged payments go to a queue that analysts clear, and their verdicts come back as labels. |
When do we learn whether a decision was right? | Label delay shapes the whole training loop: how long decision logs are kept, how often retraining is worth doing, and whether earlier signals exist. | Chargebacks arrive 30 to 90 days after the payment. Card-network fraud reports can arrive a few days earlier than the chargeback itself. |
Who changes the rules, and how fast do they need to land? | If analysts ship rules, the rules are data with their own store, editor and rollout controls; if engineers do, they are code and wait for a deploy. | Analysts add and change rules without a deploy. Fraud tactics shift often, so a new rule should be live within minutes, in a mode that can be checked first. |
What happens to a payment when the check itself fails? | It fixes the fallback: whether a slow or dead risk service lets payments through, blocks them, or decides by a cheaper path, and whether that can differ per merchant. | The payment still gets a decision. Which decision is yours to argue for: let payments through, or stop them, for as long as the check is unavailable. Pick a side and say why. |
Non-functional requirements
| Requirement | What it forces |
|---|---|
A decision in under 100 ms, as Stripe Radar does | Nothing on the hot path may grow with a card's history: counts are precomputed and read by key, lookups run in parallel so latency is one round trip rather than a sum, and the call from the payment service into the risk check carries a timeout inside that budget. |
Fraud is rare: about 1 payment in 1,000 at Stripe | Accuracy is a useless yardstick (approving everything scores about 99.9%), so the model is measured by precision and by recall, training samples the good payments down, and a review queue sized for the rare positives is realistic. |
Few good payments blocked: Radar wrongly blocks about 0.1% of legitimate payments | Block is the expensive outcome, so a new rule or model runs in shadow before it acts, review rather than block is the softer action for uncertain cases when the analysts have capacity, and the block rate of each merchant and of each rule is alerted on so a bad rule can be caught early and switched off. |
A payment still gets a decision when the fraud check is slow or down | A timeout on the call and a defined fallback behind it: the rules by themselves, or a default chosen for each merchant. Slow actions such as bans and opening reviews leave the request through a queue so they can fail without taking the payment with them. |
Estimates
| Quantity | Working | Result | What it sizes |
|---|---|---|---|
Peak payment rate | 137,000 transactions a minute at the 2024 Black Friday to Cyber Monday peak ÷ 60 seconds ≈ 2,283 a second | About 2,300 payments a second | The load each component on the payment path sees at the worst minute: risk service, feature store reads and model calls. |
Average rate and peak-to-average ratio | 465 million transactions over the four-day weekend ÷ 345,600 seconds ≈ 1,345 a second; 2,300 ÷ 1,350 ≈ 1.7 | About 1,350 a second on average; the peak is about 1.7 times the average | How much headroom to provision: the peak is not far above the average, so the design is sized for the peak and runs near it for days. |
Feature store reads a second at peak | One read per entity (card, account, device, IP) = 4 reads a decision; 4 × 2,300 ≈ 9,200 reads a second | About 9,200 key-value reads a second, issued in parallel so each decision waits one round trip | The read load on the feature store, which the design spreads across partitions and replicas; set it against a history scan per payment on the main database, whose cost grows with each card's history. |
A latency budget inside 100 ms | An example split, assumed for this design rather than measured: payment service timeout on the risk call 80 ms, leaving 20 ms of the 100 for its own work. Inside the risk service, assume parallel feature reads 15 ms, model scoring 20 ms, rules 30 ms (300 rules at about 30 ms) = 65 ms, leaving 15 ms for network hops and serialization | A plan that fits, with the rules engine and the model as the two largest fixed costs | Whether the serial chain (features, then model, then rules) fits the budget, and which step to look at first when it does not. |
Decision log size | 116 million decisions a day; if the over 1,000 features are stored at about 4 bytes each, a row is about 4 KB; 116 million × 4 KB ≈ 464 GB a day; labels take up to 90 days, so 90 × 464 GB ≈ 42 TB must be kept before the last labels for a day's decisions arrive | About 464 GB a day, about 42 TB held until labels are complete | The decision log: a columnar store in object storage, not a row database, and a retention policy tied to the label delay. |
Fraud and false positives a day | 116 million a day ÷ 1,000 ≈ 116,000 fraudulent payments; 0.1% of about 116 million good payments ≈ 116,000 good payments wrongly blocked | About 116,000 fraudulent payments a day, and roughly as many good payments blocked by mistake | The cost of each side of the threshold: at this false-positive rate the blocked good payments are about as many as the fraud, so moving the threshold is a business decision, not a tuning detail. |
Accuracy of a model that approves everything | Fraud is 1 in 1,000, so approving all payments is right 999 times in 1,000 = 99.9% accurate while catching no fraud | 99.9% accuracy with zero fraud caught | Why the model is measured by precision and by recall at a chosen threshold instead of accuracy. |
Age of the newest fully labeled payments at training time | Chargebacks arrive 30 to 90 days after the payment, so a model trained today has complete chargeback labels for payments at least 90 days old, while labels for the last 30 days are still arriving | Training data lags the present by one to three months | The retraining cadence: hourly retraining brings almost no new labels, so weekly or monthly is the sensible range, with rules covering the gap. |
Core entities
| Entity | What it holds |
|---|---|
Payment | The request being judged: who is paying whom, with what, from where. |
Feature | One precomputed value about one entity, such as payments on this card in the last hour. |
Rule | An analyst's predicate over features and the score, with the action it takes and the state it is in. |
Decision | What the system said about one payment and the exact inputs it said it from. |
Label | A later verdict about a payment: a chargeback, a network fraud report or an analyst's call. |
The API
| Endpoint | Notes |
|---|---|
| The hot path, called by the payment service with a timeout inside the 100 ms budget. Idempotent on |
| In this design a new rule is saved with status shadow, and the analyst switches it to active with a later update once its logged results look right. The service validates the predicate syntax and can replay it over recent decisions to show what it would have matched. Rules are data: nothing here triggers a deploy. |
| Fed by the label pipeline, not by humans directly. Deduplicated on the external reference, because the same fraud report can arrive from the network and the processor. Appends to the label table that training and rule evaluation join to the decision log. |
| Backed by the review queue. In this design the list is ordered by amount and age, so large payments and cases close to expiring tend to be looked at first, and recording a verdict closes the case and posts a label with source review, so manual decisions feed training too. |
The high-level design
Step 1 of 4
A risk service on the payment path, with a timeout
Start with the call itself. The payment service, before it sends the payment to the card network, calls a risk service and waits for approve, block or review. The call carries a timeout well inside the 100 ms budget, so a slow check can't hold the payment hostage. The risk service runs the rules (block lists, velocity limits) and writes each decision to a log. This already settles the first design argument: the check is synchronous and has a deadline, so whatever we add later has to fit inside it.
What breaks first
The deep dives
Scoring inside the latency budget. A payment arrives and the check has under 100 ms to say approve, block or review. What runs, in what order, how long does each step take at about 2,300 payments a second, and what does the payment path do when the check is slow or down?
Fresh features and late labels. A stolen card is used ten times in one minute, and the chargeback for its first use arrives two months later. How does a count like payments per card in the last minute stay current, and how does a label that arrives in two months reach the model?
A concept that helps here: Feature stores and training-serving skew
A concept that helps here: Stream processing and windowsA concept that helps here: Approximate countingChanging rules and models safely. Fraudsters change tactics this week and an analyst needs a new rule live today, while a rule that misfires meets about 2,300 payments a second. How does a rule go from the analyst to live traffic, how is a new model shipped the same way, and how is a bad one caught and stopped?
A concept that helps here: Dynamic configuration
A concept that helps here: Staged rollouts and canaries
A concept that helps here: Stating a trade-offConcepts this guide uses
- Fan-out on write and on read
- Failing open or failing closed
- Walking one request through the design
- Feature stores and training-serving skew
- Stream processing and windows
- Approximate counting
- Dynamic configuration
- Staged rollouts and canaries
- Stating a trade-off