Experimentation and Champion-Challenger

How to measure what fraud controls actually prevent: holdouts, control groups, champion-challenger testing, counterfactual fraud estimation, sizing, ethics and guardrails.

Core components

The selection-bias problem

Diagram: All traffic, Control decides, No outcome observed (no label), Outcome observed (labels), Model training & performance metrics

Consequences:

Inflated precision. You only see blocked transactions' "badness" through proxies.
Unknown false positive rate. Good customers you blocked look the same as fraud you blocked.
Model drift toward the status quo. New models learn from data shaped by old controls.
No way to retire controls safely. Without measuring a control's marginal effect, you can't tell whether removing it would matter.

Experiment types

Type	Design	Measures	Cost
Global holdout	A small random share of traffic bypasses some or all fraud controls (or gets a lighter action)	True fraud rate in untreated traffic; total value of the control stack	Fraud losses in the holdout
Control-specific holdout	Random share bypasses one rule or model only	Marginal value of that control	Smaller, targeted losses
Champion-challenger	Traffic split between the current strategy (champion) and an alternative (challenger)	Relative performance: catch, FPR, friction, loss	Risk that the challenger underperforms
Action test	Same detection, different actions (decline vs step-up vs hold)	Friction and loss trade-off per action	Usually low
Threshold test	Random assignment to different thresholds near the cutoff	Marginal catch and friction at the margin	Low to moderate
Shadow comparison	Challenger runs silently; compare decisions	Agreement, and disagreement cases to review	Nearly free; no counterfactual on outcomes
Random review sampling	A random share of approved traffic routed to review	False-negative estimate	Ops cost

How to build it

Design checklist

Hypothesis and primary metric. E.g., "Challenger reduces dollar-weighted fraud loss on ACH funding by ≥10% without raising the insult rate."
Guardrail metrics. Insult rate, complaint rate, step-up abandonment, review volume, latency.
Unit of randomization. Usually randomize on the customer (or entity) so one fraudster can't appear in both arms.
Stable assignment. Hash-based on a stable ID so a customer stays in one arm.
Exclusions. Segments that must never be in a reduced-control arm (see Ethics below).
Exposure caps. Per-transaction, per-customer and total-loss limits on the holdout arm.
Duration and label maturity. Long enough to cover the label window for the rail (see Fraud Labeling and Feedback Loops).
Sample size and power. Calculated in advance (see below).
Stopping rules. Pre-agreed conditions to stop early for harm.
Analysis plan. Written before launch to avoid metric shopping.

Sizing

Fraud is rare, so detecting differences in fraud rates needs large samples. The rough intuition: to compare two proportions, the sample per arm grows with p(1 − p) and inversely with the square of the difference you want to detect. Halving the detectable difference quadruples the sample.

Illustrative sizing

Baseline fraud rate in the holdout population: 0.5%. You want to detect a reduction to 0.4% (a 20% relative change) at 95% confidence and 80% power. The standard two-proportion approximation gives roughly 70,000 units per arm. With a 1% holdout, that's about 7 million eligible units over the test period. Detecting a 10% relative change takes about four times as many.

Implications:

Use dollar-weighted and higher-base-rate segments where possible
Use longer durations or pool across periods
Consider intermediate outcomes with higher base rates (e.g., confirmed-bad by QA review, early return codes) as leading indicators
Accept that some questions can only be answered approximately

Run a proper power calculation for your numbers; the figures above only show the order of magnitude.

Holdout cost management

Partial holdouts. Instead of no control, give the holdout a lighter action: step-up instead of decline, hold instead of block.
Caps. Hard per-account and total exposure limits in the holdout arm.
Exclude catastrophic paths. Don't hold out controls whose failure causes irreversible large losses to customers (e.g., ATO protections on money-out).
Rotate. Rotate holdout membership so no customer stays in it indefinitely.
Budget it. Treat holdout losses as a planned research cost, agreed with finance.

Ethics and guardrails

Champion-challenger for models

Train the challenger; validate offline on matured, out-of-time data (see Model Risk Management)
Shadow the challenger alongside the champion
Review disagreement cases: where the challenger flags and the champion doesn't, and vice versa. Blind QA review of a sample gives early signal (see Fraud QA Program).
Route a small random share of traffic to the challenger's decisions, with caps
Compare on matured labels: dollar-weighted catch, FPR, friction, review volume
Promote, iterate or retire

Vendor bake-offs

Champion-challenger is also the right way to evaluate vendors: run the candidate in shadow on your traffic, compare disagreements, then a capped live split if needed. See Vendor Selection and Evaluation.

Who does what

Activity	Data science	Fraud strategy	Compliance / legal	Finance	Product
Experiment design	A/R	R	C	C	C
Exclusions and ethics review	C	R	A		C
Holdout budget	C	R		A
Launch and monitoring	R	A	I	I	I
Analysis	A/R	R		I	I
Decision (promote / retire)	R	A	C	I	C

Artifacts and templates

Experiment brief

Question and hypothesis
Primary metric and guardrails
Population, unit of randomization, exclusions
Arms and treatment definitions
Exposure caps and stopping rules
Sample size, power, duration and label maturity
Analysis plan
Ethics and compliance sign-off
Budget for expected holdout loss
Owner and decision date

Readout template

Section	Content
Result	Primary metric difference with confidence interval
Guardrails	Each guardrail with interval
Dollar impact	Annualized loss and friction impact
Segment effects	Where the effect differs
Caveats	Label maturity, novelty effects, attacker adaptation
Decision	Promote / iterate / retire

Metrics

Metric	Why
Share of material controls with a measured counterfactual in the last 12 months	Evidence base
Holdout loss vs budget	Cost control
Decisions informed by experiments	Program value
Controls retired after a test showed no marginal value	Portfolio hygiene

Pitfalls

Transaction-level randomization for customer-level fraud contaminates both arms.
Peeking and stopping on noise. Use pre-registered analysis and sequential-testing methods if you must look early.
Ignoring label maturity. Calling a winner before chargebacks and returns arrive.
Novelty effects. Attackers may not find the holdout immediately. Early results can understate holdout fraud.
Holdout discovery. If attackers can detect the holdout, they'll concentrate there. Keep assignment opaque and capped.
Holding out customer protections crosses an ethical and regulatory line. Test variants instead.

Sources and further reading

Revised Guidance on Model Risk Management, attachment (SR 26-2, April 2026): outcomes analysis and back-testing against real-world results; out-of-time testing; ongoing monitoring
Federal Reserve SR 11-7 (2011): the original validation framework, including outcomes analysis and benchmarking
Penn State STAT 800: Sample Size Computation: sample size scaling for proportions (the basis for the sizing intuition)
Abrigo, What SR 26-2 means for financial institutions: parallel testing when fraud and AML analytics change