Fraud Labeling and Feedback Loops

Where fraud labels come from, how long each rail takes to mature, how label noise and selection bias corrupt models, and how to build feedback loops that improve decisions.

Core components

Label sources and what they actually mean

Source	What it tells you	Typical noise	Use as
Card fraud disputes / TC40-style fraud reports	Cardholder or issuer says unauthorized	A meaningful share are first-party (see Chargeback and Friendly Fraud); reason code ≠ mechanism	Third-party fraud label after first-party screening
Non-fraud card disputes	Service, delivery or quality complaint	Some are disguised fraud; some are real merchant failure	Separate dispute label, not fraud
ACH returns R05/R07/R10/R11/R29	Receiver or RDFI says unauthorized or improper	R10 can be first-party "buyer's remorse"; R11 includes authorization-terms errors	Unauthorized-funding label, with ownership-match context
ACH returns R01/R09 (NSF/uncollected)	Insufficient funds	Kiting and first-party fraud hide here	Credit or first-party candidate; needs intent rules
ACH R02/R03/R04	Closed, no account, invalid number	Mule account churn, synthetic accounts, typos	Administrative; fraud signal in combination
Check/RDC returns	Counterfeit, altered, duplicate, NSF	Return reason codes are coarse	Deposit-fraud label after investigation
Reg E claims	Customer asserts unauthorized EFT or error	Some are scams (authorized), some first-party	Claim label + investigated outcome
Customer-reported ATO / scam reports	Customer says "not me" or "I was tricked"	Late and incomplete; reporting varies by segment	High-value label; record initiator facts
Investigator decisions	Analyst confirmed fraud or not	Analyst error, inconsistent SOPs, hindsight bias	Primary label for blocked/reviewed items, QA-sampled
Manual review outcomes	Approved, declined, escalated	Reviewing ≠ confirming; approvals are rarely re-checked	Weak label unless paired with matured outcome
Enforcement actions	Account closed or offboarded for fraud	Policy actions mix fraud, abuse, ToS and AML	Label only with coded reason
Consortium / network feedback	Account or device flagged elsewhere	Different definitions, staleness	Feature, not label
Partner/merchant reversals	Retailer or affiliate reversed commission or purchase	Returns are mostly legitimate	Rewards-abuse candidate (see Receipt Fraud (Cashback))
Holdout outcomes	What happened to flagged traffic you let through	Small samples	Gold standard for precision of blocking rules

Label maturity windows by rail

Maturity window = how long after a transaction you must wait before its label is "final enough" to train or evaluate on. Set windows from your own vintage curves (see Fraud Metrics and KPIs). The ranges below are illustrative; calibrate to your data.

Money path / rail	Main late signals	Rule-defined outer bound (verify current rules)	Practical maturity window
Card purchase / card funding	Fraud and non-fraud disputes	Network dispute windows, commonly cited as 120 days for card-absent fraud (see Card Network Dispute Rules)	90–120+ days
ACH debit funding	Returns; extended unauthorized returns	Most returns 2 banking days; consumer unauthorized via extended window ≈ 60 days (see Nacha Rules and ACH Risk)	60–75 days
Check / RDC	Returns, duplicate presentment, late altered-item claims	Varies; some claims arrive months later	30–90 days, with long tail
P2P / RTP / FedNow push (outbound)	ATO claims, scam reports, Reg E claims	Reg E notice keyed to 60 days after the statement (see Regulation E and Unauthorized Transfers)	60–90 days
Payout / withdrawal	ATO claims, account-owner complaints, inbound mule reports	Depends on rail	30–90 days
Cashback / rewards	Merchant/affiliate reversals, returns, receipt audits	Program terms	30–120 days
Marketplace orders	INR/SNAD claims, chargebacks	Buyer protection windows + card windows	30–180 days
Brokerage trades after ATO	Customer complaint, statement review	Statement cycle	30–60 days

Two dates per label. Store event_time (when the transaction happened) and label_time (when you learned). Training and backtests must use only labels known as of the evaluation date to avoid leakage. A backtest that uses labels that arrived after the simulated decision date overstates performance.

Selection bias: the blocked-transaction problem

Your model decides what gets labeled. Approved transactions mature into chargebacks or returns, and declined transactions never do. Over time:

The model only learns about fraud it missed, so it underweights patterns it already blocks well.
If you retrain without the blocked population, those patterns fade from the training data and the model "forgets" them. That's a classic hidden feedback loop.
Rules that block entire segments make those segments look fraud-free.

Diagram: Model / rules, Decision, Outcome observed (chargeback, return, claim), No outcome observed, Training data

Mitigations:

Holdouts / exploration. A small randomized share of flagged traffic is approved with exposure caps, so outcomes can be observed. Log the propensity so you can reweight. See Experimentation and Champion-Challenger.
Label blocked items by review. Send a sample of blocked traffic to review purely for labeling, then QA it.
Reject inference. Statistical inference of labels for blocked items. Useful, but it carries assumptions; document them for Model Risk Management.
Keep historical "known fraud" patterns in training even when current controls block them.
Log decision context. Record which rule or model version acted, the score, and the features as served. See Fraud Data Architecture.

Label quality

Problem	Example	Control
First-party contamination	Friendly-fraud chargebacks labeled third-party	First-party screens: prior undisputed history on same device/address; customer contact patterns
Scam vs unauthorized mixing	Coached transfers labeled ATO	Record facts: who initiated, whose device, credentials shared?
Analyst inconsistency	Two analysts label the same pattern differently	Calibration sessions and inter-rater agreement (kappa) via Fraud QA Program
Hindsight labeling	Account later closed for fraud, so all prior activity labeled fraud	Label at transaction level with dates. Don't back-propagate account status blindly.
Lag-truncated negatives	Recent transactions labeled "good" before maturity	Exclude immature transactions, or use survival/time-aware methods
Taxonomy drift	New reason codes or mechanisms split mid-year	Versioned taxonomy, mapping tables (Loss Taxonomy and Attribution)

Feedback loop design

A good feedback loop uses fast labels where it can and matured labels where it must, and records which kind it's using.

Loop	Latency	Purpose
Analyst decision → features	Minutes	Real-time "this device/payee was just confirmed bad" features. See Negative Lists and Deny Lists.
Early labels → sentinels	Hours–days	Customer ATO reports, early returns and review confirmations power early-warning monitoring before chargebacks arrive. See Playbook - Novel Attack Detection Members.
Matured labels → retraining	Weeks–months	Model training on mature, de-biased labels
QA findings → SOP and rules	Weekly	Fix systematic errors at the source
Loss attribution → roadmap	Monthly/quarterly	Fund controls where loss concentrates
Consortium sharing	Varies	External labels and indicators. See Consortium and Data Sharing; for 314(b)-eligible institutions see BSA-AML and SAR Filing.

How to build it

Stage	Labels	Next step
Crawl	Chargebacks and returns in a spreadsheet	Create a label table: one row per transaction, with source, event_time and label_time
Walk	Joined label table; investigator outcomes captured	Add first-party screens; define maturity windows per rail; record initiator facts
Run	Versioned label store; holdouts; QA-measured label accuracy	Propensity logging; point-in-time training sets; label-quality dashboard
Fly	Labels feed real-time features and retraining automatically	Time-to-label tracked; consortium label exchange; reject inference validated

Data science owns label definitions for models, ops/QA owns investigator label quality, and data engineering owns the label store and point-in-time joins.

Artifacts and templates

Label record schema (minimum)

Field	Notes
txn_id / account_id	Join keys
event_time / label_time	Both required
label_source	chargeback, ach_return:R10, reg_e_claim, review, customer_report, holdout
label_value	fraud_3p, fraud_1p, scam, abuse, dispute_non_fraud, credit, legit, unknown
initiator / actor / mechanism	Per Loss Taxonomy and Attribution
confidence	confirmed, probable, weak
decision_context_id	Links to the decision log (model version, rules fired, score)
superseded_by	When a later label overrides

Label QA checklist

Maturity windows documented per rail and re-estimated at least annually
Training sets exclude immature negatives
Backtests use labels known as of the simulated decision date
First-party screen applied before using card disputes as third-party labels
Blocked traffic has some labeled sample (review or holdout), with the rate documented
Inter-rater agreement measured for analyst labels each quarter
Label taxonomy versioned; breaks flagged in model documentation

Metrics

Label coverage: % of actioned and approved transactions with a resolved label at maturity
Time to label: median days from event to first label, by source
Label agreement: investigator vs QA re-label agreement; rule-attributed vs investigated
First-party share among card fraud disputes (estimated)
Holdout rate and holdout loss: cost of exploration, shown explicitly
Label reversal rate: labels later superseded

Pitfalls

Using reason codes as ground truth. A reason code records what the claimant asserted, which may differ from what an investigation would find.
Training on yesterday's data. Recent negatives are full of future positives.
No holdout because "we can't let fraud through". Then you can't know your precision. Size holdouts by exposure, since the same percentage of traffic can carry very different dollar risk.
Labels that live only in case notes. Free text can't feed a model. Require coded fields (see Case Documentation Standards).
Letting the model's own decisions define "fraud". That's circular. Review-confirmed and matured outcomes must dominate.
Ignoring abuse. Promo and referral abuse rarely produce chargebacks. Without explicit abuse labels, models never learn it.

Sources and further reading

Nacha, Differentiating Unauthorized Return Reasons: R10 vs R11 meanings and R11's inclusion in unauthorized return rates
Nacha, Unauthorized Return Rate guidance (Mar 2021): the unauthorized return codes and 60-day measurement periods
Adyen, Visa Compelling Evidence 3.0 FAQ: prior undisputed transactions and matching data elements as first-party evidence
Mastercard, First-Party Trust announcement (Oct 2023): network recognition that many fraud disputes are first-party
CFPB, Regulation E §1005.11 error resolution: claim notice and investigation timelines that set label arrival for EFT claims
Federal Reserve SR 26-2 attachment (Apr 2026): outcomes analysis and ongoing monitoring expectations for models
McHugh, Interrater reliability: the kappa statistic (2012): measuring analyst label agreement