Manual Review Queues

What it is

Manual review is where your automated stack hands off to human judgment. When it works, reviewers see the 0.5% of decisions where context, conversation and judgment beat a model, and their decisions become the labels that improve the model. When it doesn't, it turns into an expensive backlog that delays good customers, rubber-stamps fraud under SLA pressure and feeds biased labels back into your models.

This note covers queue design as a control. Staffing, workforce management and operations are in Alert Review Operations. QA programs are in Fraud QA Program.

What it stops (and what it doesn't)

Review earns its cost when:

• The dollar exposure per decision is high enough to justify minutes of human time.
• Context the model lacks matters: customer contact, document inspection, account history narrative, external verification.
• The action is reversible or holdable while the review happens (payout holds, pending rewards, seller payout delays).
• A conversation with the customer can resolve the uncertainty, which is the main lever in scam and elder exploitation cases (Scam Victim Indicators).

Review doesn't help when:

• Volume is high and value is low (Card Testing, bot waves); automate these.
• The decision must be immediate and irreversible (instant payments without a hold option).
• Reviewers lack information the model doesn't already have. Then you're paying people to agree with the score.

Design considerations

Queue structure

Design queues by decision type and skill rather than by the rule that fired:

Prioritization by expected loss

First-in-first-out is the default and usually wrong. Prioritize by expected loss avoided per minute of review:

priority ∝ (probability of fraud × dollars at risk × time sensitivity) ÷ expected handling time

• Probability comes from calibrated model scores (Machine Learning Models). Raw scores that aren't calibrated make this unreliable.
• Dollars at risk is the exposure if released: pending payout, linked balances, downstream exposure.
• Time sensitivity captures SLA pressure, auto-release deadlines and customer impact.
• Keep an aging guardrail so low-priority cases still get reviewed within the SLA. Never let a held customer wait indefinitely.

Timeout behavior

Every queue needs a defined answer to "what happens if nobody reviews this in time?" Auto-release, auto-decline and escalate are all valid, depending on risk. The answer must be explicit, monitored and not visible to the customer as a predictable pattern. Fraudsters who learn a fixed auto-release window can simply wait it out.

Reviewer tooling

The biggest driver of review quality and speed is what the reviewer sees on one screen:

• Case summary with reason codes in plain language ("new device + new payout destination + full balance withdrawal within 2 hours").
• Account timeline: logins, changes, contacts, transactions, prior cases.
• Linked entities: shared devices, destinations and identities, with a link into Graph Analytics.
• Evidence panels by queue type: receipt image with extracted fields, check image, document, delivery proof.
• Similar past cases and their outcomes.
• Actions with required reason codes: release, hold, request info, step-up, restrict, close, escalate.
• Customer contact tools with scripts for scam and vulnerable-customer conversations.
• No raw PII beyond need (Insider Access Controls).

Labels and feedback

• Every decision records a structured outcome and reason code, which flow to Fraud Labeling and Feedback Loops.
• Treat "released" and "confirmed good" as separate outcomes. Track post-release outcomes to catch reviewer misses.
• Keep a random sample stream of low-score cases into review. It's the only unbiased estimate of what your automation misses, and it reduces model selection bias.

QA hooks

Build QA into the queue itself:

• Random QA sampling of decisions for a second review against a rubric (Fraud QA Program).
• Targeted QA on new reviewers, high-dollar releases, overrides of model recommendations and decisions made unusually fast.
• Calibration sessions on hard cases to align reviewers.
• Blind double review on a small slice to measure inter-reviewer agreement.
• Outcome-based QA, where released cases that later turned out to be fraud go back to the reviewer and QA as learning cases.
• Reviewer-level metrics (accuracy, consistency, handling time, release rate against peers), used for coaching rather than punishment.

Implementation path

Customer friction and trade-offs

Every case in a queue is a customer waiting, and the friction cost is time-to-decision, the uncertainty of not knowing why, and the contact cost of asking. Reduce it with clear messaging and expected times, self-service evidence upload, fast-tracking for trusted segments, and threshold tuning so review volume matches capacity. If your queue is always backlogged, check whether your thresholds are wrong or your automation is too weak before adding staff (Friction and Customer Experience Tradeoffs).

Measuring effectiveness

• Review precision: share of reviewed cases confirmed as fraud. Too low means you're reviewing good customers. Too high may mean thresholds are too conservative and fraud is getting through unreviewed.
• Dollar-weighted catch: fraud dollars stopped by review per reviewer hour.
• Miss rate: released cases later confirmed as fraud.
• QA accuracy and inter-reviewer agreement.
• SLA attainment and time-to-decision (median and p95).
• Timeout rate: cases resolved by timeout instead of review.
• Cost per review and cost per fraud dollar prevented.

See Fraud Metrics and KPIs.

Common failure modes

• FIFO queues where high-dollar cases wait behind low-value ones.
• Queues sized to whatever the rules generate rather than to capacity.
• Raw scores shown without reasons, so reviewers rubber-stamp the model.
• Predictable auto-release on timeout.
• Free-text decisions that can't be used as labels.
• QA run separately from operations, with no feedback loop.
• No random sample, so automation's blind spots are never seen.
• Reviewers under SLA pressure releasing to clear backlog.

Regulatory and policy notes

• Reg E timelines bound how long unauthorized-transfer investigations can take (10 business days, extendable to 45 days with provisional credit, longer for new accounts). Design SLAs accordingly (Regulation E and Unauthorized Transfers).
• Broker-dealer holds: under FINRA Rule 2165, holds on Specified Adults' accounts need notification within two business days, an immediate internal review, and documented records. Proposed Rule 2166 would add a fraud delay of up to 10 business days for all adults (Trusted Contact and Senior Investor Holds).
• SAR timelines: the SAR clock starts when review determines activity is suspicious, not when the alert fired. Queue design affects when that determination happens (BSA-AML and SAR Filing).
• Documentation: reviewers' rationale should be recorded in a form that supports audits, SARs and customer disputes (Case Documentation Standards).

Mechanisms addressed

Account Takeover (ATO) · Payout Redirection · Receipt Fraud (Cashback) · Check and Mobile Deposit Fraud · ACH Funding Fraud · Seller Non-Delivery and Fake Listings · Authorized Push Payment (APP) Scams · Elder Financial Exploitation · ACATS Transfer Fraud · Refund and Return Fraud · Synthetic Identity Fraud

Related controls

Case Management Systems · Machine Learning Models · Rules Engine · Withdrawal and Payout Controls · Funds Availability Holds · Receipt and Purchase Verification · Trusted Contact and Senior Investor Holds · Controls Index

See also Investigation Standards, Fraud Analyst Onboarding Curriculum, Workforce Management and Capacity Planning and SOP Writing Standards.

Sources and further reading

• Credit Card Fraud Detection: a Realistic Modeling and a Novel Learning Strategy (Dal Pozzolo et al.): limited investigator capacity (top-k alerts) as the operational constraint; feedback selection bias
• FFIEC BSA/AML Manual: SAR filing requirements: when the SAR clock starts; filing timelines
• CFPB: 12 CFR 1005.11, Procedures for resolving errors: Reg E investigation timelines
• FINRA Rule 2165: Financial Exploitation of Specified Adults: hold notification, internal review and recordkeeping requirements