Fraud QA Program

Core components

What QA is for

What falls in scope

Sampling design

There are three modes. Mature programs blend them and keep them separately labeled in reporting.

Sample size math

To estimate an error rate (a proportion) within ±e at a given confidence level:

n₀ = z² × p(1 − p) / e²

• z = 1.645 (90%), 1.96 (95%), 2.576 (99%)
• p = expected error rate. Use 0.5 if you don't know it; this is the most conservative choice.
• e = margin of error, as a proportion
• Always round up.

For a small, known population N (for example, one queue's monthly decisions), apply the finite population correction:

n = n₀ / (1 + (n₀ − 1) / N)

Worked examples:

What this means in practice:

• Ten cases per analyst per month are mostly noise. At about 90% accuracy, the 95% margin on 10 cases is roughly ±19 points. An analyst who scores 90% on 10 cases can't be statistically told apart from one at 75%. Use rolling 90-day windows for individual performance and keep monthly samples for coaching.
• A clean sample still allows for a real error rate. If you see 0 errors in n reviews, the 95% upper bound on the true error rate is about 3/n, the "rule of three". So 0 critical errors in 60 reviews is still consistent with a true rate of up to 5%.
• At small n, or with p near 0 or 1, the normal approximation behind the formula is poor. Report Wilson or exact binomial intervals in dashboards.
• Detecting a change is harder than estimating a level. Comparing accuracy before and after an SOP change needs a two-proportion test and usually much larger samples than either period alone.

Stratification

Stratify on the dimensions where errors differ, so each one gets enough samples to be reportable:

• Queue or mechanism. Receipt fraud reviews and withdrawal reviews fail in different ways.
• Outcome. This one matters most. Approvals usually dominate volume, so pure random sampling barely touches declines, holds and escalations. Errors are asymmetric: a false release costs money, a false block costs customers. Sample the two sides separately.
• Analyst and tenure. Set a per-analyst floor. New hires need heavier coverage.
• Dollar band. Make sure high-exposure decisions are represented.
• Channel or workforce. In-house vs outsourced vs offshore. Error patterns, and Insider Fraud exposure, often differ.

Blind re-decision vs open audit

• Blind re-decision. The QA reviewer sees the case as it stood at decision time, without the analyst's call, decides independently, then compares. This removes anchoring and is the right method for accuracy. It is slower.
• Open audit. The reviewer sees the analyst's decision and checks it against the SOP. This is faster and is fine for policy and documentation items.
• Many programs run blind re-decision on the decision item and open audit on everything else, in the same form.

Rubric design

Principles:

1. Keep the decision separate from the paperwork. Score them separately so a strong narrative can't offset a wrong decision.
2. Split critical from non-critical items. Critical items can fail the whole review.
3. Anchor every item to an SOP section, so disputes become arguments about what the SOP says.
4. Score items as binary or three-level (Met / Partially met / Not met), with written anchors. Five-point scales invite inconsistency.
5. Keep it short. Eight to twelve items. Every extra item lowers inter-rater reliability.

Sample rubric (generic, process-level)

Auto-fail conditions

Any of these sets the review score to 0, whatever the other items score:

• False release with clear evidence. Funds, rewards or goods released when the SOP required a hold or decline on the evidence present.
• Adverse action without basis. A closure, decline, clawback or hold with no documented reason the SOP recognizes.
• Skipped verification. An account change, recovery or payout-destination change processed without the required verification (see Customer Support Verification Protocols).
• Missed mandatory escalation. SAR referral, suspected elder exploitation, a law enforcement request, insider indicators.
• Privacy or access violation. Data accessed beyond case need, or account details disclosed to an unverified party.
• Regulatory clock breach caused by the analyst. For example, a missed Reg E provisional-credit or investigation deadline (see Regulation E and Unauthorized Transfers).

Error taxonomy

Tag every error by direction and root cause. The root-cause tag is what routes a finding to the team that owns the fix.

Calibration and inter-rater reliability

If two QA reviewers grade the same case differently, the QA score ends up measuring the reviewer. Calibration keeps reviewers aligned. Inter-rater reliability measures whether it worked.

Calibration sessions

• Cadence: weekly while a program or queue is new, then every two weeks. Run extra sessions after any SOP or rubric change.
• Format: 5 to 10 cases, deliberately enriched with edge cases and known disagreements. Every QA reviewer scores independently, often along with team leads and one or two senior analysts. Reveal the scores, discuss divergences, decide.
• Output: a calibration ruling log, a dated record of how ambiguous situations will be scored from now on. Rulings that change how decisions are made must flow into the SOP within a set window, or the log becomes a shadow SOP.
• QA of QA: a lead re-reviews a small random share of completed QA reviews (illustrative: 5% to 10%) to catch drift between sessions.

Cohen's kappa, briefly

Raw percent agreement overstates reliability when one answer dominates. If 90% of decisions are correct, two reviewers who rubber-stamp "correct" will agree most of the time by chance. Cohen's kappa corrects for chance agreement:

κ = (pₒ − pₑ) / (1 − pₑ)

where pₒ is the observed agreement and pₑ is the agreement expected by chance from each rater's marginal rates.

How to use it:

• McHugh (2012) argues the traditional interpretation bands are too lenient for consequential decisions. She treats kappa below about 0.60 as inadequate and 0.80+ as strong, and recommends reporting both kappa and percent agreement. Those are reasonable targets for a fraud QA team.
• Kappa falls when one category is rare, even with good reviewers. Compute it on calibration sets enriched with errors and edge cases rather than on random production samples.
• Use weighted kappa for ordinal items (Met / Partial / Not met) and Fleiss' kappa when more than two raters score the same set.
• Compute kappa per rubric item. A low score on one item (often "evidence reviewed" or "narrative quality") usually means the item's anchor is vague. Fix the anchor before blaming reviewers.

Disputes and appeals

An appeal path keeps QA honest and gives you data on rubric clarity.

• Window: a short, fixed dispute window after results are shared (illustrative: 3 to 5 business days).
• Who decides: never the original QA reviewer. Use a second reviewer for single cases and the calibration panel for anything that sets precedent.
• Every overturn triggers a rubric or SOP check. If the analyst was right and QA was wrong, something written down was ambiguous.
• Read the dispute rate both ways. A high overturn rate means rubric ambiguity or reviewer drift. A dispute rate near zero usually means analysts don't believe appeals work or fear retaliation. It rarely means QA is perfect.

Feedback and coaching loops

• Fast feedback. Feedback on a decision from six weeks ago changes little. Set a time-to-feedback target (illustrative: within 5 business days of the decision) and report on it.
• Coach on patterns. One miss is noise. Three misses on the same SOP step is a pattern. Keep a coaching log, then run a targeted re-sample after coaching to confirm the fix worked.
• Share at team level too. A monthly error-theme review, a "case of the week", and short refreshers when a theme crosses queues.
• Recognize good catches. QA that only reports failures makes analysts defensive and pushes them toward declining whenever unsure. That shows up as false blocks.
• Keep development and performance management apart. Use rolling windows with enough n for formal performance decisions (see the sample-size math). Don't put someone on a performance plan over 10 cases.
• Feed recurring knowledge gaps into Fraud Analyst Onboarding Curriculum and, for frontline teams, Support Team Fraud Training.

QA of automated decisions

Rules and models make most money-movement decisions. Extend QA to them, working with strategy and data science:

Linking QA findings back to the system

Keep a findings register separate from individual scores:

Two illustrative examples:

• Cashback app: QA finds receipt reviewers inconsistently rejecting receipts from a retailer that recently changed its receipt layout. Root cause: the reference examples in the SOP are out of date. Fixes: update the SOP, notify the receipt-verification owners in Receipt and Purchase Verification, and run a calibration session.
• Brokerage: QA finds a cluster of false blocks on ACAT-in customers whose transfer timing looks unusual but matches a known custodian's batch schedule. Fixes: tune the rule, and add an SOP note so analysts check transfer source before escalating.

Review the register monthly with strategy and ops leads. Every systemic finding needs an owner and a ticket. A useful program health metric is the share of QA findings that produced a system change rather than only coaching.

How to build it

Staffing and independence (illustrative, calibrate to your volumes)

• Ratio: roughly one QA reviewer per 10 to 20 front-line analysts, depending on case complexity and coverage targets.
• Time per review: blind re-decision typically takes 1x to 1.5x the original handle time. Documentation-only audits are faster.
• Calibration: 1 to 2 hours per reviewer per week.
• Independence: QA shouldn't report to the manager whose queue it grades. In larger organizations QA sits under a separate ops-excellence lead or, in regulated firms, has a dotted line to second-line oversight (see Three Lines of Defense).
• Who becomes a QA reviewer: senior analysts with strong calibration scores. Rotation keeps product knowledge fresh but makes consistency harder. A common middle path is a permanent QA core plus rotating guest reviewers.

Tooling

• A reproducible sampler with stratification, a logged random seed and the sample frame stored, so samples can be audited.
• A QA form inside the case management system, linked to the case and its decision-time snapshot.
• A calibration module: the same case assigned to many reviewers, scores hidden until all are submitted.
• A dispute workflow with timestamps.
• Dashboards with confidence intervals, plus exports to the analytics warehouse so QA data can be joined with labels and model scores.

Artifacts and templates

QA form outline (copy and adapt)

A. Header

• Case ID, queue, mechanism or alert type, decision date and time
• Analyst ID, analyst tenure band
• QA reviewer ID, QA date
• Sample stratum (random / outcome-balanced / risk-weighted / targeted) and selection probability
• Decision-time snapshot reference

B. Blind re-decision (completed before revealing the analyst's decision)

• Reviewer's disposition and action
• Confidence: high / medium / low
• Key evidence relied on, in one or two lines

C. Comparison

• Analyst's disposition and action
• Match? If not: error direction (false release / false block / right outcome, wrong path)

D. Rubric items (see the sample rubric), each with score and a comment field that is required on any miss

E. Critical error and auto-fail flags, each with the SOP reference

F. Root cause: analyst knowledge / judgment / SOP / tooling / rule or model / policy / other

G. Customer impact: none / friction only / funds delayed / funds lost / complaint filed. Estimated dollar impact.

H. Coaching note in plain language, written for the analyst

I. Dispute block: disputed (Y/N), date, reason, panel decision, final score

QA program charter outline

1. Purpose and objectives
2. Scope: queues, decision types, automated decisions
3. Sampling methodology and per-analyst floors
4. Rubric and scoring rules; auto-fail list
5. Calibration cadence and reliability targets
6. Dispute process and timelines
7. Feedback and coaching standards
8. Reporting: audiences, cadence, metrics, confidence intervals
9. Findings register and escalation path
10. Independence and reporting lines
11. Data retention and access controls for QA records

Monthly QA report outline

1. Headline accuracy (population-weighted) with 95% intervals, by queue
2. Critical error rate; false release and false block counts and dollars
3. Top 5 error themes with root cause and owner
4. Automated-decision QA: per-rule and per-score-band findings
5. Calibration: kappa by rubric item; rulings issued
6. Disputes: rate, overturn rate, notable rulings
7. Findings register: opened, closed, overdue
8. Asks: decisions needed from strategy, product or policy

Metrics

Pitfalls

• QA as gotcha. When QA feels punitive, analysts optimize for defensibility: they over-document and over-decline. False blocks rise and nobody connects it to QA.
• Scoring paperwork over judgment. Rubrics that are mostly documentation items reward careful writers who make wrong calls.
• Uncalibrated reviewers. Without calibration and kappa, analyst scores track which reviewer they drew.
• Hindsight grading. No decision-time snapshot, or reviewers who look at later outcomes.
• Sampling the easy cases. Pure random sampling in an approval-heavy queue mostly reviews obvious approvals, so accuracy looks excellent.
• Reporting biased samples as accuracy. Risk-weighted samples reported without reweighting.
• Ignoring automated decisions. The rules engine makes most of the calls and nobody QAs them.
• Findings with no owner. QA tells the same story every month because SOP, rule and tooling fixes have no route to the people who own them.
• Small-n performance management. Formal action taken on monthly samples too small to separate analysts.
• QA reporting to the queue owner. That creates a structural incentive to soften results.

Related

• Operations: Alert Review Operations · Investigation Standards · Case Documentation Standards · Manual Review Queues · Case Management Systems · SOP Writing Standards · Workforce Management and Capacity Planning
• Governance: Rule Governance and Change Management · Model Risk Management · Experimentation and Champion-Challenger · Three Lines of Defense
• Measurement: Fraud Metrics and KPIs · Fraud Labeling and Feedback Loops · Loss Taxonomy and Attribution
• People: Fraud Analyst Onboarding Curriculum · Support Team Fraud Training · Fraud Team Hiring and Career Paths
• Hub: Fraud Program Overview

Sources and further reading

• Penn State STAT 800: Sample Size Computation: sample size formula for estimating a proportion, conservative p = 0.5 and rounding up
• cochranSize package README (CRAN mirror): Cochran's formula and the finite population correction, with the n = 385 example
• McHugh, Interrater reliability: the kappa statistic (Biochemia Medica, 2012): kappa formula, why Cohen's bands are lenient, minimum of about 0.60, report kappa with percent agreement
• McHugh 2012 on PubMed Central: open-access copy of the same paper
• CFPB, Regulation E §1005.11 Procedures for resolving errors: investigation and provisional-credit clocks that QA should check on claims queues
• eCFR, 31 CFR 1020.320 Reports by banks of suspicious transactions: SAR timing and documentation requirements that make escalation a critical QA item
• COSO/ACFE Fraud Risk Management Guide, 2nd ed., Executive Summary (2023): the monitoring-activities principle; ongoing evaluation and reporting deficiencies