Vendor Selection and Evaluation
How to decide build vs buy for fraud capabilities, run a proof of concept that measures real lift, test vendor claims, and negotiate contracts that protect data and exit.
Core components
Build vs buy
| Factor | Lean buy | Lean build |
|---|---|---|
| Data advantage | Vendor sees cross-industry or network data you can't (consortium, device reputation, bank account intel) | Your first-party data is the main signal (behavior on your app, your money paths) |
| Speed | You need coverage in weeks | You can invest quarters |
| Differentiation | Commodity capability (OTP delivery, basic email risk) | Core to your product's economics (payout risk, reward logic) |
| Explainability / control | Acceptable to receive a score plus reasons | You need full feature-level control, e.g., for model governance |
| Talent | No in-house DS/eng capacity | Strong DS and platform teams |
| Cost curve | Low volume, so per-call pricing is fine | High volume, so per-call pricing dominates the budget |
| Regulatory | Vendor provides certified or audited processes (e.g., IDV for CIP) | You must own the logic end to end |
A common pattern is to buy signals and build decisions. Purchase data you can't generate yourself (device, consortium, identity, bank account ownership), and keep orchestration, policy and the final model in-house where your money-path context lives. See Real-Time Decisioning Architecture.
Capability categories and what to probe
| Category | Key questions | Related |
|---|---|---|
| Device and network intelligence | Persistence of IDs across resets; emulator/automation detection; coverage on your platforms; SDK size and latency | Device Fingerprinting, Emulator and Rooted Device Detection |
| Identity verification / document + liveness | Pass rates by demographic and document type; deepfake/injection resistance claims and evidence; manual fallback | Document Verification and Liveness, Identity Verification Outcomes |
| Email / phone / address risk | Data freshness; carrier signals (SIM swap, port) coverage; match logic | Email Risk Signals, Phone Risk Signals |
| Bank account verification and ownership | Coverage of institutions; ownership match method; returns-risk scoring | Bank Account Ownership Verification, Funding Source Signals |
| Consortium / network data | Contribution requirements; label definitions; dispute handling; legal basis for sharing | Consortium and Data Sharing, Credit Bureau and Consortium Signals |
| Rules engine / decision platform | Latency; versioning; shadow mode; backtesting; decision log export | Rules Engine |
| ML platform / vendor models | Custom vs consortium model; retraining cadence; feature transparency; drift monitoring | Machine Learning Models, Model Risk Management |
| Case management | Data model flexibility; coded outcomes; SAR workflow; QA sampling | Case Management Systems |
| Chargeback / dispute management | Evidence automation; network program integrations; win-rate attribution | Card Network Dispute Rules |
| Behavioral biometrics | Coverage, latency, privacy posture, consent model | Behavioral Biometrics |
Evaluating vendor claims
Treat every performance claim as a hypothesis until it's tested on your traffic.
| Claim | Ask | Red flag |
|---|---|---|
| "Catches 95% of fraud" | On whose data, which fraud types, what label definition, what false positive rate, what time window? | Recall quoted without precision or FPR |
| "Reduces false positives 50%" | What was the baseline, and what was the measurement method? | Baseline was a naive rule set |
| "AI-powered, real-time" | What's the p99 latency at your volume? What happens on timeout? | No SLA, or SLA on averages only |
| "Network of X billion transactions" | How much overlaps your customers and rails? How fresh? | Big numbers with no overlap analysis |
| "Explainable" | Reason codes per decision? Stable? Documented? | Generic reasons ("high risk") |
| "Deepfake-proof" / "spoof-proof" | Independent testing? Which attack classes? | Absolute claims; no third-party evaluation |
| Case studies | Similar industry, rails and fraud mix? | Different vertical; no numbers |
Running a proof of concept
- Hypothesis and success criteria up front. For example: "At equal approval rate, reduce payout loss bps by ≥ X on new-destination payouts, with p99 latency ≤ Y ms." Write it before seeing results.
- Retro test (blind). Send the vendor a historical sample with labels withheld. Score it, then evaluate yourself on mature labels (see Fraud Labeling and Feedback Loops). Include known-fraud and known-good, plus recent attacks.
- Beware: retro tests on enrichment vendors can leak. Vendors may see later data about those identities. Prefer data as of the transaction date, and ask how they guarantee it.
- Shadow mode. Call the vendor live and log outputs without acting. This measures real latency, timeout rate, coverage and score distribution on current traffic.
- Live controlled test. Randomize traffic between incumbent and challenger, or add the signal to a model challenger. Measure loss, approval, review rate and friction. See Experimentation and Champion-Challenger.
- Incremental lift. Measure what the vendor adds on top of what you already have. A strong standalone vendor can add little if it overlaps your existing signals.
- Economic read. Net value = (loss avoided + good-customer revenue recovered + ops savings) − (vendor fees + integration + run costs + friction cost).
Contract terms that matter
- Data rights: what the vendor may do with your data (consortium contribution, model training, resale), retention, deletion on exit
- Privacy and security: subprocessors, data location, breach notification timelines. Reg S-P service provider oversight for broker-dealers; GLBA safeguards for financial institutions (see Privacy and Data Use in Fraud Prevention)
- SLA: uptime, p99 latency, support response, credits that matter
- Change notification: advance notice of model or scoring changes; versioned outputs
- Explainability and documentation: what's needed for model validation (see Model Risk Management)
- Audit and exam access: for regulated entities, the right to provide information to examiners
- Pricing: tiers, minimums, overage, price escalators; per-call vs per-outcome
- Exit: export of your data and decision history; transition assistance; no penalty for reducing scope
- Fairness and bias: evidence of testing for disparate outcomes where decisions affect access (e.g., IDV pass rates)
- Liability and indemnity for vendor errors, including guarantees in chargeback-guarantee products
Regulated banks should align with interagency third-party risk management guidance (planning, due diligence, contracting, ongoing monitoring, termination). Model components fall under model risk guidance, which says vendor models are still subject to validation even when the vendor withholds code or data.
How to build it
- Owner: a fraud strategy or platform lead runs the evaluation; data science owns measurement; engineering owns integration and latency; procurement and legal own contract; privacy/compliance sign off. In regulated firms, second line reviews third-party and model risk (see Three Lines of Defense).
- Cadence: re-evaluate major vendors every 12–24 months against challengers. Keep a lightweight vendor scorecard quarterly.
- Portfolio view: avoid three vendors doing the same thing and nobody covering payout risk. Map vendors to the money path and the Mechanism to Control Matrix.
Artifacts and templates
Vendor scorecard (weighted, illustrative)
| Criterion | Weight | Score 1–5 | Notes |
|---|---|---|---|
| Incremental lift on your traffic (live test) | 30% | ||
| Latency and reliability (shadow) | 15% | ||
| Coverage of your segments and rails | 10% | ||
| Explainability and governance support | 10% | ||
| Data rights, privacy, security | 10% | ||
| Total cost of ownership | 15% | ||
| Integration effort and exit ease | 10% |
POC plan outline
- Problem statement and money path in scope
- Hypothesis and success/kill criteria
- Data sample definition and label maturity rules
- Retro test protocol (blind)
- Shadow configuration and duration
- Live test design, guardrails and exposure caps
- Economic model
- Decision meeting date and approvers
Metrics
- Incremental lift per vendor: loss bps, recall at fixed FPR, approval rate change
- Cost per decision and cost per $ loss prevented
- p99 latency and timeout rate, from shadow and production
- Coverage (share of traffic with a usable vendor response)
- Vendor-caused incidents per year
Pitfalls
- Testing on the vendor's sample. Use your traffic and your labels.
- Unblinded retro tests. Labels leak through the vendor's network or enrichment.
- Standalone metrics. Lift must be incremental to your existing stack.
- Signing before shadow mode. Latency and coverage surprises appear only on live traffic.
- No exit plan. Decision history locked in a vendor UI is a governance and litigation risk.
- Letting the vendor define fraud. Their labels and yours differ; reconcile before trusting their metrics.
Related
- Vendor Landscape · Open Source Tools for Fraud Teams · Consortium and Data Sharing
- Fraud Data Architecture · Real-Time Decisioning Architecture · Real-Time Transaction Risk Scoring
- Experimentation and Champion-Challenger · Fraud Metrics and KPIs · Model Risk Management
- Privacy and Data Use in Fraud Prevention · Broker-Dealer Obligations (FINRA and SEC)
Sources and further reading
- Federal Reserve SR 26-2 attachment, Revised Guidance on Model Risk Management (Apr 2026): vendor models remain subject to validation, ongoing monitoring and outcomes analysis
- Federal Reserve SR 26-2 (Apr 2026): supersedes SR 11-7; risk-based, proportional model risk management
- FINRA 2026 Oversight Report: Third-Party Risk Landscape: vendor due diligence, monitoring, data return/destruction and fourth-party risk
- FINRA 2026 Oversight Report: Cybersecurity and Cyber-Enabled Fraud: Reg S-P amendments, including service provider oversight
- Nacha, Credit-Push Fraud Monitoring Resource Center: rules don't prescribe technologies; reference list of monitoring vendors (not endorsements)