The Outlier Lab

Generative AI and Deepfake Fraud

Voice clones, deepfake video, AI-generated documents and selfies, and AI-scaled lures. GenAI lowers the cost and raises the credibility of identity and social-engineering attacks that end in transaction losses.

What it is

GenAI makes existing fraud types cheaper to run and more convincing. For a transaction-fraud team, the practical question is which controls depended on something being hard to fake.

Control assumptionWhat GenAI changes
"A photo ID plus a selfie proves a real, present person"AI-generated or altered documents and deepfake selfies; FinCEN's 2024 alert and FINRA's 2026 report both describe them defeating verification checks
"A voice on the phone that matches the customer is the customer"Voice clones that persuade firms to change account access (FINRA 2026), and "distress" calls to relatives (IC3 2025)
"Seeing a known executive on video means the instruction is real"Multi-person deepfake video calls (Case Study - Arup Deepfake Video Call (2024))
"A receipt or damage photo is evidence"Generated or edited receipts and claim images (Receipt Fraud (Cashback), Buyer Protection Claim Abuse (INR and SNAD))
"Scam messages are clumsy"Fluent, personalized, multilingual lures at scale; AI-written chat for long-con scams
"Celebrity endorsements are rare and checkable"Deepfaked public figures promoting investment schemes

Measured scale. The FBI's IC3 2025 report added its first dedicated AI section: 22,364 complaints with an AI nexus and about $893M in losses. That figure is a floor, because victims often can't tell that AI was involved. Its breakdown:

  • Investment: AI-generated celebrity and executive videos and voices, plus AI-driven chats. Over $632M.
  • Business email compromise: AI-written executive emails and voice-cloned payment requests. Over $30M (corporate; out of scope here except as a pattern).
  • Confidence/romance: AI-written profiles and scripts, plus voice-cloned "distress" and grandparent calls. Over $19M, of which over $5M is distress scams.
  • Employment: voice-spoofed or deepfaked job interviews. About $13M, with network access apparently the main aim.

How it works

Diagram: Collect source material (public video, voice, photos, breached PII), Generate: voice, face, document, receipt, script, Onboarding: synthetic or impersonated account, Support or recovery: voice or video impersonation, Victim persuasion: executive, relative, celebrity, Evidence: receipts, claim photos, statements, Mule or bust-out accounts receive funds, ATO: contact change, then payout, Authorized payment to scammer, Cashback, refund or claim paid

Transaction footprint

Entry pointWhat you see in the dataDecision point
OnboardingClean-looking documents with subtle inconsistencies; face images that reverse-match AI-generated galleries; "technical glitches" or a channel switch during liveness; device or geo inconsistent with the documentDocument Verification and Liveness; Identity Verification Outcomes; Device Fingerprinting, Emulator and Rooted Device Detection
Early account lifeRapid, high-volume transactions in new or low-history accounts, especially to higher-risk payees (FinCEN red flag)Tenure-aware Funds Availability Holds; mule detection (Money Mules)
Support and recoveryVoice-channel requests for contact or MFA changes, especially from a caller who refuses callback or pushes urgency; followed by a new-device login and payoutCustomer Support Verification Protocols: never let voice alone authorize a change
Authorized paymentCustomer insists the instruction came from a known person on video or voice; urgent, confidential, first-time payeePayee Confirmation and Payment Friction; out-of-band verification scripts
Evidence-based claimsReceipts or photos with no capture-device metadata, mismatched fonts and totals, or near-duplicate variants across accountsReceipt and Purchase Verification, Receipt and Document Image Forensics
Victim behaviorOlder customers acting on a relative's "emergency" call; investors citing a celebrity or "analyst" videoScam Victim Indicators; Customer Education and Scam Warnings

Common variants

VariantWhat's differentWhere it shows up
Deepfake KYC (document + selfie)Synthetic or altered ID with matching generated faceFintech and brokerage onboarding; mule and synthetic accounts
Camera-feed manipulation during livenessVerification sees a manipulated feed instead of a live camera (described conceptually by FinCEN as third-party webcam plugins and "glitches")Remote onboarding
Voice-clone account recoverySupport agent persuaded to change accessBanks, brokerages, rewards apps with phone support
Voice-clone family emergencyA relative's voice requesting urgent moneyOlder adults; Elder Financial Exploitation
Deepfake executive videoMulti-party call instructing transfersCorporate treasury (out of scope but instructive)
Celebrity and analyst deepfakesFake endorsements driving investment-club and crypto scamsInvestment Scams (Pig Butchering)
AI receipts and claim photosGenerated or edited purchase evidenceCashback, marketplaces, insurance-like claims
AI-scaled phishing and chatPersonalized, fluent lures; chatbots sustaining long consPhishing and Credential Harvesting, romance and investment scams

Where it shows up

Impact

  • Direct: IC3 2025 attributes about $893M to AI-nexus complaints. Investment scams dominate.
  • Control erosion: onboarding and recovery controls that relied on media evidence lose precision, which forces either more friction for everyone or investment in layered signals.
  • Detection pace: detection vendors and fraudsters iterate quickly. The IC3 report warns that AI-generated content is getting harder to detect and easier to produce.
  • Regulatory: FinCEN asks institutions to use SAR key term FIN-2024-DEEPFAKEFRAUD and to report the underlying typology too. See BSA-AML and SAR Filing.

Red flags

Prevention and detection at a glance

LayerControl or signalWhat it does here
PreventLayered Document Verification and Liveness plus device integrity and data-consistency checks within KYC and Customer Identification Program (CIP)No single media check should be decisive; non-documentary verification counters a perfect fake document
PreventPasskeys and Phishing-Resistant Authentication and device-bound recoveryTakes the human-judged voice or face out of the critical path
PreventCustomer Support Verification Protocols: callback to the number on file; in-app confirmation for changesDefeats voice-only impersonation
PreventOut-of-band verification norms for high-value instructions (pre-agreed phrases, known-channel callbacks)Independent verification through publicly listed contacts is consistent FBI guidance in impersonation PSAs
DetectMachine Learning Models on metadata, behavior and network as well as pixelsMore durable than artifact detection as generators improve
DetectReceipt and Purchase Verification against merchant or transaction dataConfirms the purchase from data instead of trusting the image
RespondRe-verification campaigns when a deepfake vector is found; SAR with the key termContains the cohort onboarded through the gap

Emerging variants and adversary adaptation

  • Shift to provenance. Artifact detection on its own loses ground as generators improve. Defenses shift toward provenance (capture-device attestation, verified data sources), behavior and network signals, and removing media from high-risk decisions altogether (passkeys, data-based purchase verification).
  • Real-time interactive deepfakes. Live voice and video conversion makes conversational impersonation possible, where earlier attacks relied on prerecorded clips. The FBI's September 2026 PSA notes AI-enabled video calls in government-impersonation scams.
  • Recovery and support. These channels are the soft spot. FINRA's 2026 report highlights voice clones persuading firms to change account access. Expect pressure to move toward support channels without strong customer-side confirmation.
  • First-party adoption. Consumer AI image tools make receipt and claim editing trivial for ordinary customers as well as organized rings. Expect volume growth in low-dollar, high-frequency claims abuse.
  • Synthetic workforce. Deepfaked or proxy candidates in remote hiring aim to become insiders. See Insider Fraud.
  • Agentic scale. AI agents can run scam conversations and account operations autonomously. See Agentic Commerce Fraud and Fraud-as-a-Service Ecosystem.

Real-world examples

  • Case Study - Arup Deepfake Video Call (2024): an employee in Arup's Hong Kong finance team received a message in January 2024 that appeared to come from the UK-based CFO. After a video conference with digitally recreated colleagues, the employee transferred HK$200M (about US$25.6M). Hong Kong police reported the case in February 2024, and Arup confirmed it was the victim in May 2024.
  • FBI IC3 PSA I-120324-PSA (December 3, 2024) catalogs GenAI use across text, images, cloned audio and real-time video. Its examples include cloned relatives' voices demanding emergency payments, AI audio used to get into bank accounts, deepfaked executives on live video calls, and AI-made IDs. Its consumer advice includes a family "secret word" for verification.
  • FinCEN (November 2024) reported a rise, starting in 2023 and continuing into 2024, in suspicious-activity reports describing deepfake media used in fraud schemes against financial institutions. Among best practices it recommended phishing-resistant MFA.
  • Case Study - AI-Generated Fake Receipts (2025-2026): image models made fabricated receipts common in under 18 months, from 0% to about 71% of one platform's flagged fakes
  • 2026-09-17 FBI updates its warning on government and law enforcement impersonation scams: FBI says impersonators now use AI, including on video calls, to pose as officials

Sources and further reading