Entity Resolution

Core components

Identity vs relationship

The most important design decision is to keep two questions apart:

• Entity resolution: are these two records the same thing? (The same person, the same device, the same bank account.)
• Linkage: are these two entities related? (Two people share a device, two accounts pay out to the same bank account.)

ER produces nodes, and linkage produces edges. If you collapse relationships into identity ("they share a device, so they're the same customer"), you can't tell a household from a controller, and every downstream feature inherits the error. Store both, with types and confidence, and let Graph Analytics and Entity Link Analysis reason about the edges.

What you're resolving

Deterministic matching

Exact matches on normalized keys: same verified bank account, same card fingerprint, same device identifier, same government ID number.

• Strengths: fast, explainable, and high precision with strong, verified keys. It runs comfortably in real time.
• Weaknesses: brittle. Typos, formatting differences and partial data break it, and adversaries vary attributes on purpose. A deterministic-only system under-merges exactly the records you most need merged.
• Normalization is most of the work: casing, whitespace, punctuation, address standardization, phone formatting to a single standard, removing known placeholder values. Keep the raw value too, for investigations.

Probabilistic matching

The Fellegi-Sunter model (1969) is still the workhorse. For each compared field:

• m = probability the field agrees when the records are a true match (less than 1, because data has errors and people change phones)
• u = probability the field agrees by chance when they don't match (roughly how common the value is)
• Agreement weight = log₂(m/u); disagreement weight = log₂((1 − m)/(1 − u))
• Sum the weights across fields. Above an upper threshold, link. Below a lower threshold, don't. In between goes to clerical review.

Parameters can be estimated without labels using expectation-maximization, which matters because labeled match/non-match pairs are scarce. Open-source implementations make this practical at scale. The UK Ministry of Justice's Splink, for example, reports linking a million records on a laptop in about a minute and use on more than 100 million records (see Open Source Tools for Fraud Teams).

Supervised and embedding approaches (a classifier on pair features, learned representations for names and addresses) can beat Fellegi-Sunter when you have good labels, but they're harder to explain. For fraud, explainability matters: an analyst has to be able to say why two accounts were treated as one controller.

Blocking

Comparing every record with every other grows quadratically: 10 million records means about 50 trillion pairs. Blocking compares only records that share a blocking key: same normalized last name and birth year, same postcode, same bank account, same device.

• Use several blocking passes (an OR of rules) so a true match that differs on one key is still caught by another. Each pass trades speed for recall.
• Watch block sizes. Common values create enormous blocks: a carrier's shared IP pool, a default "N/A" phone, a test device, a freight forwarder's address. Cap block sizes and keep an exclusion list of known shared or junk values.
• Real-time ER is blocking by index. At sign-up, bank link or payout-destination change, look up candidates by strong keys, score them, and attach the record to an existing entity or create a new one, within your latency budget (Real-Time Decisioning Architecture).

Thresholds by use case

One threshold doesn't fit every consumer of ER. Set them by the cost of being wrong:

Clustering and the chaining problem

Pairwise matches become entities by clustering. The naive method, connected components, is transitive: if A matches B and B matches C, then A, B and C are one entity even if A and C share nothing. A few weak links can chain thousands of records into one giant component.

Mitigations:

• Don't chain on weak evidence. Shared IPs and behavioral similarity are relationship edges, never identity merges.
• Require cluster coherence, for example a minimum average pairwise score inside the cluster, or a rule that every member matches at least two others.
• Cap component size for identity entities, and route oversized components to review.
• Break weak bridges. A single edge connecting two dense groups is often a shared-infrastructure artifact.
• Monitor the size distribution. A sudden jump in the largest cluster usually means a new shared value (a vendor's proxy, a new carrier pool) slipped past the exclusion list.

Shared infrastructure that causes over-merging

Over-merging also has a fairness cost. Shared addresses and devices concentrate in multi-family housing, lower-income areas and immigrant communities, so false rings can produce discriminatory outcomes. Review cluster-driven actions for disparate impact (Privacy and Data Use in Fraud Prevention).

Under-merging from varied attributes

Controllers deliberately vary names, emails, devices and addresses so records don't match. The answer is to rely on evidence that's expensive to vary, while keeping identity thresholds where they are:

• Value endpoints. Funding sources, payout destinations, cards and crypto addresses. Money has to land somewhere (Payout Destination Intelligence).
• Artifacts. Reused documents, receipts and images.
• Behavior and timing. Similar navigation, creation windows and transaction patterns, as supporting evidence on edges (Behavioral Biometrics, Session Behavior Anomalies).
• Graph structure. Many accounts converging on few exits is a farm signature even when every account looks distinct (Graph Analytics).

FINRA's 2022 notice on small-cap IPO fraud lists exactly this kind of central-control evidence across nominee brokerage accounts: similar opening dates, shared referral sources or contacts, the same bank information or IP addresses, and near-identical funding amounts.

Temporal correctness and provenance

• Entity IDs change. Merges and splits happen as evidence arrives. Keep the history of which record belonged to which entity, and when.
• Point-in-time features. Training features must use entity membership as of the event time. Otherwise a merge discovered later leaks the label into the feature.
• Edge provenance. Record which fields, rule or model version, and timestamp produced each match or link.
• Edge expiry. IP and session edges should age out in hours or days. Bank account and identity edges persist.
• Splits are features too. An entity that keeps getting split by clerical review is telling you a blocking key or exclusion list is wrong.

Industry lens

How to build it

Ownership typically sits with fraud data engineering and data science, with investigations as the main source of labels (confirmed same-controller and confirmed household decisions) and privacy and legal reviewing what may be matched and retained.

Artifacts and templates

ER specification (one per entity type)

Monthly ER health review

• Precision and recall on the labeled pair set, by entity type
• Ten largest clusters reviewed by a human
• New high-frequency values added to the exclusion list
• Clerical review volume and agreement rate
• Merge and split counts, with reasons
• Cluster-driven actions reviewed for disparate impact
• Downstream lift: entity-level vs account-level rule precision

Metrics

Pitfalls

• Merging records on a relationship, such as a shared device or IP, as if it proved identity.
• Transitive closure on weak links, which produces giant components and false rings.
• Using one threshold for every use. Deny lists need far more precision than investigation leads.
• Ignoring pooled and shared infrastructure. One fintech's pooled account or one carrier's IP pool can link half your customer base.
• Keeping no temporal history. Features leak future merges into training data, and decisions can't be reconstructed.
• Having no labels. Without investigator-confirmed pairs you can't measure precision, so you can't tune.
• Relying on deterministic-only matching, which is precise on clean data and blind to the adversaries who vary attributes on purpose.
• Acting on clusters without review. Bulk enforcement on a cluster without a human checking the largest and weakest links is how false rings become customer harm (Playbook - Fraud Ring Investigation Members).

Related

• Data and infrastructure: Fraud Data Architecture · Real-Time Decisioning Architecture · Graph Analytics · Machine Learning Models · Open Source Tools for Fraud Teams
• Signals: Entity Link Analysis · Device Reputation and Linkage · Identity Element Consistency · Referral Graph Patterns · Payout Destination Intelligence
• Controls: Negative Lists and Deny Lists · Velocity and Exposure Limits · Consortium and Data Sharing
• Mechanisms: Multi-Accounting · Account Farming and Resale · Fraud Rings and Collusion · Money Mules · Synthetic Identity Fraud
• Playbooks: Playbook - Multi-Accounting and Linkage Members · Playbook - Fraud Ring Investigation Members · Playbook - Money Mule Detection Members
• Governance: Privacy and Data Use in Fraud Prevention · Model Risk Management · Exposure Window Modeling

Sources and further reading

• Fellegi and Sunter, A Theory for Record Linkage (JASA, 1969): the original probabilistic linkage model with link, non-link and possible-link regions
• Wikipedia, Record linkage: overview of deterministic vs probabilistic linkage, m and u probabilities, agreement weights, blocking trade-offs, standardization
• Linacre, Deduplicating and linking large datasets using Splink (Real World Data Science, 2023): Fellegi-Sunter in open source, unsupervised EM estimation, blocking as the main speed driver, term-frequency adjustment, clustering at a threshold, scale figures
• FINRA Regulatory Notice 22-25, Heightened Threat of Fraud (2022): red flags of central control across nominee brokerage accounts