Skip to main content

Matching and merging

Your source systems each hold their own version of the same customer, supplier, or patient. Matching decides which of those records describe the same real-world thing. Merging consolidates them into one golden record, while keeping every contributing record intact and the whole operation reversible.

This is the part of the platform most worth understanding in depth, because it is where correctness is decided. A matching configuration that is too eager merges two different people; one that is too cautious leaves your data fragmented and every downstream answer wrong.

The problem it solves​

The match profile overview screen A match profile. Each stage described below is a tab on this screen.

Real data does not agree with itself. The same person arrives as:

Sourcefirst_namelast_namephoneaddress.line1
CRMJonPemberton+1555300100112 Old Mill Road
ERPJonathanPemberton-Hayes+1555300100112 Old Mill Rd
KYCJonathanPemberton-Hayes(none)12 Old Mill Road
Mobile appJohnnyPEMBERTON+15553007777(none)
Loan originationJonathanPemberton Hayes(none)440 Asylum Avenue
Credit bureauJPemberton-Hayes+15553001001(none)

Six records, one person, and no field agrees across all six except the date of birth, which every system records as 1979-08-22. Each spelling is correct in its own system: the CRM holds what he calls himself, KYC holds what his passport says, the bureau abbreviates, and loan origination lost the hyphen to a form that would not accept one.

Note what is not here. There is no government identifier anywhere in the set — no shortcut that settles it instantly. That is the normal case, and it is why matching has to work on accumulated evidence.

What does settle it is the weight of the evidence, and how surprising each agreement is. Two people sharing a date of birth is unremarkable. Three separate systems holding the same phone number, +15553001001, is not.

This record set runs through the whole documentation — see the worked example.

How the platform decides​

Four stages run for every candidate pair.

1. Blocking — find pairs worth comparing​

Comparing every record against every other is quadratic and quickly impossible. Instead, each record produces a small set of blocking keys, and only records sharing a key are compared.

Blocking governs recall and cost, never the decision itself. A pair the blocking stage never proposes can never match, so blocking strategies are designed to be generous.

2. Comparison — grade each attribute's agreement​

Each attribute has an ordered ladder of comparison levels — exact, then phonetic, then prefix, and finally a catch-all mismatch. The engine walks the ladder and takes the first level that fits.

Grading is separate from weighting: this stage decides how well two values agree, not what that agreement is worth.

3. Scoring — weigh the evidence​

Each resolved level carries a match weight, and the weights sum to a composite weight for the pair.

Weight comes from two probabilities per level: how often that level of agreement occurs among true matches, and how often it occurs by pure coincidence. An agreement that is common among matches and rare by chance carries a lot of weight; one that happens constantly by chance carries almost none.

The engine also adjusts for how common a specific value is. Agreeing on a very common surname tells you far less than agreeing on a rare one, and the score reflects that automatically.

Weights are not percentages

Composite weights and thresholds are on a log2 evidence scale, not a 0–1 score and not a percentage. Most platforms use a bounded score, so this is the most common source of mis-tuned configuration. A composite weight of 12 does not mean "12%" — it means the evidence is roughly four thousand times more consistent with a match than with a coincidence.

How probabilistic scoring works explains the scale in full.

4. Tiering — decide, or ask a person​

The composite weight is compared against two thresholds:

Composite weightTierWhat happens
At or above the auto-link thresholdauto-linkMerged automatically
Between the two thresholdsclerical reviewQueued for a data steward
Below the clerical review thresholdno matchNothing happens

The middle band is the point of the design. Rather than forcing every pair into a yes or no, the engine routes genuinely ambiguous pairs to a person, and routes only the confident ones automatically.

Two safeguards apply on top:

  • A deterministic level on a strong identifier can settle a match on its own, without accumulating other evidence.
  • A minimum-agreeing-attributes rule can require agreement on several distinct attributes before anything auto-links, so a single coincidental agreement cannot merge two records however heavily it is weighted.

What merging does​

When a pair auto-links, or a steward confirms one:

  • Survivorship rules choose the winning value for each attribute, producing the golden record.
  • Every contributing record keeps its crosswalk, so you can always trace a value to the system it came from.
  • Relationships, comments, interactions, consent records, and privacy requests are re-pointed onto the survivor.
  • The whole operation is recorded so it can be undone.

Merges are reversible. Unmerging restores the absorbed records and the state they carried.

When this does not apply​

  • A single trustworthy identifier settles it. If every source carries a reliable national identifier, deterministic matching alone is simpler and faster. Probabilistic scoring earns its keep when identifiers are missing, wrong, or not shared across systems.
  • The records are not the same kind of thing. Matching is scoped to one entity type at a time.
  • You want to link, not consolidate. Two entities that are genuinely different but connected are a relationship, not a merge.

Next​

Last verified against commit 315eb047 (2026-08-03)