How probabilistic scoring works
This page explains the model the matching engine implements. You do not need it to configure matching day to day — the task pages cover that. You need it when you are tuning thresholds, explaining a decision to an auditor, or satisfying yourself that the engine is principled rather than a bag of heuristics.
The engine implements the Fellegi-Sunter model of probabilistic record linkage, the standard formulation in the record-linkage literature.
The two probabilities
Every comparison level carries two numbers.
| Symbol | Question it answers | Plain language |
|---|---|---|
| m | Among pairs that genuinely are the same entity, how often does this level of agreement occur? | How reliably this field agrees when two records really are the same |
| u | Among pairs that genuinely are not, how often does it occur anyway? | How often this field agrees by pure coincidence |
An attribute is informative when those two numbers are far apart. Date of birth
agreeing exactly has a high m — matching records usually share it — and a low
u, because two unrelated people rarely share a birthday. Gender agreeing has a
high m too, but a u near one half, so it barely moves the needle.
The weight
Each level's weight is the log2 ratio of the two:
weight = log2(m / u)
Positive when agreement is evidence for a match, negative when it is evidence against. The composite weight for a pair is the sum across every attribute that contributed:
composite_weight = Σ weight_i
Because these are logarithms, summing them multiplies the underlying likelihood ratios. That is the whole point: evidence accumulates.
Reading the scale
Each whole point doubles the strength of the evidence.
| Composite weight | The evidence is… |
|---|---|
| 0 | Equally consistent with a match and a coincidence |
| 4 | ~16× more consistent with a match |
| 8 | ~256× more consistent |
| 12 | ~4,000× more consistent |
| 16 | ~65,000× more consistent |
Thresholds are set on this log2 evidence scale, not on a 0–1 score. Most
other platforms use a bounded score, so the instinct to read a threshold of
12 as "12%" or "12 out of 100" is strong — and wrong. A threshold of 12 asks
for evidence roughly four thousand times more consistent with a match than
with a coincidence.
The posterior
The engine also reports a posterior probability, which folds in how common matches are in your data at all:
posterior = σ( log2(λ / (1 − λ)) + composite_weight )
where λ is the prior probability that a random candidate pair is a match, and
σ is the logistic function.
Thresholds are compared against the composite weight, not the posterior. This is deliberate. The composite weight is pure evidence; the posterior mixes in an assumption about your data's base rate. Tying thresholds to the posterior would mean re-tuning every time your data volume changed, since the prior shifts with it. Keeping thresholds in evidence units makes them stable.
The posterior is reported because it is the number to quote to a person: "we are 99.4% confident" is meaningful in a review queue in a way that "composite weight 14.2" is not.
Term-frequency adjustment
A fixed u per level assumes every value is equally likely, which is false.
Agreeing on a very common surname is weak evidence; agreeing on a rare one is
strong.
The engine adjusts using observed frequencies in your own data:
u_adjusted = max(u_baseline, observed_frequency)
A rare value keeps its small u and therefore its large weight. A common value
gets an inflated u, and its weight is suppressed toward zero. This happens
per value, automatically, from a frequency snapshot the platform maintains.
The adjustment is deliberately conservative — it only ever raises u, so it
can only ever weaken evidence, never manufacture it.
Anonymous values
Some values carry no identifying information at all: a placeholder name, a default date, a filler phone number repeated across thousands of records. These are suppressed at the blocking stage rather than scored, because they would otherwise generate enormous candidate sets of records that share nothing meaningful.
A value is suppressed only when it is both frequent in absolute terms and a meaningful share of the data. Requiring both matters: on a small dataset a relative-only rule would suppress ordinary values that happen to appear twice.
Safeguards on auto-linking
Two rules constrain what may merge without a person:
The auto-link threshold is set by precision, not by a balanced score. A combined measure like F1 treats auto-linked and clerically-reviewed pairs alike, so it is structurally blind to where the auto-link line sits — moving that line does not change it. The auto-link threshold is therefore set from the labelled data as the point above which no known non-match falls.
A minimum number of agreeing attributes can be required. Distinct attributes must agree before anything auto-links, unless a deterministic identifier settled it. This is threshold-independent: it holds even if a threshold is mis-set, and it is the defence against a single coincidental agreement merging two records.
What the engine does not do
- It does not learn on its own. Weights come from configuration or from calibration against labelled pairs you supply.
- It does not use the posterior for tiering.
- It does not treat a high weight on one attribute as sufficient, when a minimum-agreement rule is configured.
Next
- Tune thresholds — choosing the two numbers in practice
- Match profile reference — every configuration option
- Review potential matches — what stewards see
Last verified against commit 315eb047 (2026-08-03)