--- title: "Blocking and candidate generation" summary: "You cannot compare every pair. How blocking keys work, why rarity weighting matters, how to measure retrieval with recall@k, and the silent failure mode that mints duplicates." canonical_url: https://entitybind.com/guides/blocking-and-candidate-generation last_updated: 2026-10-12 copy: full --- # Blocking and candidate generation > You cannot compare every pair. How blocking keys work, why rarity weighting matters, how to measure retrieval with recall@k, and the silent failure mode that mints duplicates. Canonical page: https://entitybind.com/guides/blocking-and-candidate-generation Last updated: 2026-10-12 For the team that owns a company master and has to defend it. Comparing every record against every other record is quadratic and therefore impossible at any real scale, so something has to narrow the comparison set to a handful of plausible candidates before any scoring happens. That stage is blocking, also called indexing or candidate generation. It is the least discussed part of an entity resolution system and it sets a hard ceiling on everything after it: a pair that blocking never retrieves cannot be matched, however good the scorer is. Recall lost here is not recoverable downstream. - Reading time: 9 min read - Sections: 8 - Questions answered: 5 - Key takeaways: 6 The same measurement, on your rows: A first file is a few hundred of your own rows with the matches you already have beside them. The numbers in this article come back measured on your data, and we write nothing to your systems. [Start free →](https://app.entitybind.com/app?signup=1) Self-service. Create your workspace online and verify your email to start matching company records. Free starts with 300 credits and no card. Paid monthly plans are available in workspace Billing; VAT is added where applicable. Usage is tracked without automatic overage charges. [Trust and security](https://entitybind.com/trust) A stewardship queue, a CRM after a migration or a merger, a supplier file before a payment run, counterparty onboarding that has to survive an audit. Read the argument here, take the takeaways at the foot into your next vendor conversation, and send us a file when you want the same measurement on your own rows. How matching works · 1,990 words · every figure carries the population it was measured over By [James Rooney](https://entitybind.com/about), founder of Entity Bind · Last revised 12 October 2026 ## The quadratic problem, with the arithmetic Take one register. Companies House holds 5,464,811 live companies. Deduplicating that against itself means n(n−1)/2 pairs, which is roughly 1.49 × 10¹³ — about fifteen trillion comparisons. At an optimistic one microsecond per comparison, that is over 170 days of continuous CPU time, and a real feature extraction and score costs far more than a microsecond. The per-record case is more relevant to an API and no less stark. Resolving a single inbound record against that register by brute force means over five million full comparisons for one call. At a millisecond each — again optimistic once normalisation, feature extraction and scoring are counted — that is an hour and a half per record. So candidate generation is not an optimisation. It is the difference between a system that exists and one that does not, and every design decision in it is a trade between how many pairs you skip and how many true matches you skip with them. ## Blocking keys The classical technique is to compute one or more keys per record and only compare records that share a key. A key is any deterministic function of the record — the first three characters of the normalised name, the postcode district, a phonetic encoding — and records are compared only within a shared bucket. A single key is always wrong, because any single key has a blind spot: block on the name and you miss every renamed company, block on the postcode and you miss every relocation. Real systems generate several keys per record and take the union of the buckets, so a record can be retrieved by any of its identities. The cost is that buckets overlap and the candidate set grows. The key families worth having for company data, roughly in order of how decisive they are: - Scoped and global identifiers — an LEI, a DUNS number, a registration number qualified by its register, a ticker with its exchange. These are decisive and should get their own retrieval pass rather than competing with weak keys. - Exact normalised name, and name qualified by country. The workhorse, and the one that suffers most from normalisation defects. - Digital identifiers — registrable domain, normalised phone. Powerful and dangerous in equal measure, because a group's shared corporate domain produces enormous buckets. - Phonetic encodings — Double Metaphone and similar — which recover transcription variants at the cost of very large buckets on common name stems. - Bounded character trigrams, which recover typos and truncations that exact keys miss and which need bounding precisely because they are the most generous family. ## Rarity weighting, and the hot bucket problem Retrieval has a cap: some maximum number of candidates that will be scored. Under a cap, buckets compete, and without a rarity term the wrong ones win. Consider a record carrying both an exact, unusual company name and a shared group domain. The exact-name bucket might hold one entity. The domain bucket might hold four hundred, because the whole group uses one website. If candidates are drawn without regard to bucket size, the hot domain bucket fills the cap and evicts the single correct candidate — and the system then concludes it has never seen the company and mints a duplicate. The fix is to rank keys by how rare each one is, counted on the data you actually hold rather than assumed. A key that matches one entity is worth far more than a key that matches four hundred — the same intuition as [the u probability in Fellegi-Sunter](https://doi.org/10.1080/01621459.1969.10501049), which is how often a field agrees between two records that are not the same company. Adding this rarity weighting, computed per tenant, to our own retrieval recovered targets that a hot corporate-domain bucket had been evicting, and it did so without touching the scorer. That is the general lesson: retrieval failures look like matching failures and are not. A related trap is the cap's own invisibility. When the limit is applied in the database, the rows past it never reach the application, so a bucket of fifty and a bucket of five thousand look identical from the outside. That matters because a retrieval that was capped and then rejected everything it saw is not the same event as a retrieval that genuinely found nothing, and scoring them the same way mis-attributes a recall failure as a judged rejection. Our own answer is to stamp the truncation on the decision so it can be counted wherever it matters, and to refuse to invent a confidence number for a population we have no labels for. ## Measuring blocking on its own Retrieval must be measured separately from matching, because pooling them hides which half is failing and the two have different fixes. Four metrics are worth having. - Recall@k — the share of records whose true entity appears anywhere in the top k candidates. This is the ceiling on everything downstream. - Mean reciprocal rank — how high in the list the true entity sits, which is a ranking diagnostic rather than a recall one and matters most when a cap is tight. - Reduction ratio — the share of all possible pairs that blocking removed. On its own it is meaningless, since blocking on nothing scores perfectly; beside recall@k it describes the trade you actually made. - Bucket size distribution — because the tail is what breaks. A mean bucket size of twelve with a maximum of forty thousand is a system that works until it does not. ## The denominator of recall@k, and what it excludes Recall@k has a denominator that is easy to misread, and misreading it always favours the system. It is normally computed over records whose true entity already exists in the catalogue, because a target that is not there cannot be retrieved. That is legitimate — it is the question retrieval is responsible for — but it means recall@k says nothing at all about first-arrival records, which are exactly the population where a retrieval failure and a correct mint look identical. Recall@k is also a property of a list, not of an answer. Our own name-only shortlist is a case in point: on 205 name-only records on the gated acquisition fixture. the true company is at rank one for 74% (151/205) and inside the top three for 100% (205/205), measured on the gated acquisition fixture. Recall is a property of the list, not of the winner. 74% is far below the 0.90 acceptance floor, so no row of that shortlist may be offered as a match, quoted as an accuracy number, or folded into an accepted-match count. The general rule: a high recall@k with a low resolution rate is a healthy state, because retrieval is doing its job and the matcher is declining on evidence that does not discriminate. A low recall@k with a high resolution rate is the alarming combination, because the system is confidently choosing between candidates that do not include the right answer. ## Context should rank candidates, not filter them Callers often have extra context — a trading address, a description, an email footer — and the tempting design is to use it as a hard constraint: only retrieve companies in the named town. That design is worse than it looks, because caller context is frequently stale or wrong, and a hard constraint deletes the true candidate outright with no way for a later stage to recover it. Ranking is the safer construction. A candidate that agrees with the context is promoted; candidates that disagree stay in the list at a lower rank. When the context is truthful, the true entity is ranked first almost always. When it is wrong, the true entity is demoted rather than removed, and nothing binds to the company that really is in the named town. We measure that rather than assert it. In the historical 50-record wrong-town fixture, 72% resolved and zero wrong bindings were observed. This is a measured adversarial sample, not a guarantee for every misleading description. What ranking buys when the context is truthful shows up in the tier table: the name-plus-country-and-town-prose input shape historically resolved 99.37% (315/317, 95% CI [97.7, 99.8]) on the live-adjudicated fixture, against 98.42% (312/317) for a name with nothing but a country. Historical 2026-08-29 recording with earlier identifier arrivals seeding the catalogue. This does not measure a business description, email footer or generic verification purpose alone. Two same-stem groups remained unresolved. ## The silent failure: keys are a derived index This is the operational hazard nobody warns you about, and it has bitten us. Blocking keys are usually written once, when an entity is saved, and stored as an index. Change how a key is spelled in the code and every entity written before that deploy remains indexed under the old spelling. Nothing errors. No test fails. Candidate retrieval simply returns an empty intersection, the cascade concludes it has never seen the company, and it mints a duplicate of an entity already in the catalogue. We have had exactly this: records blocking on a bare name stopped reaching entities indexed under name-plus-country, and it cost measurable resolution accuracy while the entire test suite stayed green. The operational rule that follows is that any change to key generation is a two-part deploy — ship the code, then rebuild the index — with a dry run first that reports how many entities would change and how many keys would be added or removed. A migration cannot fix this, because the index is derived rather than owned by the schema. The same logic applies to time. If your facts carry validity intervals — an address that was correct until 2023 — the blocking index needs those intervals too, or a record dated to an old period retrieves nothing and mints a duplicate of a company you already hold under its current address. ## Practical defaults Four choices carry most of the outcome, and none of them is exotic. - Give identifiers their own retrieval pass. A scoped registration number or an LEI should not have to compete with a phonetic key for slots under a cap. - Generate several keys per record and take the union. One key always has a blind spot, and which blind spot is determined by which key you chose rather than by the data. - Weight by rarity computed on the data you hold, not on an assumed distribution. Hot buckets are where recall goes, and they are specific to a tenant's file. - Measure recall@k before touching the scorer. Most reported matching failures are retrieval failures wearing a matching failure's clothes, and the fixes are unrelated. ## What to take away - Brute force is not an option: 5,464,811 live companies in one register is roughly 1.49 × 10¹³ pairs, and over five million comparisons for a single inbound record. - Blocking sets a hard recall ceiling. A pair never retrieved cannot be matched, so retrieval must be measured separately from matching. - Without rarity weighting, a hot bucket — a group's shared corporate domain — evicts the single correct candidate under a cap and the system mints a duplicate. - Recall@k is normally computed only over records whose true entity already exists, so it says nothing about first arrivals. It is a property of the list, not of an answer. - Caller context should rank candidates rather than filter them, so that wrong context costs coverage instead of correctness. - Blocking keys are a derived index. Changing key generation without rebuilding it fails silently and mints duplicates with every test still green. Those takeaways hold for anybody's data. The number for yours comes out of your own rows — a few hundred of them, with the matches you already have beside them. [Start free →](https://app.entitybind.com/app?signup=1) ## Read next Each article names its own further reading. These pick up a question this one raises and leaves open. - [What is entity resolution?](https://entitybind.com/guides/what-is-entity-resolution) (9 min read): Entity resolution decides which records refer to the same real-world thing. What it is, why company identity is hard, deterministic versus probabilistic matching, and why fuzzy matching is not entity resolution. - [Fellegi-Sunter explained, with worked arithmetic](https://entitybind.com/guides/fellegi-sunter-explained) (10 min read): The probabilistic record linkage model behind most matching engines: m and u probabilities, match weights, the three decision regions, and a worked example you can check with a calculator. - [How to measure entity resolution accuracy](https://entitybind.com/guides/how-to-measure-entity-resolution-accuracy) (13 min read): Precision, recall, abstention, pairwise versus B-cubed cluster metrics, Wilson intervals and out-of-fold evaluation. Why a single accuracy number for entity resolution is always meaningless, with worked arithmetic. Or try your own rows: create your Free workspace and upload a company CSV to inspect its results. Contact us for an assisted measurement against your labelled data, with the working handed back. [Start free →](https://app.entitybind.com/app?signup=1) ## Where this article is applied 3 pages name this method as the working behind what they claim. - [Entity Bind vs Build it yourself](https://entitybind.com/compare/building-it-yourself) (Comparison): The right answer if entity resolution is your product. Several teams have done it well. - [Entity Bind vs Splink](https://entitybind.com/compare/splink) (Comparison): Free, open source, and it weighs the evidence properly: Fellegi–Sunter, done right. Our scoring stage is the same family. - [Entity Bind vs Companies House](https://entitybind.com/compare/companies-house-api) (Comparison): Free, authoritative, and the right answer for a lot of UK teams. It is one of our sources too. ## Is this your file? The article is for anyone. The measurement behind it is worth commissioning for one kind of team. ### Built for Teams with a company master they have to defend — an MDM or stewardship queue, a CRM after a migration or a merger, a supplier file before a payment run, or counterparty onboarding that has to survive an audit. ### Wrong for Buying breadth as the product: we hold a fraction of an incumbent’s records and are not building a global company graph. Entity Bind resolves companies and organisations only. It does not resolve people, vessels, bank-account ownership, sanctions exposure, emissions or geolocation, and its results may not be used for eligibility decisions about individuals, including employment, credit and tenancy. ## Sources The papers, regulators and registers this article quotes, each with the caveat that travels with it. - [Fellegi and Sunter (1969), “A Theory for Record Linkage”, Journal of the American Statistical Association 64(328), 1183–1210](https://doi.org/10.1080/01621459.1969.10501049): The paper behind the three-region rule and the m and u probabilities. Its optimality result holds under the model's assumptions, conditional independence between fields among them, which real company data usually breaks. ## The basis for every figure above Internal fixtures only. The historical 2026-08-29 per-tier recording covers 317 GB companies with identifier-rich arrivals seeding the catalogue first; the context tier includes country and registered-town prose. Its legacy replay has no full-prompt-verified calls and cannot certify current calibration. The September 5 CI reports cover 1,176 labelled pairs and 705 records over 250 companies using offline stubs. These are not independent customer holdouts or live production validation; the 99% minimum-input product target remains unmet. Scope is company and organisation identity only — not people, vessels, bank-account ownership, sanctions or emissions. Results may not be used for eligibility decisions about individuals, including employment, credit and tenancy. Figures read off the product repository on 2026-09-05. The per-tier rates come from the live-adjudicated 317-company recording of 2026-08-29; the gate figures come from the stub-adjudicated continuous-integration run and are a floor rather than a ceiling. The two are never averaged. Entity Bind holds no SOC 2, no ISO 27001 and no other certification, and claims none. Every figure, with the population it was measured over and the method behind both recordings, is set out on [the accuracy page](https://entitybind.com/accuracy). The retention, erasure, tenancy and sub-processor detail a security review asks for is under [Trust and security](https://entitybind.com/trust), with where assurance stands: no external validation of the accuracy figures and no customer holdout. ## The same measurement, on your own rows. Create your workspace, verify your email and run your first company file. The Free plan needs no card. Create API keys in your workspace when you need an integration. [Start free](https://app.entitybind.com/app?signup=1) · [Plans and prices](https://entitybind.com/pricing) 300 credits. No card. For a guided review of your file, explore [the Merge Safety Sprint](https://entitybind.com/audit-pack). ## The questions this article answers ### What is blocking in entity resolution? Blocking, also called indexing or candidate generation, is the stage that narrows the set of records worth comparing before any scoring happens. Records are assigned one or more deterministic keys — an exact normalised name, a qualified registration number, a domain, a phonetic encoding, bounded trigrams — and only records sharing a key are compared. It exists because comparing every pair is quadratic: one register of five and a half million companies is about fifteen trillion pairs. ### Why does blocking limit accuracy? Because a pair that is never retrieved can never be matched. Blocking sets a hard ceiling on recall that no amount of scorer quality recovers, which is why retrieval should be measured on its own with recall@k before any effort goes into the matching model. In practice, a large share of reported matching failures turn out to be retrieval failures, and the two have entirely different fixes. ### What is recall@k and what is its denominator? Recall@k is the share of records whose true entity appears somewhere in the top k retrieved candidates. Its denominator is normally records whose true entity already exists in the catalogue, because a target that is not stored cannot be retrieved — which means recall@k says nothing about records arriving for the first time. It is also a property of the list rather than of any answer: on 205 name-only records on the gated acquisition fixture. our name-only shortlist has the right company at rank one for 74% (151/205) and in the top three for 100% (205/205), and neither figure may be quoted as an accuracy rate. ### Why do you weight blocking keys by rarity? Because retrieval has a cap and buckets compete for it. A record carrying both a rare exact name and a shared group domain will lose the rare name to the domain bucket if candidates are drawn without regard to bucket size — the group's four hundred entities fill the cap, the correct single candidate is evicted, and the system mints a duplicate. Rarity weighting, computed locally on the data actually held, is the same intuition as the u probability in Fellegi-Sunter. ### What happens if you change a blocking key? Every entity written before the change stays indexed under the old spelling, and the failure is silent: no error, no test failure, just an empty candidate intersection and a duplicate entity minted for a company already held. Any change to key generation is a two-part deploy — ship the code, then rebuild the index, with a dry run first. A database migration cannot fix it, because the index is derived rather than owned by the schema. --- The 0.90 confidence floor. It is an acceptance contract, not calibration evidence: it says what we refuse to return, not how often an accepted match is right. Read it with the coverage it costs. Historical results on an internal UK fixture with a preloaded catalogue. The September 5 replay has no full-prompt-verified calls and cannot establish current calibration. No independent customer holdout or global coverage claim. This is the Markdown copy of https://entitybind.com/guides/blocking-and-candidate-generation, assembled from the same data the page renders and carrying every sentence, table row, figure and question the page's main content carries. The HTML page is canonical; it adds the layout, the navigation and the interactive parts (the ask panel, the forms), which a copy can name but not run. Every page on the site, with a line on each: https://entitybind.com/llms.txt.