--- title: "Deduplication" summary: "Deduplication is entity resolution applied within a single dataset: finding the rows that are the same thing and deciding which of them may safely be collapsed." canonical_url: https://entitybind.com/glossary/deduplication last_updated: 2026-10-15 copy: full --- # Deduplication > Deduplication is entity resolution applied within a single dataset: finding the rows that are the same thing and deciding which of them may safely be collapsed. Canonical page: https://entitybind.com/glossary/deduplication Last updated: 2026-10-15 Glossary · Foundations ## How to read this The definition as the field writes it comes first, weaknesses and all. What Entity Bind does about it comes last, in its own paragraph, so you can take one without the other. [Filed under foundations](https://entitybind.com/glossary#foundations). By [James Rooney](https://entitybind.com/about), founder of Entity Bind · Last revised 15 October 2026 ## What deduplication means Deduplication is the case where the file is compared against itself. A CRM with four rows for the same customer, a supplier master with three vendor codes for the same legal entity, a marketing list assembled from six imports — all of them are the same problem, and all of them have the same trap, which is that collapsing is destructive and finding is not. Separate the two halves. Finding duplicates is a retrieval and scoring problem with a tolerable error profile: a false candidate costs a reviewer a few seconds. Merging duplicates is a write, and a wrong merge is expensive to detect and often impossible to reverse cleanly once downstream systems have replicated the surviving id. Most deduplication disasters are not detection failures; they are detection successes that were applied automatically at a threshold nobody measured. The two failure modes have names. Merging records that are not the same company is an over-merge; leaving records that are the same company in separate clusters is fragmentation. They trade off against each other, and any single-number 'match rate' hides which one a tool prefers. Entity Bind makes the merge a decision with a receipt, refuses any binding below the 0.90 confidence floor, and writes nothing into your systems: a Merge Safety Sprint returns which rows are the same company, which of your existing matches it would not have made, and what it could not decide. It is an acceptance contract, not calibration evidence: it says what we refuse to return, not how often an accepted match is right. Read it with the coverage it costs. ## Where this shows up in Entity Bind Three pages where this term stops being vocabulary and starts changing an answer, a payload or a price. - [What is entity resolution?](https://entitybind.com/guides/what-is-entity-resolution): Entity resolution decides which records in one or more datasets refer to the same real-world entity, and joins them under a single stable identifier. - [CRM and RevOps](https://entitybind.com/use-cases/crm-and-revops): The duplicate-account problem in a running CRM, and what a merge that should not have happened costs to unwind. - [Compared with fuzzy matching](https://entitybind.com/compare/fuzzy-matching): A score for how alike two names look, against a decision about which company they name — and the rows where the fuzzy matching you already run wins. The fastest way past a definition is a file: which of your rows are the same company, which of your existing matches we would not have made, and what we could not decide. [Start free →](https://app.entitybind.com/app?signup=1) ## The terms next to this one Each with its own sentence, because the next question is usually about the word beside this one. - [Entity resolution](https://entitybind.com/glossary/entity-resolution): Entity resolution is the process of deciding which records, across one or many datasets, refer to the same real-world thing — and, when the evidence does not settle it, deciding that too. - [Over-merge](https://entitybind.com/glossary/over-merge): An over-merge is a cluster containing records from more than one real entity — the failure that turns two customers, two suppliers or two counterparties into one. - [Fragmentation](https://entitybind.com/glossary/fragmentation): Fragmentation is the opposite failure to over-merging: one real entity split across several clusters, so the same company is counted, contracted with or invoiced more than once. - [Golden record](https://entitybind.com/glossary/golden-record): A golden record is the single assembled view of an entity built by choosing one value per field from the records attached to it, ideally with the provenance of each choice carried alongside it. - [Confidence floor](https://entitybind.com/glossary/confidence-floor): A confidence floor is the score below which a system refuses to return a match at all, expressed as an acceptance contract rather than as a promise about accuracy. - [Decision receipt](https://entitybind.com/glossary/decision-receipt): A decision receipt is the durable, retrievable record of why a particular match was made — the features, the probability, the thresholds in force, the sources cited and the version of the pipeline that decided — addressable by an identifier returned with the answer. ### Answered at length elsewhere - [How to deduplicate a CRM](https://entitybind.com/guides/how-to-deduplicate-a-crm) (Guide): Export, preview, tier the file, run it, review the abstentions, merge with receipts, measure. The full sequence for deduplicating a CRM account object without making a wrong merge you cannot undo. - [How to deduplicate a supplier master](https://entitybind.com/guides/how-to-deduplicate-a-supplier-master) (Guide): Duplicate vendors, payment leakage, remittance conflicts and parent concentration. The accounts payable playbook for finding which vendor records are one company, with a citation for every finding. - [Salesforce](https://entitybind.com/integrations/salesforce) (Integration): An integration pattern, not a managed package: a named credential, a callout that resolves an account, and fields for the receipt. Nothing is written back. - [Entity resolution vs record linkage: which term is right](https://entitybind.com/guides/entity-resolution-vs-record-linkage) (Guide): Record linkage, entity resolution, deduplication, entity matching, entity linking and identity resolution name overlapping problems from different fields. What each term means and when each one is correct. ## The other 9 terms on this shelf What the problem is called, and what the objects in it are. Entity, record, cluster, and the four industry names for the same job. - [Entity resolution](https://entitybind.com/glossary/entity-resolution) - [Record linkage](https://entitybind.com/glossary/record-linkage) - [Entity](https://entitybind.com/glossary/entity) - [Record](https://entitybind.com/glossary/record) - [Cluster](https://entitybind.com/glossary/cluster) - [Golden record](https://entitybind.com/glossary/golden-record) - [Survivorship](https://entitybind.com/glossary/survivorship) - [Master data management (MDM)](https://entitybind.com/glossary/master-data-management) - [Reference data](https://entitybind.com/glossary/reference-data) Previous, alphabetically: [Decision receipt](https://entitybind.com/glossary/decision-receipt) · Next, alphabetically: [Deterministic matching](https://entitybind.com/glossary/deterministic-matching) ## Evidence basis Internal fixtures only. The historical 2026-08-29 per-tier recording covers 317 GB companies with identifier-rich arrivals seeding the catalogue first; the context tier includes country and registered-town prose. Its legacy replay has no full-prompt-verified calls and cannot certify current calibration. The September 5 CI reports cover 1,176 labelled pairs and 705 records over 250 companies using offline stubs. These are not independent customer holdouts or live production validation; the 99% minimum-input product target remains unmet. Scope is company and organisation identity only — not people, vessels, bank-account ownership, sanctions or emissions. Results may not be used for eligibility decisions about individuals, including employment, credit and tenancy. ## Start with your own company records. Create your workspace, verify your email and run your first company file. The Free plan needs no card. Create API keys in your workspace when you need an integration. [Start free](https://app.entitybind.com/app?signup=1) · [All 64 terms](https://entitybind.com/glossary) 300 credits. No card. For a guided review of your file, explore [the Merge Safety Sprint](https://entitybind.com/audit-pack). --- Entity Bind resolves companies and organisations only. It does not resolve people, vessels, bank-account ownership, sanctions exposure, emissions or geolocation, and its results may not be used for eligibility decisions about individuals, including employment, credit and tenancy. This is the Markdown copy of https://entitybind.com/glossary/deduplication, assembled from the same data the page renders and carrying every sentence, table row, figure and question the page's main content carries. The HTML page is canonical; it adds the layout, the navigation and the interactive parts (the ask panel, the forms), which a copy can name but not run. Every page on the site, with a line on each: https://entitybind.com/llms.txt.