--- title: "Calibration and confidence scores" summary: "What a calibrated confidence actually means, how expected calibration error is computed, why a similarity score is not a probability, and why an LLM's self-reported confidence is neither." canonical_url: https://entitybind.com/guides/calibration-and-confidence-scores last_updated: 2026-10-12 copy: full --- # Calibration and confidence scores > What a calibrated confidence actually means, how expected calibration error is computed, why a similarity score is not a probability, and why an LLM's self-reported confidence is neither. Canonical page: https://entitybind.com/guides/calibration-and-confidence-scores Last updated: 2026-10-12 For the team that owns a company master and has to defend it. A confidence score is calibrated when it means what it says: of all the answers returned at 0.90, about 90% should be right. Most scores that reach an API are not calibrated. A string similarity is a statement about characters, a model's self-reported confidence is a statement about how much evidence it felt it had, and neither is a frequency. The distinction is not academic, because every threshold a caller writes — every "only act above 0.9" in downstream code — is meaningless unless the number underneath it is a probability. - Reading time: 10 min read - Sections: 9 - Questions answered: 6 - Key takeaways: 6 The same measurement, on your rows: A first file is a few hundred of your own rows with the matches you already have beside them. The numbers in this article come back measured on your data, and we write nothing to your systems. [Start free →](https://app.entitybind.com/app?signup=1) Self-service. Create your workspace online and verify your email to start matching company records. Free starts with 300 credits and no card. Paid monthly plans are available in workspace Billing; VAT is added where applicable. Usage is tracked without automatic overage charges. [Trust and security](https://entitybind.com/trust) A stewardship queue, a CRM after a migration or a merger, a supplier file before a payment run, counterparty onboarding that has to survive an audit. Read the argument here, take the takeaways at the foot into your next vendor conversation, and send us a file when you want the same measurement on your own rows. How matching works · 2,167 words · every figure carries the population it was measured over By [James Rooney](https://entitybind.com/about), founder of Entity Bind · Last revised 12 October 2026 ## What calibrated means Pull every answer that came back at 0.80 and check them. If 80% are right, the score is calibrated at that point. If 55% are right, it is overconfident there, and any caller thresholding at 0.75 is acting on a claim the system cannot support. Stated formally: a scoring function is calibrated if, among all the cases it assigns a score of p, the proportion that are actually positive is p. Calibration is a property of a population, not of an answer. There is no such thing as checking whether one 0.90 was calibrated; you can only check whether the set of 0.90s was. This is why every calibrated confidence must name the population it was derived from, and why a confidence field with no stated population is decoration. It is also worth separating calibration from sharpness, because they are different virtues and only one of them is useful alone. A system that returns its population's base rate — say 0.7 — for every single pair is perfectly calibrated and completely useless, because it never distinguishes anything. Sharpness is the willingness to be near 0 and near 1; calibration is being right about it when you are. You want both, and calibration is the one that can be checked. ## A similarity score is not a probability A Jaro-Winkler of 0.92, a Levenshtein ratio of 0.88, a cosine similarity of 0.94 between two embeddings: none of these is a probability of anything. They are distances in a space chosen for computational convenience, and their relationship to the frequency of true matches is unknown until somebody measures it. The measurement is usually surprising. String similarity in company data is badly behaved at exactly the point where you would want it to be reliable: the most similar pairs in a company register are frequently sibling entities you must not merge — a holding company and its operating company, a numbered successor vehicle, a dormant name-protection entity. So the highest similarity scores contain a concentration of the worst errors, and a naive threshold makes the highest-confidence decisions the least reliable ones. Embedding cosine has the same problem plus one more: the score's scale drifts with the model. Change the embedding model and every threshold in the system silently becomes a different threshold, with no error and no test failure. A calibrated probability does not have that property, which is a decent practical reason to want one even where a similarity would technically do. ## Reliability diagrams and expected calibration error The standard way to check calibration is to bin the predictions, compute the observed accuracy in each bin, and compare it against the mean claimed confidence in that bin. Plotted, that is a reliability diagram; the perfectly calibrated system lies on the diagonal, an overconfident one sags below it. Expected calibration error summarises the diagram in a single number: the weighted mean absolute gap between claimed and observed, weighted by how many predictions are in each bin. Maximum calibration error is the largest single gap. Here is the arithmetic on an illustrative table — these numbers are made up so the sum can be checked, and are not a measurement. Illustrative reliability table for 1,000 decisions: | Bin | n | Mean claimed | Observed | Gap | | --- | --- | --- | --- | --- | | 0.6–0.7 | 40 | 0.65 | 0.80 | +0.15 | | 0.7–0.8 | 60 | 0.75 | 0.72 | −0.03 | | 0.8–0.9 | 100 | 0.86 | 0.88 | +0.02 | | 0.9–1.0 | 800 | 0.97 | 0.99 | +0.02 | ECE = (40×0.15 + 60×0.03 + 100×0.02 + 800×0.02) / 1000 = 0.0258. MCE = 0.15, and it rests on 40 observations, which is why a maximum computed from a small bin is reported rather than acted on. ## What an ECE hides Before you compare two suppliers' ECE figures, know three things about the statistic, because it moves more easily than it looks. It depends on the binning. Equal-width bins and equal-mass bins give different answers over the same predictions, and a system whose mass sits almost entirely in the top bin — which is normal for a matcher operating above an acceptance floor — will report a flattering ECE under equal-width binning simply because one enormous bin dominates the weighted mean. Ask which binning was used. It is dominated by wherever the mass is. In the table above, the 0.9–1.0 bin contributes 0.016 of the 0.0258 total, and the visibly worst bin contributes 0.006. That is arithmetically correct and it means ECE tells you almost nothing about the tails, which is where the interesting failures often are. And the direction of the error matters more than its size. A system that is under-confident where the mass is — claiming less than it delivers — is failing safely. A system overconfident by the same margin is authorising decisions its evidence does not support. A single unsigned ECE hides that distinction entirely, so the per-bin table should be published with it. ## A model's self-reported confidence is not a probability either When a language model is asked to adjudicate a borderline pair and return a confidence alongside its verdict, that confidence is tempting to surface directly. It should not be. Measured against outcomes, we found our own adjudicator's self-report badly miscalibrated — and, usefully, miscalibrated in the safe direction: on a population of pairs it was willing to decide, it was right every time while claiming a mean confidence well below certainty. That is not a probability. It is a report of how much evidence the model felt it had, which is a different quantity with a different distribution, and surfacing it as a confidence would mean callers thresholding on the model's rhetorical caution rather than on its accuracy. Mapping it to a probability is harder than it sounds when the model is very accurate. Both Platt scaling and isotonic regression degenerate under perfect separation: Platt has no maximum likelihood, and isotonic returns a single constant for every block. So a population with zero observed errors cannot be fitted into a smooth curve — it can only support a bound. Which brings us to how a calibrated field actually gets built. ## How a calibrated confidence field is built Define the field once — the probability that this record is bound to the right real-world company — then derive its value per decision stage from the observed accuracy of that stage's own population, and publish the interval's lower bound rather than the observed rate. Three steps, in that order. That produces conservative numbers and it produces defensible ones. The nominal 95% Wilson lower bound of 1,190 historical prompt-v8 uncorroborated MATCH verdicts with zero recorded errors. The 2026-08-29 population pools six input shapes with repeated company families and calibration/evaluation overlap. Its independence and transfer to new customer inputs are unproven. Legacy replay cannot certify current calibration. Three properties make this an engineering artefact rather than a marketing one. The values are pure functions of recorded trial and error counts with no tunable term, so there is no dial. The tests recompute every shipped value from its counts, so an edited constant fails the build. And the populations are keyed by model, prompt version and evidence tier together, so a verdict from a new prompt version has no calibrated population and is withheld rather than inheriting the old one's number. That last property has a visible cost, and it is the right cost. A prompt change invalidates its calibration, and until a new recording is fitted, verdicts from the new prompt cannot authorise a bind. A system that let them would be reporting a confidence measured on a model that is no longer running. ## Confidence limits and the coverage a filter costs A consequence of deriving confidence from interval lower bounds is that the field has a maximum, and it is not 1.0. Nothing can claim more certainty than its population supports, and no population is infinite. The highest shipped population bound is 0.9968. The nominal 95% Wilson lower bound of 1,190 historical prompt-v8 uncorroborated MATCH verdicts with zero recorded errors. The 2026-08-29 population pools six input shapes with repeated company families and calibration/evaluation overlap. Its independence and transfer to new customer inputs are unproven. Legacy replay cannot certify current calibration. Confidence bounds depend on the recorded evidence population; there is no universal score for every match. Read match_confidence_basis and the evidence. A 0.99 filter can return some v8 adjudicated matches while excluding valid identifier matches, so validate its coverage cost on your own inputs. Evaluate a stricter threshold on your own labelled records and measure the accepted matches it removes. The reject direction is measured separately and is weaker. The historical prompt-v8 rejection population has 54 verdicts and four errors. Its bound authorises no accepted match; other non-match populations carry their own values. The reject-side bound is 0.8245. Symmetry here would be an assumption rather than a measurement: the evidence for "these are the same company" and the evidence for "this company is not in the catalogue at all" are different populations with different sizes, and pooling them to produce a tidier pair of numbers would invent a population that never ran. ## A floor is not calibration evidence The commonest confusion in this area is between an acceptance threshold and a calibrated score. They answer different questions. It is an acceptance contract, not calibration evidence: it says what we refuse to return, not how often an accepted match is right. Read it with the coverage it costs. Concretely: A match below 0.90 confidence is unrepresentable. The record comes back as an abstention with a ranked shortlist instead. It is a rule the code cannot break rather than a target it aims at — Enforced as a domain invariant on the resolution outcome and again by a predicate called on the online, batch and replay paths, so a sub-floor accepted match cannot be constructed on any of them. A cached replay whose inherited confidence has fallen below the floor is withheld rather than re-served. So it tells you what the system refuses to return. It does not tell you how often what it does return is right, and a vendor quoting a floor as though it were an accuracy figure is answering a question you did not ask. The two carve-outs are worth naming because they are the places the invariant deliberately does not apply. A merge a human reviewer approves is recorded against that reviewer and excluded from the automated population every accuracy figure is measured over, so a reviewed decision never launders itself into the automated statistics. And a new entity minted because nothing matched is not a match at all — it carries reject-side confidence, which is reported separately rather than folded into the floor. ## What to ask a supplier about their confidence field Five questions, and the answers are usually short enough to be revealing. - Is it a probability or a similarity? If the answer involves a distance metric, it is a similarity and your threshold is arbitrary. - What population was it calibrated on, and how large was it? A confidence with no denominator is decoration. - Is it a point estimate or an interval bound? A field derived from lower bounds is conservative by construction, which is the direction you want. - What is the maximum value it can take? If they do not know, nobody has checked what a strict filter does. - What invalidates the calibration? A model change, a prompt change and a fixture change should all invalidate it, and a supplier who says nothing does has not keyed it to anything. ## What to take away - Calibrated means the score is a frequency: of everything returned at 0.90, about 90% is right. It is a property of a population, never of a single answer. - Similarity scores are not probabilities, and in company data the most similar pairs concentrate the worst errors — sibling and holding-company entities. - ECE is a weighted mean gap dominated by whichever bin holds the mass, and it depends on the binning. Publish the per-bin table beside it. - A language model's self-reported confidence measures how much evidence it felt it had. Ours was miscalibrated in the safe direction and is not surfaced as a confidence. - Confidence limits depend on the recorded population. Confidence bounds depend on the recorded evidence population; there is no universal score for every match. Read match_confidence_basis and the evidence. A 0.99 filter can return some v8 adjudicated matches while excluding valid identifier matches, so validate its coverage cost on your own inputs. - An acceptance floor says what will not be returned. It is not evidence about how often what is returned is right. Those takeaways hold for anybody's data. The number for yours comes out of your own rows — a few hundred of them, with the matches you already have beside them. [Start free →](https://app.entitybind.com/app?signup=1) ## Read next Each article names its own further reading. These pick up a question this one raises and leaves open. - [How to measure entity resolution accuracy](https://entitybind.com/guides/how-to-measure-entity-resolution-accuracy) (13 min read): Precision, recall, abstention, pairwise versus B-cubed cluster metrics, Wilson intervals and out-of-fold evaluation. Why a single accuracy number for entity resolution is always meaningless, with worked arithmetic. - [Fellegi-Sunter explained, with worked arithmetic](https://entitybind.com/guides/fellegi-sunter-explained) (10 min read): The probabilistic record linkage model behind most matching engines: m and u probabilities, match weights, the three decision regions, and a worked example you can check with a calculator. - [Precision, recall and abstention](https://entitybind.com/guides/precision-recall-and-abstention) (8 min read): Entity resolution is a three-way trade, not a two-way one. Why abstention is a design choice rather than a failure, what an abstention has to carry to be useful, and how to measure a system that declines. Or try your own rows: create your Free workspace and upload a company CSV to inspect its results. Contact us for an assisted measurement against your labelled data, with the working handed back. [Start free →](https://app.entitybind.com/app?signup=1) ## Where this article is applied 5 pages name this method as the working behind what they claim. - [AI agents](https://entitybind.com/use-cases/ai-agents) (Use case): A counterparty check before the agent acts, and a receipt for what it acted on. - [Entity Bind vs Fuzzy matching](https://entitybind.com/compare/fuzzy-matching) (Comparison): Free, already there, and perfectly good until a wrong merge costs money. - [Entity Bind vs Splink](https://entitybind.com/compare/splink) (Comparison): Free, open source, and it weighs the evidence properly: Fellegi–Sunter, done right. Our scoring stage is the same family. - [Entity Bind vs AWS Entity Resolution](https://entitybind.com/compare/aws-entity-resolution) (Comparison): Cheap, inside your own account, and it needs no new vendor. It also matches only what you loaded. - [Entity Bind vs Tamr](https://entitybind.com/compare/tamr) (Comparison): Machine-learning mastering at enterprise scale. It learns your labels — including the wrong ones. ## Is this your file? The article is for anyone. The measurement behind it is worth commissioning for one kind of team. ### Built for Teams with a company master they have to defend — an MDM or stewardship queue, a CRM after a migration or a merger, a supplier file before a payment run, or counterparty onboarding that has to survive an audit. ### Wrong for Buying breadth as the product: we hold a fraction of an incumbent’s records and are not building a global company graph. Entity Bind resolves companies and organisations only. It does not resolve people, vessels, bank-account ownership, sanctions exposure, emissions or geolocation, and its results may not be used for eligibility decisions about individuals, including employment, credit and tenancy. ## The basis for every figure above Internal fixtures only. The historical 2026-08-29 per-tier recording covers 317 GB companies with identifier-rich arrivals seeding the catalogue first; the context tier includes country and registered-town prose. Its legacy replay has no full-prompt-verified calls and cannot certify current calibration. The September 5 CI reports cover 1,176 labelled pairs and 705 records over 250 companies using offline stubs. These are not independent customer holdouts or live production validation; the 99% minimum-input product target remains unmet. Scope is company and organisation identity only — not people, vessels, bank-account ownership, sanctions or emissions. Results may not be used for eligibility decisions about individuals, including employment, credit and tenancy. Figures read off the product repository on 2026-09-05. The per-tier rates come from the live-adjudicated 317-company recording of 2026-08-29; the gate figures come from the stub-adjudicated continuous-integration run and are a floor rather than a ceiling. The two are never averaged. Entity Bind holds no SOC 2, no ISO 27001 and no other certification, and claims none. Every figure, with the population it was measured over and the method behind both recordings, is set out on [the accuracy page](https://entitybind.com/accuracy). The retention, erasure, tenancy and sub-processor detail a security review asks for is under [Trust and security](https://entitybind.com/trust), with where assurance stands: no external validation of the accuracy figures and no customer holdout. ## The same measurement, on your own rows. Create your workspace, verify your email and run your first company file. The Free plan needs no card. Create API keys in your workspace when you need an integration. [Start free](https://app.entitybind.com/app?signup=1) · [Plans and prices](https://entitybind.com/pricing) 300 credits. No card. For a guided review of your file, explore [the Merge Safety Sprint](https://entitybind.com/audit-pack). ## The questions this article answers ### What does a calibrated confidence score mean? It means the score is a frequency you can act on: among all the answers returned at 0.90, about 90% are correct. Calibration is a property of a population rather than of an individual answer, so a calibrated confidence has to name the population it was derived from and its size. A score with no stated population cannot be checked and should not be thresholded against. ### Is a similarity score the same as a confidence? No. A Jaro-Winkler, Levenshtein or cosine value is a distance in a space chosen for computational convenience, and its relationship to the frequency of true matches is unknown until somebody measures it. In company data that relationship is particularly badly behaved, because the most similar record pairs in a register are often sibling entities — holding and operating companies, numbered successor vehicles — that must not be merged. ### What is expected calibration error? Bin the predictions by claimed confidence, compute the observed accuracy in each bin, and take the weighted mean absolute gap between claimed and observed, weighted by bin size. It is a useful summary and it has three caveats: it depends on the binning scheme, it is dominated by whichever bin holds most of the predictions, and it is unsigned, so it hides whether a system is overconfident (dangerous) or underconfident (safe). Publish the per-bin table with it. ### Why does your confidence score cap below 1.0? Every value in the field is bounded by a confidence interval over a finite population rather than an observed rate. The highest shipped population bound is 0.9968. The nominal 95% Wilson lower bound of 1,190 historical prompt-v8 uncorroborated MATCH verdicts with zero recorded errors. The 2026-08-29 population pools six input shapes with repeated company families and calibration/evaluation overlap. Its independence and transfer to new customer inputs are unproven. Legacy replay cannot certify current calibration. Confidence bounds depend on the recorded evidence population; there is no universal score for every match. Read match_confidence_basis and the evidence. A 0.99 filter can return some v8 adjudicated matches while excluding valid identifier matches, so validate its coverage cost on your own inputs. ### Can I trust a language model's confidence output? Not as a probability. Measured against outcomes, our own adjudicator's self-reported confidence was badly miscalibrated — safely, in that it understated its accuracy, but miscalibrated all the same. It measures how much evidence the model felt it had, which is a different quantity. Mapping it properly is also harder than it sounds: on a population with zero observed errors, both Platt scaling and isotonic regression degenerate, so such a population can support a bound and not a curve. ### Is a confidence floor the same as calibration? It is an acceptance contract, not calibration evidence: it says what we refuse to return, not how often an accepted match is right. Read it with the coverage it costs. --- This is the Markdown copy of https://entitybind.com/guides/calibration-and-confidence-scores, assembled from the same data the page renders and carrying every sentence, table row, figure and question the page's main content carries. The HTML page is canonical; it adds the layout, the navigation and the interactive parts (the ask panel, the forms), which a copy can name but not run. Every page on the site, with a line on each: https://entitybind.com/llms.txt.