--- title: "Expected calibration error (ECE)" summary: "Expected calibration error summarises miscalibration as the average gap between stated confidence and observed accuracy across confidence buckets, weighted by how many predictions fall in each." canonical_url: https://entitybind.com/glossary/expected-calibration-error last_updated: 2026-10-15 copy: full --- # Expected calibration error (ECE) > Expected calibration error summarises miscalibration as the average gap between stated confidence and observed accuracy across confidence buckets, weighted by how many predictions fall in each. Canonical page: https://entitybind.com/glossary/expected-calibration-error Last updated: 2026-10-15 Glossary · Measurement ## How to read this The definition as the field writes it comes first, weaknesses and all. What Entity Bind does about it comes last, in its own paragraph, so you can take one without the other. [Filed under measurement](https://entitybind.com/glossary#measurement). By [James Rooney](https://entitybind.com/about), founder of Entity Bind · Last revised 15 October 2026 ## What expected calibration error (ECE) means ECE partitions predictions into bins by stated confidence — ten equal-width bins is the convention — computes the absolute difference between the mean confidence and the observed accuracy within each bin, and averages those differences weighted by bin population. A perfectly calibrated system scores zero. It is usually read beside a reliability diagram, which plots the same information without collapsing it. Its weaknesses are well documented and worth knowing before quoting one. ECE is sensitive to the binning scheme, so two systems' ECEs are not comparable unless the bins match; it averages over-confidence and under-confidence into one number that hides which is happening; and it is dominated by whichever bin holds most of the mass, which in a well-behaved resolver is the top bin. Adaptive binning and class-wise variants exist and are less quoted. It also becomes uninformative in exactly the regime a safety-first resolver operates in. If a system only returns decisions above a high floor, nearly all its predictions land in one or two bins, and an ECE computed over them says little beyond what the top bin's accuracy already said. Entity Bind does not publish an ECE. It publishes instead the worst error rate its observations are still consistent with at 95% — a Wilson lower bound — per decision family, with the observation count behind it. That is a narrower claim, and a reader can check it arithmetically without trusting anybody's binning choice. ## Where this shows up in Entity Bind Three pages where this term stops being vocabulary and starts changing an answer, a payload or a price. - [Calibration and confidence scores](https://entitybind.com/guides/calibration-and-confidence-scores): A confidence score is calibrated when it means what it says: of all the answers returned at 0.90, about 90% should be right. Most scores that reach an API are not… - [The accuracy method](https://entitybind.com/accuracy): Every published rate with the population it was measured over, its interval and its fixture — including the input shapes that fail. - [The audit pack](https://entitybind.com/audit-pack): The fixtures, the verdict logs and the replay path: what an evaluator needs to re-run a published rate for themselves. The fastest way past a definition is a file: which of your rows are the same company, which of your existing matches we would not have made, and what we could not decide. [Start free →](https://app.entitybind.com/app?signup=1) ## The terms next to this one Each with its own sentence, because the next question is usually about the word beside this one. - [Calibration](https://entitybind.com/glossary/calibration): A confidence score is calibrated when it matches observed reality — of the decisions returned at 0.95, about 95% should be correct — which is a different property from being accurate, and a much rarer one. - [Wilson interval](https://entitybind.com/glossary/wilson-interval): The Wilson score interval is a confidence interval for a proportion that stays sensible at the extremes, which is why it is the right interval to publish when a system has made no observed errors at all. - [Confidence floor](https://entitybind.com/glossary/confidence-floor): A confidence floor is the score below which a system refuses to return a match at all, expressed as an acceptance contract rather than as a promise about accuracy. - [Precision](https://entitybind.com/glossary/precision): Precision is the share of the matches a system asserted that were correct — and it is meaningless without the share of records the system was willing to decide at all. ## The other 13 terms on this shelf How you tell whether any of it worked, and which numbers may not be quoted without which other numbers. - [Precision](https://entitybind.com/glossary/precision) - [Recall](https://entitybind.com/glossary/recall) - [F1 score](https://entitybind.com/glossary/f1) - [B-cubed (B³)](https://entitybind.com/glossary/b-cubed) - [Pairwise metrics](https://entitybind.com/glossary/pairwise-metrics) - [Cluster metrics](https://entitybind.com/glossary/cluster-metrics) - [Recall@k](https://entitybind.com/glossary/recall-at-k) - [Calibration](https://entitybind.com/glossary/calibration) - [Wilson interval](https://entitybind.com/glossary/wilson-interval) - [False positive](https://entitybind.com/glossary/false-positive) - [False bind](https://entitybind.com/glossary/false-bind) - [Over-merge](https://entitybind.com/glossary/over-merge) - [Fragmentation](https://entitybind.com/glossary/fragmentation) Previous, alphabetically: [EUID (European Unique Identifier)](https://entitybind.com/glossary/euid) · Next, alphabetically: [F1 score](https://entitybind.com/glossary/f1) ## Evidence basis Internal fixtures only. The historical 2026-08-29 per-tier recording covers 317 GB companies with identifier-rich arrivals seeding the catalogue first; the context tier includes country and registered-town prose. Its legacy replay has no full-prompt-verified calls and cannot certify current calibration. The September 5 CI reports cover 1,176 labelled pairs and 705 records over 250 companies using offline stubs. These are not independent customer holdouts or live production validation; the 99% minimum-input product target remains unmet. Scope is company and organisation identity only — not people, vessels, bank-account ownership, sanctions or emissions. Results may not be used for eligibility decisions about individuals, including employment, credit and tenancy. ## Start with your own company records. Create your workspace, verify your email and run your first company file. The Free plan needs no card. Create API keys in your workspace when you need an integration. [Start free](https://app.entitybind.com/app?signup=1) · [All 64 terms](https://entitybind.com/glossary) 300 credits. No card. For a guided review of your file, explore [the Merge Safety Sprint](https://entitybind.com/audit-pack). --- The 0.90 confidence floor. It is an acceptance contract, not calibration evidence: it says what we refuse to return, not how often an accepted match is right. Read it with the coverage it costs. Entity Bind resolves companies and organisations only. It does not resolve people, vessels, bank-account ownership, sanctions exposure, emissions or geolocation, and its results may not be used for eligibility decisions about individuals, including employment, credit and tenancy. This is the Markdown copy of https://entitybind.com/glossary/expected-calibration-error, assembled from the same data the page renders and carrying every sentence, table row, figure and question the page's main content carries. The HTML page is canonical; it adds the layout, the navigation and the interactive parts (the ask panel, the forms), which a copy can name but not run. Every page on the site, with a line on each: https://entitybind.com/llms.txt.