Expected calibration error (ECE)

ECE partitions predictions into bins by stated confidence — ten equal-width bins is the convention — computes the absolute difference between the mean confidence and the observed accuracy within each bin, and averages those differences weighted by bin population. A perfectly calibrated system scores zero. It is usually read beside a reliability diagram, which plots the same information without collapsing it.

Its weaknesses are well documented and worth knowing before quoting one. ECE is sensitive to the binning scheme, so two systems' ECEs are not comparable unless the bins match; it averages over-confidence and under-confidence into one number that hides which is happening; and it is dominated by whichever bin holds most of the mass, which in a well-behaved resolver is the top bin. Adaptive binning and class-wise variants exist and are less quoted.

It also becomes uninformative in exactly the regime a safety-first resolver operates in. If a system only returns decisions above a high floor, nearly all its predictions land in one or two bins, and an ECE computed over them says little beyond what the top bin's accuracy already said.

Entity Bind does not publish an ECE. It publishes instead the worst error rate its observations are still consistent with at 95% — a Wilson lower bound — per decision family, with the observation count behind it. That is a narrower claim, and a reader can check it arithmetically without trusting anybody's binning choice.