Measuring calibration: reliability diagrams, ECE, and the Brier score

An LLM classifier that reports 90% confidence is making a testable claim: answers like this one are right about 90% of the time. Calibration metrics test that claim across many decisions. They matter because every confidence threshold assumes the number means something, and a threshold set on a miscalibrated number automates more mistakes than it appears to.

This guide covers the three measurements you will meet most often, what each one hides, and which to use for which question. The calibration guide covers collecting the labeled data they all need.

What calibration measures, and what it doesn’t

A classifier is well calibrated when, among the decisions where it reports confidence c, the share that are correct is close to c. That is a different property from accuracy, and from the ability to tell easy cases from hard ones.

A simple example shows the difference. Suppose 30% of refund requests should be approved, and a model gives every request a 30% probability of approval. It is perfectly calibrated: the requests it scores at 30% are approved 30% of the time. It is also useless, because it says the same thing about every request. Calibration tells you whether to believe the number; it does not tell you whether the number is informative.

Reliability diagrams

Group decisions into bins by confidence, then compare each bin’s average confidence with its observed accuracy. Plotted, perfect calibration is the diagonal. Points below it are overconfident (the model claims more than it delivers); points above it are underconfident. As a table, it looks like this:

Two choices change what the diagram shows, so state them whenever you report one.

• Bin edges. Equal-width bins (0.9–1.0, 0.8–0.9, and so on) are easy to read but often leave the low bins nearly empty, because LLM confidence often clusters near the top. Equal-count bins put the same number of decisions in each, at the cost of uneven edges.
• Counts. A bin of 30 decisions with 80% observed accuracy is compatible with a true accuracy anywhere from about 63% to 90%. Show the count beside every point, and treat small bins as rough.

Expected calibration error (ECE)

ECE condenses the reliability diagram into one number: the average gap between confidence and accuracy, weighted by how many decisions fall in each bin.

For the table above, with 680 decisions in total: (412 × 1 + 138 × 4 + 71 × 13 + 59 × 8) ÷ 680 is about 3.5 percentage points of ECE. The MCE is 13 points, in the 0.70–0.80 bin. ECE was popularized as a measure for modern neural networks by Guo and colleagues in 2017, and it is the number most calibration comparisons report.

It is also easy to misread:

• It depends on the binning. The same predictions give different ECE with 10 bins and with 15. Compare ECE only across runs that used the same bins.
• A low ECE can describe a useless model. The model that says 30% for every refund request has an ECE of zero.
• It averages away the bin you care about. In the example, 3.5 points sounds small, yet decisions between 0.70 and 0.80 are 13 points less accurate than claimed. If your threshold sits in that range, the average is the wrong number to look at.

The Brier score

The Brier score, introduced by Glenn Brier in 1950 for weather forecasts, needs no bins. For each yes/no prediction, square the difference between the predicted probability and what happened (1 if it happened, 0 if not), then average.

The Brier score is the mean, 0.125. Zero is perfect, and lower is better. Two reference points make it readable:

• Always predicting 0.5 scores 0.25, whatever happens.
• Always predicting the base rate b scores b × (1 − b): 0.21 for the refund example where 30% are approved. A model has to beat this to add anything.

Because it rewards probabilities that are both calibrated and decisive, the Brier score is the better single number for asking whether a change (a new prompt, a new model, new question wording) made the probabilities better overall. It can be split into a calibration term and a resolution term (Murphy, 1973) when you need to know which one moved.

Computing them

This works on the same rows the calibration tool reads: expected label, predicted label, and the confidence in the predicted label. It returns top-label ECE, MCE, and Brier score.

// rows: [{ expected, predicted, confidence }], where confidence is the support
// for the predicted label (the format the calibration tool reads).
function calibrationMetrics(rows, bins = 10) {
  const groups = Array.from({ length: bins }, () => ({ n: 0, confidence: 0, correct: 0 }));
  let brier = 0;

  for (const { expected, predicted, confidence } of rows) {
    const correct = predicted === expected ? 1 : 0;
    const group = groups[Math.min(bins - 1, Math.floor(confidence * bins))];
    group.n += 1;
    group.confidence += confidence;
    group.correct += correct;
    brier += (confidence - correct) ** 2;
  }

  let ece = 0;
  let mce = 0;
  for (const group of groups) {
    if (group.n === 0) continue;
    const gap = Math.abs(group.correct / group.n - group.confidence / group.n);
    ece += (group.n / rows.length) * gap;
    mce = Math.max(mce, gap);
  }
  return { ece, mce, brier: brier / rows.length, groups };
}

Which number for which question

When calibration is off

Usually you don’t need to fix the model. You need to set thresholds by observed accuracy instead of stated confidence: if answers at 0.85 are right 81% of the time, choose thresholds from the 81%. That is what the calibration tool does.

If you need calibrated probabilities themselves, for example to multiply them by costs, fit a mapping from stated confidence to observed accuracy on labeled data you hold out for the purpose. Histogram binning (replace each bin’s confidence with its observed accuracy) and isotonic regression are the common methods. Check the mapping on a third set of examples, because a mapping always looks perfect on the data it was fitted to.

Pitfalls

• Mixing tasks. Calibration is a property of a model on a task. Pooling refund decisions with spam decisions produces a number that describes neither.
• Leaving out refusals. Report the refusal rate beside the metrics, or they describe only the questions the model chose to answer.
• Measuring once. A change of model, prompt, or input mix can move calibration. Re-measure after each one.
• Trusting labels blindly. Labels are human judgments. Some “overconfident” answers are labeling errors; check disagreements by hand.