How to calibrate AI decisions with labeled examples

A model that says “90% confident” is making a claim you can test. Calibration is the test: take the decisions where it said 90%, check how many were right, and see if the answer is close to 90%. Evaluation tells you how often the system is right overall. Calibration tells you whether its confidence is a reliable guide to which decisions are right, and that is what thresholds depend on.

This guide is the practical procedure: what data you need, what to compute, and how to read the result without fooling yourself.

Step 1: Build a labeled set

You need real inputs with the correct answer attached by a person. Three things determine whether the set is worth anything.

• It must look like production. Sample randomly from real traffic over a period long enough to include quiet days and bad days. A set of memorable examples that someone collected is a set of unusual examples.
• Labels must be made without seeing the model’s answer. People shown a suggestion tend to agree with it, which makes the model look better than it is.
• Hard cases need a second opinion. Have two people label a subset. Where they disagree, the policy is ambiguous. No model will be more consistent than your own reviewers are with each other, and their agreement rate is the ceiling on measurable accuracy.

How many examples?

More than feels necessary. With small samples, the uncertainty in the measurement is larger than the differences you are trying to see. Suppose 48 of 50 automated decisions were correct. That is 96%, but a 95% confidence interval on that figure runs from roughly 87% to 99%. You could not tell a 4% error rate from a 13% one.

A few hundred examples is a reasonable minimum for a first threshold, and the part of the set that matters most is the part above your threshold. If only 30 examples land in the automate zone, your estimate of automated accuracy rests on 30 examples, however large the full set is.

Step 2: Run the decision and record three things

For each example you need the expected label, the predicted label, and the confidence in the prediction. For a predicate, the confidence in the predicted side is the probability if the answer is yes and one minus the probability if it is no.

// For every labeled example, record what the model said and how sure it was.
const rows = [];
for (const example of labeledExamples) {
  const { answers } = await client.decisions.create({
    model: "gpt-6-luna",
    input: example.input,
    questions: [{ type: "predicate", name: "approve", instructions: QUESTION }],
  });
  const answer = answers[0];
  if (answer.type !== "predicate") continue; // count refusals separately

  const approve = answer.probability >= 0.5;
  rows.push({
    expected: example.label,                       // what a person decided
    predicted: approve ? "approve" : "deny",       // what the model decided
    confidence: approve ? answer.probability : 1 - answer.probability,
  });
}

That gives you a table like this, which is the input format our calibration tool reads:

expected,predicted,confidence
approve,approve,0.97
deny,deny,0.91
approve,deny,0.58
deny,approve,0.74
approve,approve,0.88

Step 3: Measure accuracy and the two kinds of error

Overall accuracy is the share of rows where predicted equals expected. It is the number everyone asks for and the least useful on its own, because it hides which mistakes are being made. For a yes/no decision there are two:

• False approve (false positive): the model said approve; the right answer was deny.
• False reject (false negative): the model said deny; the right answer was approve.

These almost never cost the same, and a system can have excellent accuracy while making only the expensive kind. With more than two labels, the confusion matrix shows the same thing in full: every expected label against every predicted label, so you can see that “billing” is being confused with “sales” specifically.

Step 4: Check confidence against reality

Group the rows by confidence and compare the average confidence in each group with the accuracy you observed there.

In this example the top band can be trusted and the lower bands cannot. That is a useful finding in itself. It says the automation threshold belongs at or above 0.90, and that answers under 0.70 should not even be shown to reviewers as suggestions.

To summarize calibration in one number, or to compare two models or prompts, see reliability diagrams, ECE, and the Brier score, including what each one hides.

Step 5: Choose thresholds from the trade-off

For each candidate threshold, compute two numbers: the share of decisions at or above it (coverage, the part you would automate) and the accuracy within that share. Raising the threshold improves accuracy and lowers coverage. There is no free point on that curve, only the one whose costs you prefer.

Put prices on it. Each automated error costs something, each review costs something, and the cheapest threshold is the one that minimizes the total. The thresholds guide works through the arithmetic. Then set a lower threshold too, below which answers are too unreliable to suggest.

Ways to fool yourself

• Tuning and testing on the same examples. A threshold chosen on a dataset will look better on that dataset than on new data. Hold some examples back and check the threshold on those.
• Ignoring refusals. If the model declines 3% of cases and you drop them, your accuracy describes the other 97%. Report the refusal rate beside it.
• Treating the label as truth. Labels are human decisions and contain errors. Some “model mistakes” are labeling mistakes. Review the disagreements before drawing conclusions.
• Calibrating once. Policies change, products change, and model versions change. Any of them can move the relationship between confidence and accuracy.

Step 6: Keep measuring

After launch, your audit sample and review queue keep producing labeled examples. Re-run the analysis monthly, and immediately after any change to the question wording, the policy, or the model. If accuracy in the automate zone has dropped below your target, raise the threshold first and investigate second.

Related

• Confidence thresholds: the cost model behind a threshold.
• Benchmarks: our evaluation method and published datasets.
• Refund eligibility template: a decision worth calibrating before you automate it.