Calibration
Find out what your confidence scores actually mean.
Give it decisions a person has already checked. It tells you how often the system is right, which mistakes it makes, and where to draw the lines between automating, reviewing, and not deciding at all.
Your data stays on your device
1. Add labeled examples
One row per decision: what a person decided (expected), what the system decided (predicted), and its confidence from 0 to 1 or 0 to 100. CSV or JSON. Everything stays in this browser tab.
No data to hand? Try a synthetic sample
Accepted column names: expected (or label, actual, truth), predicted (or prediction, decision, output), confidence (or probability, score). Up to 20,000 rows and 2 MB. At least 20 rows; a few hundred is better.
Reading the results honestly
There is no universal threshold. The right one depends on what a wrong automated decision costs, what a review costs, how many decisions you make, and how much risk you can carry. Two teams with the same model and the same data can rightly choose different thresholds.
Confidence is not correctness. A decision made at 95% confidence is either right or wrong. What the reliability table shows is how often decisions at each confidence level turned out right in this sample. If that differs from the stated confidence, set thresholds by what was observed.
The sample has to look like production. These numbers transfer to real traffic only if the examples were drawn at random from it and labeled without seeing the system’s answer. Re-run the analysis whenever the question, the policy, or the model changes.
Go deeper
How to calibrate AI decisions
Building a labeled set, how many examples you need, and the ways to fool yourself.
Calibration metrics
Reliability diagrams, ECE, and the Brier score: what each measures and what it hides.
Confidence thresholds
Why thresholds come from costs, and when each direction deserves its own bar.
Abstention
How the “don’t decide” zone is set, and what to do with the cases in it.
Evaluating LLM classifiers
Test sets, metrics that carry cost, and reading failures before you automate.
Calibrate with us
Pilot teams get help building the labeled set and choosing thresholds.
Need data to calibrate? Run labeled examples through the playground or the benchmark harness.