How to evaluate AI decisions before you automate them

Before you let an AI system make a decision on its own, whether it is an LLM classifier labeling tickets or a model approving refunds, you need to know three things: how often it is right, which mistakes it makes when it is wrong, and what those mistakes cost. Evaluation answers those questions on data you control. It is a different job from calibration, which asks whether the system’s confidence tells you which answers to trust. You need both, and evaluation comes first.

Start from the decision, not the model

An evaluation is only as good as its definition of “right”. Before collecting any data, write down the decision as if you were handing it to a new colleague: the exact question, what happens on each answer, and what each kind of mistake costs.

decision:   needs_engineering
question:   "Does this support issue need to go to engineering?"
type:       predicate
on yes:     create an engineering ticket, notify the on-call channel
on no:      stays with support
false yes:  an engineer spends ~30 min triaging something support could handle
false no:   a real defect waits in the support queue, ~1 day of delay
volume:     ~1,000 tickets a month, ~6% truly need engineering

If you can’t fill in the two “false” lines, stop there. Every metric below is a way of counting mistakes, and you can’t choose which mistakes to count until you know which ones hurt. Writing the spec also flushes out disagreement early: it is common to find that support and engineering have different definitions of “needs engineering”, which no model can resolve for you.

Build a test set worth trusting

The test set is the part of an evaluation most often done badly. Four rules:

• Sample from real traffic, at random. Hand-picked examples are memorable, which means unusual. A random sample over several weeks includes the boring cases that make up most of production.
• Label blind. People who see the model’s answer before labeling tend to agree with it. Label first, run the model second.
• Measure your own agreement. Have two people label the same 50 to 100 cases. Their agreement rate is roughly the ceiling on how accurate any system can look against those labels, and the cases they disagree on usually point at an ambiguous policy.
• Freeze it. Version the set and stop editing it once you start comparing systems. Fixing a genuine labeling error is fine; log the change. Quietly relabeling cases the model “got wrong” is how evaluations end up measuring nothing.

Keep a separate challenge set of hard cases you collect deliberately: sarcasm, two problems in one message, a customer who is wrong about the facts. Report it separately. Mixing it into the random sample makes the headline number describe neither production nor the hard cases.

Report the numbers that carry cost

Accuracy is the number everyone asks for and the one most likely to mislead. Consider the spec above: about 6% of tickets truly need engineering. In a test set of 1,000 tickets that is 60.

The system that never escalates anything has the higher accuracy. The second system makes more mistakes in total, 60 false escalations and 10 misses, and is far more useful, because it catches 50 of the 60 real defects. Whether its 60 false escalations are acceptable is a question about engineers’ time, which is why the spec comes first.

Report these, at minimum:

If confidence will drive automation, add a calibration measure as well: how closely stated confidence matches observed accuracy. The calibration metrics guide compares the reliability diagram, ECE, and the Brier score.

Put an interval on every rate. “96% (95% interval: 92%–98%, n = 200)” tells a reader how much to trust the number. “96%” on its own invites them to trust it completely.

// Run every labeled example twice: once for the answer, once to check consistency.
const results = [];
for (const example of testSet) {
  const [first, second] = await Promise.all([decide(example.input), decide(example.input)]);
  results.push({
    id: example.id,
    expected: example.label,                  // "yes" | "no", labeled blind
    predicted: first.label,                   // the model's answer
    confidence: first.confidence,             // support for that answer
    refused: first.refused,
    consistent: first.label === second.label, // same answer on a repeat call?
    latencyMs: first.latencyMs,
  });
}

const answered = results.filter((r) => !r.refused);
const count = (fn) => answered.filter(fn).length;
const tp = count((r) => r.predicted === "yes" && r.expected === "yes");
const fp = count((r) => r.predicted === "yes" && r.expected === "no");
const fn = count((r) => r.predicted === "no" && r.expected === "yes");
const tn = count((r) => r.predicted === "no" && r.expected === "no");

console.table({
  accuracy: (tp + tn) / answered.length,
  precision: tp / (tp + fp),   // of the tickets sent to engineering, how many belonged there
  recall: tp / (tp + fn),      // of the tickets that belonged there, how many were sent
  refusalRate: (results.length - answered.length) / results.length,
  inconsistency: 1 - results.filter((r) => r.consistent).length / results.length,
});

Read the failures

Numbers tell you how many mistakes there are. Only reading them tells you why, and the reason determines the fix. Go through every error in a sample of at least 30 and put each in one of these buckets:

In practice the first four buckets are often larger than the last. That is good news: they are fixed on your side, and fixing them improves every system you evaluate, not only the current one.

Compare against something

A number with nothing to compare it to cannot tell you whether to ship. Evaluate at least one baseline on the same frozen set:

• The majority answer. “Always no” above. Any system must beat it on the metric you care about.
• Your current process. If people make this decision today, their agreement with the labels is the bar. If rules make it, run the rules.
• A simple alternative. Keyword rules or a general-purpose model with Structured Outputs. If the simple option is nearly as good, the gap may not justify the change.

Our benchmarks publish a keyword-rule baseline for exactly this reason, failures included, alongside the method we use to run live models.

When is it good enough?

An evaluation supports automating a decision when all of these hold:

1. Accuracy in the zone you will automate meets your target, using the lower end of its interval, not the point estimate.
2. The expensive kind of mistake is rare enough that its total cost is acceptable.
3. You have read the failures and none of them is a category you cannot tolerate.
4. Someone has the capacity to review what you don’t automate.
5. You know how you will keep measuring after launch.

Even then, roll out in stages. Run in shadow mode first, where the system decides but nothing acts on it, and compare with what people did. Then use it as a suggestion people approve. Only then let it act alone above a threshold, and keep a random audit sample of what it decides. Each stage produces labeled data for the next.

Related

• Calibrating AI decisions: whether confidence tracks accuracy.
• Confidence thresholds: putting costs on the trade-off.
• Engineering escalation template: the decision used in this guide.