Abstention: when an AI classifier should not decide

A classifier that must always pick a label will pick one for inputs it cannot classify: a message that is half billing question and half bug report, a request missing the one fact the policy turns on, an input that belongs to none of the categories. Abstention gives it a third option besides right and wrong: no answer, plus a reason. Designed well, it is how an AI decision system stays honest about the cases it should not handle. Designed badly, it is a queue nobody owns.

This guide covers when an LLM classifier should abstain, how to set the threshold from labeled data, and what to do with the cases it hands back.

Abstain, review, refuse: three different outcomes

These get used interchangeably, and keeping them apart makes the system easier to reason about.

In the three-zone policy used across this site, abstention is the bottom zone, labeled “don’t decide”. The research literature calls the same idea a reject option or selective classification: C. K. Chow analyzed the trade-off between errors and rejections for pattern recognition in 1970, and it applies unchanged to LLM classification.

Why a classifier needs a way out

Three kinds of input produce answers that look like decisions but carry little information.

• Ambiguous inputs. The text supports two labels about equally. Any single label is close to a guess, and a confident-looking guess is worse than none.
• Missing information. The policy turns on a fact that isn’t in the input, such as the purchase date for a refund window. The right action is to ask for it.
• Out-of-scope inputs. A spam message sent to a routing decision whose choices are sales, support, and billing. Every available label is wrong.

Low confidence is one signal for all three, but not the only one. The next sections cover confidence first, then the other signals.

The trade-off: coverage against accuracy

Every abstention threshold trades coverage (the share of inputs that get an answer) for accuracy on the inputs that do. Raise the threshold and the answered set shrinks and gets more accurate. Plotted across thresholds, this is a risk–coverage curve.

Read the second row carefully. Moving from 0.50 to 0.70 removes 100 inputs and 50 errors: the inputs it set aside were right only half the time, so the system loses little by not answering them. Moving from 0.90 to 0.95 removes 200 inputs and only 8 errors: those answers were 96% right, and abstaining on them throws away useful work. The shape of the curve, not a rule of thumb, tells you where abstention stops paying for itself.

Setting the abstention threshold

The automation threshold and the abstention threshold answer different questions, so they need different measurements.

• Automation asks: is everything above this line accurate enough to act on without a person? That is a cumulative measure over the whole automated set.
• Abstention asks: are answers at this level worth showing at all? That is a local measure: accuracy within the band just above the line.

A cumulative figure hides weak bands. If answers between 0.50 and 0.60 are right 46% of the time but everything above 0.50 averages 88%, the average says “fine” while the band says “coin flip”. Measure band by band:

// Accuracy within each confidence band: the question for the abstention threshold.
// rows: [{ expected, predicted, confidence }], as in the calibration guide.
function bandAccuracy(rows, edges = [0.5, 0.6, 0.7, 0.8, 0.9, 1.0001]) {
  return edges.slice(0, -1).map((low, i) => {
    const inBand = rows.filter((r) => r.confidence >= low && r.confidence < edges[i + 1]);
    const correct = inBand.filter((r) => r.predicted === r.expected).length;
    return { band: [low, edges[i + 1]], n: inBand.length, accuracy: inBand.length ? correct / inBand.length : null };
  });
}

Compare each band with what you would get without the model. For a balanced yes/no decision the floor is 50%. In general, it is the share of the most common label: if 70% of messages belong to support, a band where the model is right 65% of the time is worse than always answering “support”. A band with only a handful of decisions is too noisy to judge on its own; merge it with a neighbor before deciding.

Abstain on more than confidence

Confidence catches ambiguity. The other two causes are better caught directly.

• Refusals. In the Decisions API any question can come back as { "type": "refusal" }. Route it like an abstention and keep it as a separate reason, so a rise in refusals is visible.
• Missing inputs. Check required fields before asking the model. If the refund window depends on a purchase date you don’t have, no confidence threshold will produce the right answer.
• Out-of-scope inputs. Give a choice an explicit other option, as the feature request classification template does, and treat it as an abstention. Or add an evidence check such as “the message is a customer request” and abstain when it is unlikely.

// Decide whether there is an answer worth using before deciding whether to automate it.
function triage(answer, policy) {
  if (answer.type === "refusal") return { zone: "abstain", reason: "declined" };

  // Support for the selected answer: for a predicate, the probability of the side it chose.
  const support =
    answer.type === "predicate"
      ? Math.max(answer.probability, 1 - answer.probability)
      : answer.confidence;

  if (support >= policy.automateAt) return { zone: "automate" };
  if (support >= policy.abstainBelow) return { zone: "review" };
  return { zone: "abstain", reason: "low support" };
}

What to do with an abstained case

An abstention is only useful if something happens next. Pick the action by the reason.

Give the abstention queue an owner and a response time, like any other queue. The human review guide covers why these cases should not share a queue, or a presentation, with ordinary reviews.

Measure abstention like any other outcome

• Report coverage beside accuracy. “98% accurate” means little without “on the 60% of inputs it answered”.
• Check that abstention catches the hard cases. When abstained cases are later resolved, compare the model’s (unused) answers on them with the outcome. If they were right as often as everything else, abstention is costing coverage and catching nothing.
• Watch the abstention rate. A rising rate is often the earliest sign that inputs have changed: a new product, a new kind of request, a new language. It moves before accuracy does, because accuracy needs labels and the rate does not.

Common mistakes

• Dropping abstentions from the denominator and reporting accuracy as if every input was answered.
• One threshold for every decision. Bands differ by task, and a choice with four labels has a different floor from a yes/no question.
• Setting the threshold on the data you report. Choose it on one set of labeled examples and confirm it on another.
• Never revisiting it. A model, prompt, or policy change can move every band. Re-measure after each one.