Human-in-the-loop AI: when AI should decide and when it should ask a person

“Human in the loop” is often treated as a safety feature you switch on. In practice it is a queue with a staffing cost, a design that can fail quietly, and the best source of training signal you will ever have. This guide covers when a person should be involved, how to design the review step, and how to tell whether it is working.

When should AI decide, and when should it ask?

Confidence is one input. The nature of the decision matters at least as much. A decision is a better candidate for automation the more of these are true:

Notice that most of the column on the right is about the decision, not the model. A model can be very accurate at something you still shouldn’t automate, and a mediocre model can safely sort a queue.

Three review patterns

1. Gate: a person approves before anything happens

The model recommends and a person acts. This is the right starting point for any decision that moves money or changes a customer’s access. It saves less time than full automation, but it produces a labeled example for every decision, which is exactly what you need to justify automating later.

2. Exception queue: automate the confident, review the rest

Decisions above your automation threshold proceed; the rest go to a person. This is the three-zone policy, and it is where most production systems settle. Its weakness is that automated decisions are never seen, so their errors are invisible unless you add the third pattern.

3. Audit: sample what was automated

Take a random sample of automated decisions, perhaps 2 to 5 percent, and have someone review them after the fact. This is how you find out that accuracy has slipped. Without an audit, an exception queue tells you only about the cases the model was unsure of, which are not the ones that hurt.

“Review” and “don’t decide” are different

A case in the review zone has a plausible answer that needs confirming. A case below it has no usable answer. Sending both to the same queue with the same presentation wastes the reviewer’s time on the second kind, because what that case needs is usually more information: a question back to the customer, or a lookup in another system. The abstention guide covers setting that lower threshold and handling the cases below it.

What the reviewer should see

A reviewer needs enough to make the call quickly and independently:

• The original input, in full. Summaries hide the detail that makes a case unusual.
• The specific question, including the policy it refers to.
• The evidence checks. “Within refund window: unclear” tells a reviewer where to look. A bare 71% tells them nothing.
• Why it reached them: below threshold, refused by the model, or sampled for audit. These call for different levels of scrutiny.

The Decisions API returns estimates rather than reasoning, so the evidence checks are something you design. Asking several questions against the same input is how you get them.

The rubber-stamp problem

People shown a recommendation tend to agree with it, especially when they are busy and the recommendation is usually right. A review step where the reviewer approves 99% of suggestions in four seconds each is not providing oversight. It is providing the appearance of oversight, which is worse than nothing because everyone believes the decisions are being checked.

Ways to keep review real:

• Hide the suggestion on a sample. For a fraction of cases, ask the reviewer to decide before revealing the model’s answer. Disagreement on that sample is your true error estimate.
• Track the override rate. If it is near zero, either the model is excellent or nobody is looking. Blind samples tell you which.
• Track time per review. A sudden drop usually means the queue outgrew the team.
• Make disagreeing easy. If overriding takes three clicks and a written reason while approving takes one, you have built a bias into the tool.

Size the queue before you launch

Review volume is decision volume multiplied by the share that falls outside the automate zone. At 20,000 decisions a month with 30% sent to review, that is 6,000 reviews. At two minutes each, it is 200 hours, more than one full-time person. If you don’t have that capacity, the choices are a lower automation threshold, which means more errors, or a narrower rollout. Decide deliberately. Otherwise the queue decides for you by backing up.

Close the loop

Every reviewed decision is a labeled example: the input, the model’s answer, its confidence, and what a person decided. Store those. They let you re-measure accuracy monthly, notice drift, and move thresholds with evidence. Review that doesn’t feed back into measurement is only a cost.

Related

• Automate or review template: a gate in front of any automation.
• Policy exception template: a decision that should usually keep a person involved.
• Calibrating AI decisions: measuring what your review data tells you.