Skip to content
TheThinkLoop
Try the playground

Template - Policy

Predicate

Automate or review

Whether a request is safe to resolve automatically or needs a person.

A predicate decision that acts as a gate in front of any automation: only simple, low-risk requests get through.

Try it in the playgroundCopy the template

The decision

PredicateA probability from 0 to 1 that the condition holds.

Can this request be resolved automatically without a person?

Evidence checks (asked in the same request)

  • Simple request: This is a simple question or routine account change.
  • Legal or chargeback risk: The customer mentions a chargeback, bank dispute, or legal action.
  • Negative sentiment: The customer expresses frustration or dissatisfaction.

Expected output: A probability from 0 to 1 that the request can be resolved automatically, plus one probability per evidence check.

Input

The Decisions API reads one input. Combine these fields into it as labeled text so the model sees the facts the decision depends on.

Input fields
FieldTypeRequiredDescription
messagestringYesThe customer's message, verbatim. Include the subject line if there is one.
available_automationsstring[]YesWhat your automation can actually do, e.g. password resets, invoice copies, order status.

Worked examples

  1. A password reset

    How do I reset my password? I can't find the reset link.

    A careful reviewer would say: Yes. A routine request with a standard answer and no risk.

    Run this example

  2. A disputed charge with a chargeback threat

    I've been charged for a plan I cancelled, this is the second time, and I want a refund or I'll file a chargeback.

    A careful reviewer would say: No. Money, a repeat problem, and a chargeback threat. A person should handle this.

    Run this example

Expected answers are human judgments written for this template, not model output.

How it works

Many automation failures are scoping failures: the bot was fine at password resets and was then pointed at everything. This decision is the scope check, run before any automated reply.

List what your automation can do in the input. "Safe to automate" only has meaning relative to a set of available actions.

The legal-risk and sentiment checks act as vetoes. Whatever the headline probability says, a detected legal threat should send the request to a person.

Human review and thresholds

Recommended review policy

Default to a person. Automate only at high probability with no veto check firing, and let customers reach a person easily when the automated answer misses.

Choosing thresholds

A wrongly automated sensitive request can do real damage, while a wrongly reviewed simple one costs a few minutes. Start with a high bar and lower it one request type at a time as measured accuracy supports it.

Starting automation threshold
92%
Starting don’t-decide threshold
60%

Starting points for the playground, not recommendations. The right values depend on your costs and on how the model’s confidence tracks accuracy on your data. Calibrate them.

Limitations

  • This judges the request, not the quality of the automated answer that follows. You need to evaluate that separately.
  • Requests can look simple and hide risk: "please update my email" from an account that has been taken over.
  • As your automations change, the meaning of "Yes" changes with them. Recalibrate when you add or remove one.

Use it in your code

The template definition is plain JSON: the question, the evidence checks, the input fields, and the starting policy. The SDK call below sends the same question and checks to the Decisions API from your server.

automate-or-review.jsonjson
{
  "$format": "thethinkloop.template/v1",
  "slug": "automate-or-review",
  "title": "Automate or review",
  "decides": "Whether a request is safe to resolve automatically or needs a person.",
  "question": {
    "name": "decision",
    "type": "predicate",
    "instructions": "Can this request be resolved automatically without a person?"
  },
  "evidenceChecks": [
    {
      "name": "factor_1",
      "label": "Simple request",
      "type": "predicate",
      "instructions": "This is a simple question or routine account change."
    },
    {
      "name": "factor_2",
      "label": "Legal or chargeback risk",
      "type": "predicate",
      "instructions": "The customer mentions a chargeback, bank dispute, or legal action."
    },
    {
      "name": "factor_3",
      "label": "Negative sentiment",
      "type": "predicate",
      "instructions": "The customer expresses frustration or dissatisfaction."
    }
  ],
  "inputs": [
    {
      "name": "message",
      "type": "string",
      "required": true,
      "description": "The customer's message, verbatim. Include the subject line if there is one."
    },
    {
      "name": "available_automations",
      "type": "string[]",
      "required": true,
      "description": "What your automation can actually do, e.g. password resets, invoice copies, order status."
    }
  ],
  "expectedOutput": "A probability from 0 to 1 that the request can be resolved automatically, plus one probability per evidence check.",
  "examples": [
    {
      "title": "A password reset",
      "input": "How do I reset my password? I can't find the reset link.",
      "expected": "Yes"
    },
    {
      "title": "A disputed charge with a chargeback threat",
      "input": "I've been charged for a plan I cancelled, this is the second time, and I want a refund or I'll file a chargeback.",
      "expected": "No"
    }
  ],
  "startingPolicy": {
    "automateAt": 0.92,
    "abstainBelow": 0.6,
    "note": "A starting point for exploration, not a recommendation. Calibrate against your own labeled examples."
  }
}
Equivalent SDK calljavascript
import OpenAI from "openai"; // openai >= 7.30.0

const client = new OpenAI(); // reads OPENAI_API_KEY on the server

const decision = await client.decisions.create({
  model: "gpt-6-luna",
  input: "How do I reset my password? I can't find the reset link.",
  questions: [
    {
      type: "predicate",
      name: "decision",
      instructions: "Can this request be resolved automatically without a person?"
    },
    {
      type: "predicate",
      name: "factor_1",
      instructions: "This is a simple question or routine account change."
    },
    {
      type: "predicate",
      name: "factor_2",
      instructions: "The customer mentions a chargeback, bank dispute, or legal action."
    },
    {
      type: "predicate",
      name: "factor_3",
      instructions: "The customer expresses frustration or dissatisfaction."
    }
  ],
});

// Answers come back in question order. A question can also be refused.
const [main, ...checks] = decision.answers;
if (main.type === "refusal") throw new Error("The model declined to answer.");
if (main.type === "predicate") {
  console.log(main.probability); // 0–1 estimate that the condition is true
}

Download it as automate-or-review.json, or browse all developer resources.

Need this decision tuned to your policy?

We are running a small number of pilots with B2B teams: your policy, your examples, your thresholds. Request early access.

Related