Skip to content
TheThinkLoop
Try the playground

Template - Support

Choice

Feature request classification

Whether a message is a bug report, a feature request, a how-to question, or something else.

A choice decision that sorts inbound product feedback into the categories product and support teams handle differently.

Try it in the playgroundCopy the template

The decision

ChoiceOne of the listed values, a confidence value, and a probability for every option.

What kind of message is this?

Choices

  1. •bug: Something that should work does not: an error, crash, or regression.
  2. •feature_request: A request for a new capability or an enhancement.
  3. •how_to_question: A question about how to use an existing feature.
  4. •other: Anything that fits none of the above.

Evidence checks (asked in the same request)

  • Technical failure: The message reports an error, crash, or failed request.

Expected output: One of the four values, a confidence value, and a probability for every option, plus one probability per evidence check.

Input

The Decisions API reads one input. Combine these fields into it as labeled text so the model sees the facts the decision depends on.

Input fields
FieldTypeRequiredDescription
messagestringYesThe customer's message, verbatim. Include the subject line if there is one.

Worked examples

  1. Asking for a capability that doesn't exist

    It would be great if the CSV export supported custom date ranges. Any plans to add that?

    A careful reviewer would say: feature_request. The customer wants something the product doesn't do yet.

    Run this example

  2. Something that used to work

    The export button returns an error message since Tuesday. It worked fine last week.

    A careful reviewer would say: bug. A regression: existing functionality now fails.

    Run this example

Expected answers are human judgments written for this template, not model output.

How it works

Customers rarely label their own messages correctly. "Feature request: the export is broken" is a bug, and "Bug: there's no dark mode" is a feature request. The option descriptions carry the definitions, so write them carefully.

Always include a catch-all option. Without one, the model must pick a category for messages that fit none, and it will do so with misleading confidence.

The per-option probabilities show close calls. A message split evenly between bug and feature request often describes behavior that is working as designed but surprising, which is worth knowing.

Human review and thresholds

Recommended review policy

Tagging is low-risk and can be automated broadly. If a category triggers real work, such as bugs opening engineering tickets, require a higher bar or confirmation for that category.

Choosing thresholds

For tagging and analytics, a moderate bar is fine because a single wrong tag costs little. For routing that creates work for another team, treat each category as its own decision with its own bar.

Starting automation threshold
75%
Starting don’t-decide threshold
50%

Starting points for the playground, not recommendations. The right values depend on your costs and on how the model’s confidence tracks accuracy on your data. Calibrate them.

Limitations

  • Messages that contain both a bug and a request get one label. If that is common for you, ask two predicates instead of one choice.
  • The boundary between "missing feature" and "bug" depends on what your product promises, which the model doesn't know unless you tell it.
  • Category counts from this decision are estimates. Don't report them as exact without measuring accuracy first.

Use it in your code

The template definition is plain JSON: the question, the evidence checks, the input fields, and the starting policy. The SDK call below sends the same question and checks to the Decisions API from your server.

feature-request-classification.jsonjson
{
  "$format": "thethinkloop.template/v1",
  "slug": "feature-request-classification",
  "title": "Feature request classification",
  "decides": "Whether a message is a bug report, a feature request, a how-to question, or something else.",
  "question": {
    "name": "decision",
    "type": "choice",
    "instructions": "What kind of message is this?",
    "choices": [
      {
        "value": "bug",
        "description": "Something that should work does not: an error, crash, or regression."
      },
      {
        "value": "feature_request",
        "description": "A request for a new capability or an enhancement."
      },
      {
        "value": "how_to_question",
        "description": "A question about how to use an existing feature."
      },
      {
        "value": "other",
        "description": "Anything that fits none of the above."
      }
    ]
  },
  "evidenceChecks": [
    {
      "name": "factor_1",
      "label": "Technical failure",
      "type": "predicate",
      "instructions": "The message reports an error, crash, or failed request."
    }
  ],
  "inputs": [
    {
      "name": "message",
      "type": "string",
      "required": true,
      "description": "The customer's message, verbatim. Include the subject line if there is one."
    }
  ],
  "expectedOutput": "One of the four values, a confidence value, and a probability for every option, plus one probability per evidence check.",
  "examples": [
    {
      "title": "Asking for a capability that doesn't exist",
      "input": "It would be great if the CSV export supported custom date ranges. Any plans to add that?",
      "expected": "feature_request"
    },
    {
      "title": "Something that used to work",
      "input": "The export button returns an error message since Tuesday. It worked fine last week.",
      "expected": "bug"
    }
  ],
  "startingPolicy": {
    "automateAt": 0.75,
    "abstainBelow": 0.5,
    "note": "A starting point for exploration, not a recommendation. Calibrate against your own labeled examples."
  }
}
Equivalent SDK calljavascript
import OpenAI from "openai"; // openai >= 7.30.0

const client = new OpenAI(); // reads OPENAI_API_KEY on the server

const decision = await client.decisions.create({
  model: "gpt-6-luna",
  input: "It would be great if the CSV export supported custom date ranges. Any plans to add that?",
  questions: [
    {
      type: "choice",
      name: "decision",
      instructions: "What kind of message is this?",
      choices: [
        {
          value: "bug",
          description: "Something that should work does not: an error, crash, or regression."
        },
        {
          value: "feature_request",
          description: "A request for a new capability or an enhancement."
        },
        {
          value: "how_to_question",
          description: "A question about how to use an existing feature."
        },
        {
          value: "other",
          description: "Anything that fits none of the above."
        }
      ]
    },
    {
      type: "predicate",
      name: "factor_1",
      instructions: "The message reports an error, crash, or failed request."
    }
  ],
});

// Answers come back in question order. A question can also be refused.
const [main, ...checks] = decision.answers;
if (main.type === "refusal") throw new Error("The model declined to answer.");
if (main.type === "choice") {
  console.log(main.choice, main.confidence, main.probabilities);
}

Download it as feature-request-classification.json, or browse all developer resources.

Need this decision tuned to your policy?

We are running a small number of pilots with B2B teams: your policy, your examples, your thresholds. Request early access.

Related