Benchmarks
How we measure AI decisions, in the open
Labeled datasets, a fixed method, and every failure published next to the successes. A number without a method is marketing; we would rather show you the method first.
What is published today
| Component | Status | Detail |
|---|---|---|
| Datasets | Published | 4 datasets, 102 labeled cases, 31 of them deliberately hard. |
| Keyword-rule baseline | Published | Measured below. Deterministic, recomputed on every build. |
| OpenAI Decisions API | Not yet measured | The harness supports it. We publish results only after a complete run, with repeats, against the frozen datasets. |
| Structured Outputs comparison | Planned | Same datasets and questions, a general-purpose model with a strict schema and log probabilities. |
No model accuracy figures yet
Method
Each dataset is one decision taken from our template library, asked with the template’s exact question wording. Every system under test gets the same cases and the same question, with no per-system prompt tuning. Datasets are versioned and frozen once published; corrections bump the version and are logged.
Cases and labels
Cases are synthetic customer messages written to resemble B2B SaaS support traffic. No customer data is used. Each dataset has a written labeling guide, and every case has one expected answer. About a third of cases are marked hard: a misleading tone, a decisive detail buried in the message, or a request that looks like one category and is another. Hard and typical cases are reported separately.
What we report
- Accuracy, with a 95% Wilson interval, overall and by difficulty.
- The full confusion matrix: which answers are mistaken for which.
- Calibration: stated confidence against observed accuracy, in bands.
- Refusals and failed calls, counted as wrong rather than dropped.
- Consistency: the share of cases that get the same answer on a repeat call.
- Latency (median and 95th percentile) for networked providers only.
- Every failure, with the input, the expected answer, and what the system said.
What we won’t do
- Report a single headline number without its interval and sample size.
- Tune a system on these cases and then report its score on them.
- Present results from 100 cases as if they predict accuracy on your traffic.
Baseline results: keyword rules
The baseline is the deterministic rule set behind the playground’s demo mode: weighted keyword patterns, no model. It is the kind of system many teams already run. Its rules were written for the template examples before these datasets existed and have not been adjusted to them. Its “confidence” is a squashed rule score, not a probability, which is why its calibration is poor.
| Dataset | Correct | Accuracy | 95% interval | Typical | Hard |
|---|---|---|---|---|---|
| Refund eligibility | 21/26 | 81% | 62%–91% | 83% | 75% |
| Team routing | 20/28 | 71% | 53%–85% | 80% | 50% |
| Ticket urgency | 12/24 | 50% | 31%–69% | 56% | 33% |
| Engineering escalation | 19/24 | 79% | 60%–91% | 87% | 67% |
Confident and wrong
3 failures carry 90% confidence or more. For example, refund eligibility case refund-017 was answered “Yes” at 96% when the expected answer was “No”. This is why a confidence threshold is not a substitute for checking confidence against labeled examples.
Refund eligibility
Predicate- Accuracy
- 21/26
- 95% interval
- 62%–91%
- Typical cases
- 15/18
- Hard cases
- 6/8
Confusion matrix, calibration, and all 5 failures
| Expected / Predicted | No | Yes |
|---|---|---|
| No | 11 | 2 (errors) |
| Yes | 3 (errors) | 10 |
| Confidence band | Decisions | Mean confidence | Observed accuracy | Reading |
|---|---|---|---|---|
| 0.9–1.0 | 6 | 94% | 83% | Overconfident |
| 0.7–0.8 | 7 | 71% | 100% | Underconfident |
| 0.6–0.7 | 13 | 64% | 69% | Close to calibrated |
Failures, most confident first
- refund-017Hard case
“You already refunded me for the duplicate charge last week, thanks! But I still see a pending charge of $49 on my statement. Can you refund that too?”
Expected No, baseline said Yes with 96% confidence.
Most likely the same charge whose reversal is still pending. Approving would refund twice; a person should verify.
- refund-005
“Your checkout charged my card three times when the page froze. I only want one subscription. Refund the extra two, please.”
Expected Yes, baseline said No with 65% confidence.
- refund-019Hard case
“Great product honestly, no complaints. I just bought a second subscription by accident on my work account, 3 days after the first. Can the second one be refunded?”
Expected Yes, baseline said No with 65% confidence.
Positive sentiment and 'no complaints' sit alongside a valid accidental duplicate purchase.
- refund-025
“I meant to buy the monthly plan but clicked annual by mistake ten minutes ago. Can you switch me and refund the difference?”
Expected Yes, baseline said No with 65% confidence.
- refund-011
“We renewed our annual subscription in January. It's now May and we've decided to cancel. Please refund the unused portion.”
Expected No, baseline said Yes with 62% confidence.
Team routing
Choice- Accuracy
- 20/28
- 95% interval
- 53%–85%
- Typical cases
- 16/20
- Hard cases
- 4/8
Confusion matrix, calibration, and all 8 failures
| Expected / Predicted | billing | customer_success | sales | support |
|---|---|---|---|---|
| billing | 5 | 0 | 0 | 2 (errors) |
| customer_success | 1 (errors) | 3 | 0 | 3 (errors) |
| sales | 0 | 0 | 6 | 0 |
| support | 2 (errors) | 0 | 0 | 6 |
| Confidence band | Decisions | Mean confidence | Observed accuracy | Reading |
|---|---|---|---|---|
| 0.9–1.0 | 7 | 92% | 71% | Overconfident |
| 0.8–0.9 | 9 | 81% | 100% | Underconfident |
| 0.7–0.8 | 3 | 75% | 33% | Too few to judge |
| 0.6–0.7 | 1 | 65% | 100% | Too few to judge |
| 0.5–0.6 | 2 | 60% | 50% | Too few to judge |
| 0.3–0.4 | 6 | 31% | 50% | Underconfident |
Failures, most confident first
- route-015Hard case
“Our invoice PDF won't download. The button just spins forever.”
Expected support, baseline said billing with 93% confidence.
Mentions an invoice, but the problem is a broken download.
- route-022Hard case
“We'd like to cancel our subscription effective at the end of the term and get a final invoice.”
Expected customer_success, baseline said billing with 93% confidence.
Under this labeling guide, cancellations go to customer success for a save attempt even though an invoice is mentioned.
- route-016Hard case
“We were charged for 50 seats, but the admin panel only lets us assign 35. It says 'seat limit reached'. Is this a bug?”
Expected support, baseline said billing with 78% confidence.
The charge matches the purchase; the product is blocking seat assignment.
- route-021
“Why did our bill go up from $400 to $520 this month? We didn't add anyone.”
Expected billing, baseline said support with 73% confidence.
- route-010Hard case
“Honestly we're not getting much value. Half the team stopped logging in. Before we decide whether to renew, can someone help us figure out what we're doing wrong?”
Expected customer_success, baseline said support with 60% confidence.
Frustrated tone, but the request is about adoption and renewal.
- route-009
“Could we schedule a quarterly business review? Our new VP wants to see usage across teams.”
Expected customer_success, baseline said support with 31% confidence.
- route-026
“Can we get copies of the receipts for all payments made in 2025?”
Expected billing, baseline said support with 31% confidence.
- route-027
“New admin here. I just took over our workspace from a colleague who left. Is there someone who can get me up to speed on how our team uses the product?”
Expected customer_success, baseline said support with 31% confidence.
Ticket urgency
Score- Accuracy
- 12/24
- 95% interval
- 31%–69%
- Typical cases
- 10/18
- Hard cases
- 2/6
Within one level of the expected level: 75%.
Confusion matrix, calibration, and all 12 failures
| Expected / Predicted | Critical | High | Low | Normal |
|---|---|---|---|---|
| Critical | 1 | 0 | 0 | 4 (errors) |
| High | 0 | 1 | 1 (errors) | 4 (errors) |
| Low | 0 | 1 (errors) | 3 | 2 (errors) |
| Normal | 0 | 0 | 0 | 7 |
| Confidence band | Decisions | Mean confidence | Observed accuracy | Reading |
|---|---|---|---|---|
| 0.7–0.8 | 4 | 74% | 25% | Too few to judge |
| 0.6–0.7 | 4 | 64% | 100% | Too few to judge |
| 0.5–0.6 | 16 | 58% | 44% | Overconfident |
Failures, most confident first
- urgency-018Hard case
“Hi, sorry to bother you. I think our nightly data sync may not have run for a few days? Not sure if it matters; our reports are just showing last week's numbers. Whenever you have time.”
Expected High, baseline said Low with 78% confidence.
A polite tone hides a multi-day data pipeline failure.
- urgency-020Hard case
“Just a heads-up: your status page says all systems are operational, but our API calls have been returning 503 for about 20 minutes. Our checkout integration depends on it.”
Expected Critical, baseline said Normal with 72% confidence.
Casual framing of a revenue-affecting outage.
- urgency-017Hard case
“URGENT!!! The logo on our login page is slightly blurry on retina screens. Please fix ASAP.”
Expected Low, baseline said High with 70% confidence.
Urgent language, cosmetic impact.
- urgency-003
“Is there a keyboard shortcut for creating a new project?”
Expected Low, baseline said Normal with 59% confidence.
- urgency-004
“Typo on the billing page: 'recieve' should be 'receive'.”
Expected Low, baseline said Normal with 59% confidence.
- urgency-010
“None of our sales reps can log their calls since this morning; the Save button errors out. They're using a spreadsheet meanwhile, but it's a mess.”
Expected High, baseline said Normal with 59% confidence.
- urgency-011
“We go live with our customers on Monday and webhook signatures are failing validation in our staging environment.”
Expected High, baseline said Normal with 59% confidence.
- urgency-014
“Customers are reporting that payments on our storefront have been failing since your last release. We're losing orders every minute.”
Expected Critical, baseline said Normal with 59% confidence.
- urgency-015
“We just noticed that records created in the last 6 hours are missing from every project. Did something delete our data?”
Expected Critical, baseline said Normal with 59% confidence.
- urgency-016
“Security issue: one of our users can see another company's invoices in their dashboard.”
Expected Critical, baseline said Normal with 59% confidence.
- urgency-022Hard case
“We're presenting your product to our board in 2 hours and the demo workspace won't load.”
Expected High, baseline said Normal with 59% confidence.
One workspace, but an immediate deadline.
- urgency-012
“SSO login is failing for our whole support team, about 15 agents. They can use password login as a fallback, but most don't remember their passwords.”
Expected High, baseline said Normal with 54% confidence.
Engineering escalation
Predicate- Accuracy
- 19/24
- 95% interval
- 60%–91%
- Typical cases
- 13/15
- Hard cases
- 6/9
Confusion matrix, calibration, and all 5 failures
| Expected / Predicted | No | Yes |
|---|---|---|
| No | 12 | 1 (errors) |
| Yes | 4 (errors) | 7 |
| Confidence band | Decisions | Mean confidence | Observed accuracy | Reading |
|---|---|---|---|---|
| 0.9–1.0 | 5 | 92% | 100% | Underconfident |
| 0.8–0.9 | 1 | 87% | 100% | Too few to judge |
| 0.7–0.8 | 17 | 76% | 76% | Close to calibrated |
| 0.5–0.6 | 1 | 57% | 0% | Too few to judge |
Failures, most confident first
- eng-017Hard case
“I get a 403 Forbidden when opening the Admin settings page.”
Expected No, baseline said Yes with 79% confidence.
Usually a role permission. Support can check the user's role before anyone escalates.
- eng-005
“Our scheduled report has run twice every night this week, emailing customers duplicate summaries.”
Expected Yes, baseline said No with 75% confidence.
- eng-006
“Dates in exported CSVs are shifted by one day for users in time zones west of UTC.”
Expected Yes, baseline said No with 75% confidence.
- eng-007Hard case
“We deleted a project by accident 3 days ago. Is there any way to restore it from a backup?”
Expected Yes, baseline said No with 75% confidence.
Restoring from backup needs engineering access in most organisations. If support has a restore tool, this becomes No.
- eng-021Hard case
“Hey team, quick one: the 'Forgot password' link on your login page goes to a 404 page.”
Expected Yes, baseline said No with 57% confidence.
Casual tone, real bug.
Reproduce it
The harness sends every case through the same provider boundary as the playground and writes a run file with each answer, its confidence, and timing. Datasets are plain JSON in a documented format, so you can also run them through your own stack.
# Keyword baseline: deterministic, free, no credentials
npm run benchmark -- --dataset refund-eligibility
# Live Decisions API run (spends credits; needs DECISION_PROVIDER=openai and OPENAI_API_KEY)
DECISION_PROVIDER=openai npm run benchmark -- --dataset all --repeats 2 --saveLimitations
- The datasets are small. With 24 to 28 cases each, intervals are wide; differences of a few cases between systems are not meaningful.
- The cases are synthetic and written by one team. Real traffic is messier, longer, and distributed differently, so treat any score here as a comparison between systems, not a forecast for your inbox.
- Labels come from a single annotator. We have not yet measured agreement between labelers.
- The questions do not spell out every policy detail. Some hard cases depend on the labeling guide’s reading of the policy, which a system can only infer.
Measure your own decision
A public benchmark tells you how systems compare. Only your own labeled examples tell you where to set thresholds.