AI decision models compared: run OpenAI, Perplexity, TypeSafe and Cloudflare side by side

The five models at a glance

Every row says where its numbers come from: (vendor) is the company selling the model, (ours) is a call we made on 2026-10-07.

Credits are standard / large: an evaluation is large above 8,000 characters of state or five questions. Our latencies are round trips from one machine to OpenRouter, so compare them with each other, not with vendor figures.

These are not the only decision models. OpenRouter listed 14 with a “decisions” output on the day we tested, including Liquid d1, Upstage Solar Decide, Inception Mercury Decide and Together’s experimental Tev1. We cover the five with the most behind them.

What we measured

Same inputs, same questions, same endpoint (OpenRouter’s /api/v1/systemone), five models, two full runs that agreed with each other. Every question was zero-shot: one line of plain description per option, nothing tuned.

• Banking77, card intents. 96 messages from the Banking77 test set across 12 intents that are easy to confuse: card arrival vs delivery estimate, lost vs compromised, declined vs not recognised. One 12-option Choice question.
• AG News. 100 headlines and leads from the AG News test set, 25 per topic. One four-option Choice question.
• Long state. One sentence asking to cancel a subscription, hidden at the start, 6,000 characters in, 12,000 characters in, or at the very end of 20,000 characters of neutral text, plus a control with no such sentence. One Noul question.

What stands out:

• Clef-flash, the smallest model, was the most accurate on fine-grained intents. That matches Cloudflare’s own Banking77 claim, which until now nobody outside Cloudflare had checked.
• The Clef models are the most cautious. They rarely go above 0.99 and were never wrong when they did. Luna Decisions and Jev often answer with a confidence of 1.00, and that is where their mistakes cluster: six of Luna’s errors and four of Jev’s came at 0.99 or above. If you plan to auto-act above a threshold, set it on your own labelled data per model.
• OpenAI refused one input. The plain banking question “How do I freeze my account?” came back as an error, “OpenAI refused to answer”, in both runs. Plan for refusals as a failure mode; on JevStation the credit is refunded.
• All five read the whole 20,000-character state. Every model found the cancel request even at the very end. On 2026-10-04 we had measured Clef on Cloudflare Workers AI reading only about the first 2,048 tokens. On 2026-10-07 it read everything, both through OpenRouter and when we retested Workers AI directly, so that cap is gone.

With about 100 examples per task, a difference smaller than roughly six points could be noise. Only Clef-flash’s lead on Banking77 is clearly outside it. This is a small sample, not a leaderboard.

The cost catch: Perplexity Decider bills the state once per question

Decider has the lowest list price, and on a single question it was the cheapest of the five models we ran on 2026-10-07: $0.007 per thousand AG News calls. (Microsoft-Decision-1, added on 2026-10-10, came in lower still at $0.005.) But its token count grows with the number of questions, because it reads the whole state again for each one. On 8,000 characters of text, one question used 1,392 input tokens, three used 4,192 and five used 6,985. Luna Decisions, for comparison, used about 2,000 for five questions on the same text.

So the cheapest model depends on the shape of the request. Use the calculator above with your own numbers. JevStation charges Decider 2 credits for that reason; the other one-pass models cost 1.

How to choose

• Fixed intent lists, many similar labels: start with Clef-flash. It led our Banking77 run and costs 1 credit here.
• One question per call at high volume: Microsoft-Decision-1 or Perplexity Decider, the two cheapest per call in that shape.
• Several questions on long text: Jev or Luna Decisions, which read the state once.
• You want the OpenAI model without waiting for preview access: Luna Decisions through OpenRouter or here. Expect occasional refusals.
• The data cannot leave your machines: Clef, Clef-flash or Decider, all open-weight under Apache 2.0.
• Cheapest per call at the Jev price, with strong confidence: Microsoft-Decision-1, added on 2026-10-10. It matched Jev on accuracy, used fewer tokens and was better calibrated; see Jev vs Microsoft-Decision-1. Azure rate-limits it more tightly.
• You need a confidence you can threshold safely: test calibration on your own data; in our run the Clef models were the most conservative.

The deciding test is still your data. Paste twenty real inputs into the side-by-side runner above, run two models at once and compare the answers with what you know is right.

More comparisons

• Jev vs Perplexity Decider: the two cheapest models, head to head.
• Jev vs Microsoft-Decision-1: Microsoft’s model, tested the day after launch.
• OpenAI Decisions API alternatives: what to use while the preview is closed, and how to port a request.
• Jev vs Clef: Cloudflare’s models, including the Workers AI truncation we measured at launch and the retest that found it gone.
• Decision model pricing calculator: monthly cost for all five on your workload.

Sources

• OpenAI Decisions API preview, DevDay 2026-09-29. Coverage on Hugging Face and r/OpenAI (independent)
• Perplexity Decider model, price and context. OpenRouter model page and Perplexity Decisions API docs (vendor)
• System One endpoint and request format. OpenRouter TypeSafe SDK guide (vendor)
• Clef. Cloudflare blog (vendor) and Cloudflare Clef, tested (ours)
• Datasets. Banking77 and AG News test splits
• Accuracy, calibration, latency, token counts and long-state test. Our runs on 2026-10-07 (ours)

JevStation is independent. Jev, Clef, Decider and GPT-6 Luna are names of their respective owners, TypeSafe, Cloudflare, Perplexity and OpenAI; we are not affiliated with any of them.