Jev vs an LLM classifier: when a typed decision beats a prompt

At a glance

The comparison here is between Jev and a general LLM prompted to pick a label, with or without structured output. For zero-shot classifiers and models you train yourself, see Jev vs open models and the overview in Jev Alternatives.

The evaluation rows come from TypeSafe’s four published workflows as reported by a third-party write-up. They are vendor numbers, and the reference answers in them are the average of two frontier LLMs, not human labels. Prices checked 2026-09-19.

Cost

Jev bills input tokens only, and a classification call is small. We measured real evaluations at $0.000012 to $0.000377, with a typical three-question call at about $0.00002 (the measurement). The LLM side depends on the model, and output tokens are billed there too.

Two independent reports point the same way. The CTO of Bryo AI told TechCrunch that in his test of business-email classification Gemini was slightly more accurate but 10 to 20 times more expensive. A DevelopersIO router test measured $0.000025–$0.000027 per Jev call.

Batching widens the gap. Jev answers every question in a request in parallel, and TypeSafe’s docs say 13 questions in one call is 11.5x cheaper and 9.6x faster than 13 separate calls. With an LLM, asking more questions usually means more output tokens or more calls.

Latency

TypeSafe claims 70–500 ms end to end, against a 3–329 second range for LLMs. That is vendor-stated. Independent numbers are smaller and more useful:

• Model routing: DevelopersIO measured a median of 0.643–0.674 s per Jev call, about 3x faster than a Gemini 3.5 Flash classifier and 10–11x faster than a DeepSeek V4 Flash one. The author notes it called TypeSafe standalone, with one sample per tier.
• Code monitoring: a LessWrong AI-control pilot recorded a median latency of 0.34 s.
• Command safety review: TechCrunch reports that Vercel got results 5 to 18 times faster, and more accurate, after replacing an OpenAI Luna 5.6 classifier with Jev.

Output shape: typed values vs parsed text

An LLM with structured output still generates text; the schema constrains the shape, and your code parses and validates it. Jev does not generate text. You declare the options, and the answer is one of them, with a probability for each. It cannot return a label you did not list.

TypeSafe says Jev makes no type errors and is candid that this is guaranteed by schema matching, not measured. The consequence shows up in unrelated tests: on the LLM Chess leaderboard, Jev completed every game with perfect protocol compliance. The caveat is equally plain — a typed answer can still be the wrong answer.

Accuracy: what has been measured

No independent, human-labelled head-to-head between Jev and an LLM classifier exists in our sources. What does exist:

• TypeSafe’s workflow evals (vendor): Jev 67.8%, GPT-5.6 Terra 67.9%, GPT-5.6 Sol 74.1%. TypeSafe itself warns these were built by its own team, that the reference answers favour OpenAI and Anthropic models, and that the LLMs ran through its wrapper.
• An independent re-scoring by jev-on-a-laptop of 343 question pairs from TypeSafe’s public examples: Opus 89.8%, Sol 89.2%, Jev 86.6%. The author’s reading is that Jev sits in the frontier cluster at a fraction of the price.
• Anecdotes: Bryo AI found Gemini slightly more accurate; Vercel found Jev more accurate than the classifier it replaced.

Read together: parity, give or take a few points, depending on the task. Test it on your own labelled data before relying on either.

Calibration and consistency

This is the part an LLM cannot easily match. Jev is trained to return probabilities that mean what they say across groups of predictions, and TypeSafe describes LLMs as overconfident and inconsistent (a vendor claim). The independent evidence is mixed, and worth knowing:

• Consistency is high. In the LessWrong pilot, re-scoring the same code moved scores by 0.008 on average, and 92% of the flagged set stayed the same across calls.
• Calibration varies by task. A pre-registered study found Jev well calibrated on two datasets and badly on a third (details in Jev vs open models).
• Jev can be under-confident. The LessWrong author called the raw score a great ranking and a poor probability.
• Equivalent questions disagree. TypeSafe documents that a Noul and the equivalent yes/no Choice can give different probabilities, and that a question and its negation can sum to more than 1. Tune thresholds on the exact questions you ship.

When an LLM is the better choice

• You need the reasoning written down, for a reviewer, an audit trail or a regulated decision.
• The decision involves counting, arithmetic or comparing dates. TypeSafe’s jaggedness notes say Jev is not a calculator.
• You cannot list the possible answers in advance, or you need text back.
• Your inputs are mostly not English; TypeSafe says other languages, including CJK scripts, are handled but not as well.
• A few points of accuracy matter more than cost and latency.

When Jev is the better choice

• The answer space is known and the same decision repeats at volume.
• You want to act on confidence: automatically above one threshold, by escalation below it.
• Latency matters, as in a pre-execution check or a live support queue.
• You ask several questions about the same input and want them answered in one call.

Using both

The pattern most of the evidence points to is a split, not a swap. Let the LLM extract or draft, and let Jev verify with a typed yes/no and a probability. Or put Jev first: act on the answers it is confident about, and send the middle band to an LLM or a person. Jev as a judge and content moderation show both shapes, and you can try your own questions in the playground.