Zero-shot classification: label text with categories you define

What this tool checks

The example looks like a boundary case: a company’s share price and revenue forecast, which is business, but the company makes processors, which could pull towards science and technology. With business defined as companies, markets and earnings, Jev puts it firmly in business. Rewrite the text to be about the processor itself and watch how the distribution moves. One evaluation asks three questions:

The four topics are the label set of AG News, a standard topic-classification benchmark, so you can set what you see here against the published numbers below. Nothing was trained for them: the only thing that defines each label is the line of text next to it. Change those lines and the classifier changes with them.

How zero-shot classification works here

Most zero-shot classifiers turn the task into something the model already knows how to do. The classic approach, used by Hugging Face’s zero-shot-classification pipeline, turns each label into a hypothesis — by default This example is {}. — and scores it with a natural-language-inference model. Prompting a general LLM to "answer with one of these labels" is the other common route.

Jev takes the labels directly. A Choice question is a list of options with criteria, and the answer comes back as a probability for every option, not as text you have to match against your label names. Two consequences matter in practice:

• It cannot answer outside your labels. There is no "Business/Tech" invented on the fly and no parsing step to fail.
• You get a distribution, not just a winner. A text at 0.55 business and 0.40 sci_tech is a different decision from one at 0.97 business. The confidence value condenses that shape into one number you can threshold.

How it compares with other zero-shot classifiers

The only independent measurement we know of is jev-benchmarks, a pre-registered pilot that ran jev-1.13.0 against GLiNER (fastino/gliner2.5-multi-v1) on 100 held-out examples from each of three datasets:

Coverage is the share of texts you could act on automatically while keeping errors at or below 5% — closer to how a classifier is used than raw accuracy. Jev is clearly ahead on news topics and banking intents. On emotions neither model is usable, and Jev was the worse calibrated of the two. It is a 300-example pilot, not a leaderboard, and it did not include NLI models such as bart-large-mnli or a prompted LLM. Jev Alternatives covers the trade-offs in more depth.

How to write the labels

Define every label, not just name it. "Business" means different things to an editor and an accountant. A line of criteria — "Companies, markets, earnings and the economy" — is where your definition of the task lives.

Make the boundary explicit. If your team would file a chipmaker’s earnings under business, say so in the criteria: "Company results belong here even when the company is a technology firm."

Add an escape option. Real data contains texts that fit no label. An other option keeps them from being forced into the nearest one, and its probability tells you how often your label set misses.

Use Nouls for tags that overlap. If a text can be about pricing and about bugs at the same time, that is multi-label classification. Ask one Noul per tag instead of one Choice.

For more on phrasing, see Designing Questions Jev Can Answer.

From a test to a pipeline

1. Label a sample first. A hundred or so texts your team has already classified are enough to see where Jev agrees and where your label definitions are ambiguous.
2. Classify a backlog without code. The batch page runs one question set over every row of a CSV.
3. Pick a threshold from the sample. Act automatically on answers above it and send the rest to a person — that split is what the coverage column above measures.
4. Classify new texts through the API. The same evaluation runs through the System One API with an API key, at the same credits as the playground.

Where Jev is the wrong tool

• Fine-grained emotion. That was the failure case in the benchmark. Test it hardest, or train a model on your own labels.
• The text must stay on your machines. Jev is a hosted model. GLiNER or an NLI model can run locally.
• You have a large, stable label set and plenty of labelled data. A classifier trained on your data will usually beat any zero-shot model on its own task, at a lower cost per call.
• You need the reasoning written down. Jev returns typed answers, not explanations.