Intent classification: detect what a customer message is asking for

What this tool checks

The example is a message that fits two intents at once. The customer was charged twice — a duplicate charge — and one of the charges is still pending, which is its own intent in most banking flows. A classifier that returns a single label hides that. One evaluation asks two questions:

The intents are written in the style of Banking77, a public dataset of banking customer messages, so the published results below are a reasonable reference point. None of them was trained: the line of text next to each intent is the whole definition.

Why probabilities matter for intent detection

A chatbot that routes on the top intent alone treats a 0.95 answer and a 0.45 answer the same way. With a probability for every intent you can do three different things:

• Act automatically when one intent is far ahead — the confidence value is high.
• Ask a clarifying question when two intents are close. "Do you want to cancel the pending charge or dispute the duplicate?" is better than guessing.
• Hand off when other is likely or needs_human is high.

Because Jev can only answer with the options you defined, there is no invented intent name and no output to parse — the routing code reads a key and a number.

How it compares

The only independent measurement we know of is jev-benchmarks, a pre-registered pilot that ran jev-1.13.0 against the zero-shot model GLiNER on 100 held-out examples per dataset. On Banking77, with 72 intents in the sample:

GLiNER ran locally on an Apple M4 Max CPU and Jev was a hosted API called over the network, so the latency column compares two deployments, not two models. It is one 100-message pilot, not a leaderboard, and it did not include a prompted LLM or a classifier trained on Banking77, which would set the bar higher. On JevStation a Choice holds up to 12 options, so a 72-intent set like this one needs the two-stage layout described below.

How to define intents

Describe what the customer wants, not the words they use. "The customer was charged more than once for the same purchase" catches "double charged", "billed twice" and "two payments for one order" without listing phrases.

Separate intents that lead to different actions. If a duplicate charge and a pending payment go to the same flow, merge them. If they go to different flows, keep both and let the probabilities show when a message is about both.

Always include other. Its probability tells you how often real messages fall outside your flows — a direct signal of which intent to add next.

Go two levels deep for large sets. Ask a first Choice for the area — cards, transfers, account, other — then a second, area-specific Choice for the intent. Because one evaluation can ask up to 12 questions, you can send the area question and every area’s intent question together, then read only the intent question for the area that wins.

For more on phrasing, see Designing Questions Jev Can Answer.

From a test to a pipeline

1. Pull a sample from your logs. A few hundred real messages, each labelled with the intent your team would assign, show where the definitions are ambiguous.
2. Run them in one go. The batch page runs one question set over every row of a CSV, no code needed.
3. Choose thresholds. Pick the confidence above which the bot acts alone, and the needs_human probability above which it hands off.
4. Classify live messages through the API. The same evaluation runs through the System One API with an API key, at the same credits as the playground.

Where Jev is the wrong tool

• You also need entity extraction. Jev classifies; it does not pull out the amount, merchant or date. Extract those with code or a generative model.
• The answer depends on arithmetic. "Was I charged more than my limit?" is a comparison of numbers, a weakness TypeSafe documents. Compare in code.
• You already have a large labelled set and stable intents. A classifier trained on your own data will usually beat a zero-shot model on that task.
• Messages must not leave your infrastructure. Jev is a hosted model.

For routing whole support tickets by team and urgency rather than by intent, see the support ticket triage tool.