Which AI model classifies your data best?
Public benchmarks rarely match your task. Paste or upload labeled examples, run them through up to 8 decision models at once, and compare accuracy, calibration, latency and cost side by side — then get a recommended model, a starting confidence threshold and the code to call it.
How it works
- 1
Bring labeled examples
Paste rows straight from a spreadsheet, upload CSV or JSONL, or start from a sample dataset. You need a text column and a label column.
- 2
Define each label
Each distinct label becomes an option. One plain-English line per option is all the model sees — no training.
- 3
Pick models and run
See the credit estimate before you start. Each model runs in parallel; failed rows are refunded.
- 4
Ship the winner
Get the recommended model and threshold for your target error rate, the rows models get wrong, and a ready-to-run API call.
What you can run
- Your first evaluation after signing up is free: up to 100 rows × 4 models.
- After that, up to 500 rows × all 8 models, paid in credits — 200 rows on every model costs about 2,200 credits.
Example reports
Already run on public datasets, so you can see a full report before bringing your own data.
Run an evaluation
1. Data
FAQ
- What is a decision model eval?
- An LLM eval for classification: the same labeled examples go through several decision models — zero-shot classifiers that return a probability for each option — and you compare how often each one is right, how well its confidence matches reality, how fast it is and what it costs. Each row has one correct label.
- Which models can I compare?
- Eight: Jev (TypeSafe), Clef-flash and Clef (Cloudflare), Perplexity Decider, OpenAI Luna Decisions, Microsoft-Decision-1, and Liquid AI d1-ultra and d1. All of them take the same plain-English option descriptions, so the comparison is like for like. Untick any model you don't want to run.
- How is this different from promptfoo, OpenAI Evals or other LLM eval frameworks?
- Those frameworks test prompts and generated text, and you write the harness, graders and API calls yourself. This eval is narrower and needs no code: closed-set classification only, decision models only, with accuracy, calibration and a threshold computed for you. If you want to compare general chat models prompted to classify, use one of those frameworks or our Jev vs LLM classification comparison.
- How many examples do I need?
- Accuracy is shown with a 95% confidence interval, so you can see how much it would move with more data. Threshold recommendations need at least 50 answered rows; 100–300 rows covering every label is a good start.
- How is calibration measured?
- Rows are grouped into 10 confidence bins, and expected calibration error (ECE) is the average gap between stated confidence and actual accuracy in each bin, weighted by the number of rows. Lower is better: an ECE of 0.05 means confidence is off by about 5 points on average. Accuracy comes with a 95% Wilson interval, and macro-F1 shows whether rare labels are handled as well as common ones.
- How is the recommended threshold chosen?
- For each model we scan confidence thresholds from 0.50 to 0.99 and keep the one that covers the most rows while the error rate among covered rows stays at or below your target (1%, 5% or 10%). The recommended model is the one with the highest coverage. The threshold is picked on the same rows it is measured on, so treat it as a starting point: confirm it on a fresh set of rows before you rely on it, especially with fewer than 300 rows.
- When is a decision model the wrong choice?
- When answers are open-ended rather than one of a fixed set of labels, when one row needs several labels at once, or when you already have thousands of labeled examples and very high volume — a fine-tuned small classifier can then be cheaper and more accurate. The eval tells you quickly whether a zero-shot model is good enough before you invest in training one.
- What does an evaluation cost?
- Your first evaluation after signing up (up to 100 rows × 4 models) is free, and a new account also gets 50 free credits. After that, 200 rows on all 8 models costs about 2,200 credits — roughly $4.40 at the $9.90 pack of 5,000 credits. You see the estimate before you start, and failed rows are refunded.
- Can I export the results?
- Signed-in users can download a CSV with every row, its label and each model's prediction and confidence, so you can recompute the metrics or test other thresholds yourself. Signed-in runs are kept for 30 days.
- Which data is sent where?
- Each row goes to the providers of the models you select: TypeSafe (Jev), Cloudflare (Clef), and through OpenRouter to Perplexity, OpenAI, Microsoft and Liquid AI. Untick a model to keep data away from its provider. Evaluations are kept for 30 days.
- Is a decision model just a zero-shot classifier?
- Largely, yes: you describe the options in plain English and get a probability for each, without training. What matters in production is whether those probabilities are calibrated on your data and which threshold keeps errors acceptable — which is what this eval measures.