Eval

AG News · 4 sections: 8 decision models compared

100 labeled rows, one zero-shot question, every model given the same one-line definition per label. Source: fancyzhx/ag_news test split: 4 classes × 25.

Run this on your own data

Question and definitions

Which section of a news site does this article belong to?

world: World news, politics and international affairs
sports: Sports
business: Business, markets and the economy
sci_tech: Science and technology

Target error rate

Recommendation

Use Clef: automate answers with confidence ≥ 0.55. That covers 92% of rows with a 4.3% error rate; send the rest to a person.

Leaderboard

Model	Accuracy	Macro F1	ECE	Auto-handled	Latency p50 / p90	$ / 1k	Credits
Clef Cloudflareresults from 2026-10-07	93.0%86%–97%	0.930	0.080	92%≥ 0.55	627 / 866 ms	$0.0566	×2
Clef-flash Cloudflareresults from 2026-10-07	92.0%85%–96%	0.920	0.073	90%≥ 0.70	578 / 794 ms	$0.0212	×1
Decider Perplexityresults from 2026-10-25	91.0%84%–95%	0.910	0.040	90%≥ 0.62	522 / 573 ms	$0.0035	×1
Decision-1 Microsoftresults from 2026-10-16	91.0%84%–95%	0.911	0.064	83%≥ 0.90	471 / 523 ms	$0.0050	×1
d1-ultra Liquid AIresults from 2026-10-16	91.0%84%–95%	0.910	0.064	91%≥ 0.65	525 / 663 ms	$0.0051	×2
d1 Liquid AIresults from 2026-10-16	91.0%84%–95%	0.910	0.078	83%≥ 0.83	533 / 586 ms	$0.0024	×1
Jev TypeSaferesults from 2026-10-16	89.0%81%–94%	0.889	0.070	83%≥ 0.93	377 / 419 ms	$0.0173	×1
Luna Decisions OpenAIresults from 2026-10-07	89.0%81%–94%	0.885	0.094	—	480 / 521 ms	$0.0201	×1

Clef Cloudflare93.0%
Macro F1
0.930
ECE
0.080
Latency p50 / p90
627 ms
$ / 1k
$0.0566
Auto-handled
92% (≥ 0.55)
results from 2026-10-07
Clef-flash Cloudflare92.0%
Macro F1
0.920
ECE
0.073
Latency p50 / p90
578 ms
$ / 1k
$0.0212
Auto-handled
90% (≥ 0.70)
results from 2026-10-07
Decider Perplexity91.0%
Macro F1
0.910
ECE
0.040
Latency p50 / p90
522 ms
$ / 1k
$0.0035
Auto-handled
90% (≥ 0.62)
results from 2026-10-25
Decision-1 Microsoft91.0%
Macro F1
0.911
ECE
0.064
Latency p50 / p90
471 ms
$ / 1k
$0.0050
Auto-handled
83% (≥ 0.90)
results from 2026-10-16
d1-ultra Liquid AI91.0%
Macro F1
0.910
ECE
0.064
Latency p50 / p90
525 ms
$ / 1k
$0.0051
Auto-handled
91% (≥ 0.65)
results from 2026-10-16
d1 Liquid AI91.0%
Macro F1
0.910
ECE
0.078
Latency p50 / p90
533 ms
$ / 1k
$0.0024
Auto-handled
83% (≥ 0.83)
results from 2026-10-16
Jev TypeSafe89.0%
Macro F1
0.889
ECE
0.070
Latency p50 / p90
377 ms
$ / 1k
$0.0173
Auto-handled
83% (≥ 0.93)
results from 2026-10-16
Luna Decisions OpenAI89.0%
Macro F1
0.885
ECE
0.094
Latency p50 / p90
480 ms
$ / 1k
$0.0201
Auto-handled
—
results from 2026-10-07

Calibration · Confusion matrix

Model

Bars show how often the model was right at each confidence level. On the dashed line, "90% sure" means right 90% of the time.

true ↓ / predicted →	1	2	3	4
1. business	22	3	0	0
2. sci_tech	2	23	0	0
3. sports	0	0	25	0
4. world	1	0	1	23

Rows are the true labels, columns the predicted ones. Green is correct; red marks the pairs the model confuses.

Rows models got wrong

15 rows have at least one wrong answer. Rows every model misses often point to an unclear definition or a wrong label.

#	Text	Label	Clef	Clef-flash	Decider	Decision-1	d1-ultra	d1	Jev	Luna Decisions
14	India's Tata expands regional footprint via NatSteel buyout (AFP) AFP - India's Tata Iron and Steel Company Ltd. took a strategic step to expand its Asian footprint with the announcement it will buy the Asia-Pacific steel operations of Singapore's NatSteel Ltd.	world	business62%	business67%	business82%	business99%	business97%	business97%	business100%	business100%
23	Live: Olympics day four Richard Faulds and Stephen Parry are going for gold for Great Britain on day four in Athens.	world	sports93%	sports86%	sports99%	sports100%	sports99%	sports100%	sports100%	sports100%
98	Some People Not Eligible to Get in on Google IPO Google has billed its IPO as a way for everyday people to get in on the process, denying Wall Street the usual stranglehold it's had on IPOs. Public bidding, a minimum of just five shares, an open process with 28 underwriters - all this pointed to a new level of public participation. But this isn't the case.	sci_tech	business32%	business82%	business62%	business90%	business72%	business82%	business92%	business100%
99	Rivals Try to Turn Tables on Charles Schwab By MICHAEL LIEDTKE SAN FRANCISCO (AP) -- With its low prices and iconoclastic attitude, discount stock broker Charles Schwab Corp. (SCH) represented an annoying stone in Wall Street's wing-tipped shoes for decades...	sci_tech	business54%	business92%	business98%	business100%	business95%	business100%	business100%	business100%
94	Storage, servers bruise HP earnings update Earnings per share rise compared with a year ago, but company misses analysts' expectations by a long shot.	sci_tech	sci_tech40%	business80%	business50%	business90%	business63%	business87%	business95%	business100%
57	Intel to delay product aimed for high-definition TVs SAN FRANCISCO -- In the latest of a series of product delays, Intel Corp. has postponed the launch of a video display chip it had previously planned to introduce by year end, putting off a showdown with Texas Instruments Inc. in the fast-growing market for high-definition television displays.	business	sci_tech91%	sci_tech69%	sci_tech59%	business41%	sci_tech65%	sci_tech82%	sci_tech61%	business88%
84	Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated card fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.	sci_tech	sci_tech54%	sci_tech25%	business28%	business60%	business55%	business47%	business57%	business77%
95	IBM to hire even more new workers By the end of the year, the computing giant plans to have its biggest headcount since 1991.	sci_tech	sci_tech57%	sci_tech30%	business55%	business88%	business61%	business57%	business87%	business100%
70	California slashes legal fees in settlement of Microsoft case California lawyers who reached a \$1.1 billion class-action settlement with Microsoft will get less than half the legal fees they requested.	business	sci_tech76%	sci_tech32%	sci_tech64%	business75%	business53%	sci_tech37%	business69%	business99%
87	Dutch Retailer Beats Apple to Local Download Market AMSTERDAM (Reuters) - Free Record Shop, a Dutch music retail chain, beat Apple Computer Inc. to market on Tuesday with the launch of a new download service in Europe's latest battleground for digital song services.	sci_tech	sci_tech88%	sci_tech86%	sci_tech44%	business37%	business34%	sci_tech40%	business48%	business69%
6	1994 Law Designed to Preserve Guard Jobs (AP) AP - A 1994 law strengthened job protections for National Guard and Reserve troops called to active duty. Here are major provisions of the Uniformed Services Employment and Reemployment Rights Act (USERRA).	world	world63%	world41%	world48%	business37%	world44%	world41%	business37%	world72%
79	Calif. Aims to Limit Farm-Related Smog (AP) AP - Southern California's smog-fighting agency went after emissions of the bovine variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure.	sci_tech	sci_tech44%	world18%	sci_tech61%	sci_tech41%	sci_tech44%	sci_tech26%	sci_tech51%	world87%
80	Open Letter Against British Copyright Indoctrination in Schools The British Department for Education and Skills (DfES) recently launched a "Music Manifesto" campaign, with the ostensible intention of educating the next generation of British musicians. Unfortunately, they also teamed up with the music industry (EMI, and various artists) to make this popular. EMI has apparently negotiated their end well, so that children in our schools will now be indoctrinated about the illegality of downloading music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area.	sci_tech	sci_tech75%	sci_tech75%	sci_tech69%	sci_tech58%	sci_tech53%	sci_tech58%	world32%	world61%
73	Qualcomm Raises Earnings Forecast Qualcomm Inc. on Friday raised its quarterly profit forecast due to strong demand for its mobile phone technology. The San Diego company said it expects earnings per	business	sci_tech32%	business83%	business70%	business94%	business75%	business78%	business95%	business100%
83	E-mail scam targets police chief Wiltshire Police warns about "phishing" after its fraud squad chief was targeted.	sci_tech	sci_tech83%	sci_tech84%	sci_tech96%	sci_tech85%	sci_tech87%	sci_tech44%	sci_tech59%	world96%

Call it from your code

The question you wrote, the selected model and its threshold, as an API call.

curl -X POST https://jevstation.com/api/v1/systemone \
  -H "Authorization: Bearer $JEVSTATION_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
      "model": "clef",
      "state": "<the text to classify>",
      "questions": {
        "label": {
          "type": "choice",
          "instructions": "Which section of a news site does this article belong to?",
          "criteria": {
            "world": "World news, politics and international affairs",
            "sports": "Sports",
            "business": "Business, markets and the economy",
            "sci_tech": "Science and technology"
          }
        }
      }
    }'

# Automate when data.answers.label.confidence >= 0.55;
# send everything in between to a person.