JevStation
← 全部评测
评测

AG News · 4 sections：8 个决策模型对比

100 条带标签的样本，一道零样本问题，每个模型拿到的每个标签说明都相同。数据来源：fancyzhx/ag_news test split: 4 classes × 25。

用你自己的数据跑一遍

问题与标签说明

Which section of a news site does this article belong to?

world
World news, politics and international affairs
sports
Sports
business
Business, markets and the economy
sci_tech
Science and technology
目标错误率

推荐

用 Clef：置信度 ≥ 0.55 的答案自动处理，可以覆盖 92% 的样本，错误率 4.3%；其余交给人工。

排行榜

  1. Clef Cloudflare93.0%
    宏平均 F1
    0.930
    ECE
    0.080
    延迟 p50 / p90
    627 ms
    $ / 千次
    $0.0566
    自动处理
    92% (≥ 0.55)

    结果来自 2026-10-07

  2. Clef-flash Cloudflare92.0%
    宏平均 F1
    0.920
    ECE
    0.073
    延迟 p50 / p90
    578 ms
    $ / 千次
    $0.0212
    自动处理
    90% (≥ 0.70)

    结果来自 2026-10-07

  3. Decider Perplexity91.0%
    宏平均 F1
    0.910
    ECE
    0.040
    延迟 p50 / p90
    522 ms
    $ / 千次
    $0.0035
    自动处理
    90% (≥ 0.62)

    结果来自 2026-10-25

  4. Decision-1 Microsoft91.0%
    宏平均 F1
    0.911
    ECE
    0.064
    延迟 p50 / p90
    471 ms
    $ / 千次
    $0.0050
    自动处理
    83% (≥ 0.90)

    结果来自 2026-10-16

  5. d1-ultra Liquid AI91.0%
    宏平均 F1
    0.910
    ECE
    0.064
    延迟 p50 / p90
    525 ms
    $ / 千次
    $0.0051
    自动处理
    91% (≥ 0.65)

    结果来自 2026-10-16

  6. d1 Liquid AI91.0%
    宏平均 F1
    0.910
    ECE
    0.078
    延迟 p50 / p90
    533 ms
    $ / 千次
    $0.0024
    自动处理
    83% (≥ 0.83)

    结果来自 2026-10-16

  7. Jev TypeSafe89.0%
    宏平均 F1
    0.889
    ECE
    0.070
    延迟 p50 / p90
    377 ms
    $ / 千次
    $0.0173
    自动处理
    83% (≥ 0.93)

    结果来自 2026-10-16

  8. Luna Decisions OpenAI89.0%
    宏平均 F1
    0.885
    ECE
    0.094
    延迟 p50 / p90
    480 ms
    $ / 千次
    $0.0201
    自动处理
    —

    结果来自 2026-10-07

校准 · 混淆矩阵

0.3–0.4: 0% (2)0.4–0.5: 100% (4)0.5–0.6: 67% (3)0.6–0.7: 83% (6)0.7–0.8: 86% (7)0.8–0.9: 100% (13)0.9–1.0: 97% (65)000.50.511

柱子表示模型在各个置信度区间里实际答对的比例。落在虚线上，说明“90% 有把握”时确实有 90% 答对。

真实 ↓ / 预测 →1234
1. business22300
2. sci_tech22300
3. sports00250
4. world10123

行是真实标签，列是模型预测。绿色是答对；红色标出模型容易混淆的组合。

模型答错的样本

15 行至少有一个模型答错。所有模型都答错的行，往往说明定义不清楚或者标签本身有误。

#文本标签ClefClef-flashDeciderDecision-1d1-ultrad1JevLuna Decisions
14India's Tata expands regional footprint via NatSteel buyout (AFP) AFP - India's Tata Iron and Steel Company Ltd. took a strategic step to expand its Asian footprint with the announcement it will buy the Asia-Pacific steel operations of Singapore's NatSteel Ltd.worldbusiness62%business67%business82%business99%business97%business97%business100%business100%
23Live: Olympics day four Richard Faulds and Stephen Parry are going for gold for Great Britain on day four in Athens.worldsports93%sports86%sports99%sports100%sports99%sports100%sports100%sports100%
98Some People Not Eligible to Get in on Google IPO Google has billed its IPO as a way for everyday people to get in on the process, denying Wall Street the usual stranglehold it's had on IPOs. Public bidding, a minimum of just five shares, an open process with 28 underwriters - all this pointed to a new level of public participation. But this isn't the case.sci_techbusiness32%business82%business62%business90%business72%business82%business92%business100%
99Rivals Try to Turn Tables on Charles Schwab By MICHAEL LIEDTKE SAN FRANCISCO (AP) -- With its low prices and iconoclastic attitude, discount stock broker Charles Schwab Corp. (SCH) represented an annoying stone in Wall Street's wing-tipped shoes for decades...sci_techbusiness54%business92%business98%business100%business95%business100%business100%business100%
94Storage, servers bruise HP earnings update Earnings per share rise compared with a year ago, but company misses analysts' expectations by a long shot.sci_techsci_tech40%business80%business50%business90%business63%business87%business95%business100%
57Intel to delay product aimed for high-definition TVs SAN FRANCISCO -- In the latest of a series of product delays, Intel Corp. has postponed the launch of a video display chip it had previously planned to introduce by year end, putting off a showdown with Texas Instruments Inc. in the fast-growing market for high-definition television displays.businesssci_tech91%sci_tech69%sci_tech59%business41%sci_tech65%sci_tech82%sci_tech61%business88%
84Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated card fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.sci_techsci_tech54%sci_tech25%business28%business60%business55%business47%business57%business77%
95IBM to hire even more new workers By the end of the year, the computing giant plans to have its biggest headcount since 1991.sci_techsci_tech57%sci_tech30%business55%business88%business61%business57%business87%business100%
70California slashes legal fees in settlement of Microsoft case California lawyers who reached a \$1.1 billion class-action settlement with Microsoft will get less than half the legal fees they requested.businesssci_tech76%sci_tech32%sci_tech64%business75%business53%sci_tech37%business69%business99%
87Dutch Retailer Beats Apple to Local Download Market AMSTERDAM (Reuters) - Free Record Shop, a Dutch music retail chain, beat Apple Computer Inc. to market on Tuesday with the launch of a new download service in Europe's latest battleground for digital song services.sci_techsci_tech88%sci_tech86%sci_tech44%business37%business34%sci_tech40%business48%business69%
61994 Law Designed to Preserve Guard Jobs (AP) AP - A 1994 law strengthened job protections for National Guard and Reserve troops called to active duty. Here are major provisions of the Uniformed Services Employment and Reemployment Rights Act (USERRA).worldworld63%world41%world48%business37%world44%world41%business37%world72%
79Calif. Aims to Limit Farm-Related Smog (AP) AP - Southern California's smog-fighting agency went after emissions of the bovine variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure.sci_techsci_tech44%world18%sci_tech61%sci_tech41%sci_tech44%sci_tech26%sci_tech51%world87%
80Open Letter Against British Copyright Indoctrination in Schools The British Department for Education and Skills (DfES) recently launched a "Music Manifesto" campaign, with the ostensible intention of educating the next generation of British musicians. Unfortunately, they also teamed up with the music industry (EMI, and various artists) to make this popular. EMI has apparently negotiated their end well, so that children in our schools will now be indoctrinated about the illegality of downloading music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area.sci_techsci_tech75%sci_tech75%sci_tech69%sci_tech58%sci_tech53%sci_tech58%world32%world61%
73Qualcomm Raises Earnings Forecast Qualcomm Inc. on Friday raised its quarterly profit forecast due to strong demand for its mobile phone technology. The San Diego company said it expects earnings per businesssci_tech32%business83%business70%business94%business75%business78%business95%business100%
83E-mail scam targets police chief Wiltshire Police warns about "phishing" after its fraud squad chief was targeted.sci_techsci_tech83%sci_tech84%sci_tech96%sci_tech85%sci_tech87%sci_tech44%sci_tech59%world96%

在代码里调用

你写的问题、所选模型及其阈值，整理成一个 API 调用。

curl -X POST https://jevstation.com/api/v1/systemone \
  -H "Authorization: Bearer $JEVSTATION_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
      "model": "clef",
      "state": "<the text to classify>",
      "questions": {
        "label": {
          "type": "choice",
          "instructions": "Which section of a news site does this article belong to?",
          "criteria": {
            "world": "World news, politics and international affairs",
            "sports": "Sports",
            "business": "Business, markets and the economy",
            "sci_tech": "Science and technology"
          }
        }
      }
    }'

# Automate when data.answers.label.confidence >= 0.55;
# send everything in between to a person.