← 全部评测评测
Banking77 · card intents：8 个决策模型对比
96 条带标签的样本，一道零样本问题，每个模型拿到的每个标签说明都相同。数据来源：mteb/banking77 test split: 12 easily confused card intents × 8。
用你自己的数据跑一遍问题与标签说明
Which intent best describes this banking customer message?
- card_arrival
- A card that was ordered has not arrived yet
- card_delivery_estimate
- How long delivery of a new card takes
- card_not_working
- The card exists but does not work
- card_swallowed
- An ATM kept or swallowed the card
- lost_or_stolen_card
- The card was lost or stolen
- compromised_card
- Someone else may have the card details (fraud risk)
- card_payment_wrong_exchange_rate
- A card payment used the wrong exchange rate
- card_payment_not_recognised
- A card payment the customer does not recognise
- declined_card_payment
- A card payment was declined
- activate_my_card
- How to activate a card
- card_about_to_expire
- The card is about to expire
- getting_spare_card
- Ordering an additional or spare card
目标错误率
推荐
用 Clef-flash：置信度 ≥ 0.50 的答案自动处理，可以覆盖 97% 的样本，错误率 2.2%；其余交给人工。
排行榜
- Clef-flash Cloudflare97.9%
- 宏平均 F1
- 0.978
- ECE
- 0.080
- 延迟 p50 / p90
- 594 ms
- $ / 千次
- $0.0373
- 自动处理
- 97% (≥ 0.50)
结果来自 2026-10-07
- Clef Cloudflare94.8%
- 宏平均 F1
- 0.947
- ECE
- 0.096
- 延迟 p50 / p90
- 649 ms
- $ / 千次
- $0.0994
- 自动处理
- 95% (≥ 0.50)
结果来自 2026-10-07
- Luna Decisions OpenAI90.5%
- 宏平均 F1
- 0.902
- ECE
- 0.032
- 延迟 p50 / p90
- 486 ms
- $ / 千次
- $0.0296
- 自动处理
- 86% (≥ 0.67)
结果来自 2026-10-07
1 行失败
- Decider Perplexity89.6%
- 宏平均 F1
- 0.894
- ECE
- 0.045
- 延迟 p50 / p90
- 528 ms
- $ / 千次
- $0.0053
- 自动处理
- 86% (≥ 0.58)
结果来自 2026-10-25
- Decision-1 Microsoft88.5%
- 宏平均 F1
- 0.884
- ECE
- 0.060
- 延迟 p50 / p90
- 474 ms
- $ / 千次
- $0.0083
- 自动处理
- 79% (≥ 0.73)
结果来自 2026-10-16
- d1-ultra Liquid AI87.5%
- 宏平均 F1
- 0.870
- ECE
- 0.035
- 延迟 p50 / p90
- 525 ms
- $ / 千次
- $0.0067
- 自动处理
- 78% (≥ 0.72)
结果来自 2026-10-16
- Jev TypeSafe86.5%
- 宏平均 F1
- 0.863
- ECE
- 0.065
- 延迟 p50 / p90
- 378 ms
- $ / 千次
- $0.0235
- 自动处理
- 81% (≥ 0.86)
结果来自 2026-10-16
- d1 Liquid AI86.5%
- 宏平均 F1
- 0.863
- ECE
- 0.065
- 延迟 p50 / p90
- 548 ms
- $ / 千次
- $0.0032
- 自动处理
- 86% (≥ 0.68)
结果来自 2026-10-16
校准 · 混淆矩阵
柱子表示模型在各个置信度区间里实际答对的比例。落在虚线上，说明“90% 有把握”时确实有 90% 答对。
| 真实 ↓ / 预测 → | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1. activate_my_card | 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 2. card_about_to_expire | 0 | 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 3. card_arrival | 0 | 0 | 6 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| 4. card_delivery_estimate | 0 | 0 | 0 | 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 5. card_not_working | 0 | 0 | 0 | 0 | 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 6. card_payment_not_recognised | 0 | 0 | 0 | 0 | 0 | 8 | 0 | 0 | 0 | 0 | 0 | 0 |
| 7. card_payment_wrong_exchange_rate | 0 | 0 | 0 | 0 | 0 | 0 | 8 | 0 | 0 | 0 | 0 | 0 |
| 8. card_swallowed | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 8 | 0 | 0 | 0 | 0 |
| 9. compromised_card | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 8 | 0 | 0 | 0 |
| 10. declined_card_payment | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 8 | 0 | 0 |
| 11. getting_spare_card | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 8 | 0 |
| 12. lost_or_stolen_card | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 8 |
行是真实标签，列是模型预测。绿色是答对；红色标出模型容易混淆的组合。
模型答错的样本
18 行至少有一个模型答错。所有模型都答错的行，往往说明定义不清楚或者标签本身有误。
| # | 文本 | 标签 | Clef-flash | Clef | Luna Decisions | Decider | Decision-1 | d1-ultra | Jev | d1 |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | How do I locate my card? | card_arrival | lost_or_stolen_card78% | lost_or_stolen_card68% | lost_or_stolen_card60% | lost_or_stolen_card64% | lost_or_stolen_card77% | lost_or_stolen_card91% | lost_or_stolen_card93% | lost_or_stolen_card89% |
| 2 | When will I get my card? | card_arrival | card_delivery_estimate63% | card_delivery_estimate53% | card_delivery_estimate71% | card_delivery_estimate70% | card_delivery_estimate67% | card_delivery_estimate79% | card_delivery_estimate63% | card_delivery_estimate55% |
| 9 | I need my card now! | card_delivery_estimate | card_delivery_estimate49% | card_arrival35% | card_arrival48% | card_arrival66% | card_arrival73% | card_arrival35% | card_arrival92% | card_arrival80% |
| 43 | What do I do to stop unauthorized transactions on my card? I was never in the place from which the transactions on my bill were made and I never made them. Please help! | compromised_card | compromised_card59% | card_payment_not_recognised53% | card_payment_not_recognised97% | card_payment_not_recognised82% | card_payment_not_recognised61% | card_payment_not_recognised93% | card_payment_not_recognised47% | card_payment_not_recognised47% |
| 14 | Send my card as soon as you are able to. | card_delivery_estimate | card_delivery_estimate78% | card_delivery_estimate43% | card_arrival40% | card_arrival38% | card_arrival75% | card_arrival27% | card_arrival85% | card_arrival59% |
| 85 | going to need a new card what are the fees and time it takes | card_about_to_expire | card_about_to_expire73% | card_about_to_expire85% | card_delivery_estimate57% | getting_spare_card38% | card_delivery_estimate53% | card_delivery_estimate59% | card_delivery_estimate91% | card_delivery_estimate74% |
| 41 | I think someone is using my card to make transactions I don't remember. | compromised_card | compromised_card47% | card_payment_not_recognised40% | card_payment_not_recognised89% | card_payment_not_recognised47% | compromised_card55% | card_payment_not_recognised72% | compromised_card48% | card_payment_not_recognised67% |
| 81 | Am I able to order a new card and have it sent to me in China? | card_about_to_expire | card_about_to_expire81% | card_about_to_expire75% | getting_spare_card67% | card_about_to_expire39% | getting_spare_card60% | getting_spare_card71% | getting_spare_card77% | getting_spare_card77% |
| 88 | What would the price be for an expired card replacement and how long will it take me to get it? | card_about_to_expire | card_about_to_expire63% | card_about_to_expire86% | card_delivery_estimate66% | card_about_to_expire69% | card_delivery_estimate47% | card_delivery_estimate45% | card_delivery_estimate73% | card_delivery_estimate56% |
| 15 | I need my card to come as soon as possible. | card_delivery_estimate | card_delivery_estimate82% | card_delivery_estimate53% | card_delivery_estimate79% | card_delivery_estimate49% | card_arrival66% | card_delivery_estimate47% | card_arrival85% | card_arrival55% |
| 72 | My card didn't work in one of the shops. | declined_card_payment | declined_card_payment86% | declined_card_payment57% | declined_card_payment51% | card_not_working49% | declined_card_payment52% | declined_card_payment69% | card_not_working52% | card_not_working52% |
| 10 | Can you get me my card fast? | card_delivery_estimate | card_delivery_estimate88% | card_delivery_estimate64% | card_delivery_estimate39% | card_arrival58% | card_delivery_estimate63% | card_delivery_estimate68% | card_arrival41% | card_delivery_estimate46% |
| 19 | How can I get my physical card to work? | card_not_working | card_not_working96% | card_not_working80% | card_not_working87% | card_not_working52% | activate_my_card88% | activate_my_card71% | card_not_working54% | card_not_working76% |
| 46 | How do I freeze my account? | compromised_card | compromised_card72% | compromised_card78% | — | compromised_card54% | compromised_card56% | lost_or_stolen_card67% | compromised_card60% | lost_or_stolen_card62% |
| 61 | Please put a freeze on my card, I am worried there has been some payments on it and I don't know what for. | card_payment_not_recognised | card_payment_not_recognised97% | card_payment_not_recognised90% | card_payment_not_recognised90% | compromised_card46% | card_payment_not_recognised64% | card_payment_not_recognised59% | compromised_card48% | card_payment_not_recognised81% |
| 63 | I see a charge on my account that I don't recall making. I feel like my account may have been compromised. | card_payment_not_recognised | card_payment_not_recognised47% | card_payment_not_recognised44% | card_payment_not_recognised80% | card_payment_not_recognised48% | compromised_card73% | card_payment_not_recognised78% | compromised_card62% | card_payment_not_recognised73% |
| 45 | It seems someone used my card! There are a few transactions from a small town in the middle of nowhere that I definitely have not made! Please prevent them from using it immediately! | compromised_card | compromised_card53% | compromised_card43% | compromised_card33% | compromised_card51% | compromised_card65% | card_payment_not_recognised44% | compromised_card75% | compromised_card49% |
| 74 | How long will it take to activate my new card? | activate_my_card | activate_my_card91% | activate_my_card59% | activate_my_card96% | activate_my_card92% | activate_my_card83% | activate_my_card75% | activate_my_card100% | card_delivery_estimate45% |
在代码里调用
你写的问题、所选模型及其阈值，整理成一个 API 调用。
curl -X POST https://jevstation.com/api/v1/systemone \
-H "Authorization: Bearer $JEVSTATION_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "clef-flash",
"state": "<the text to classify>",
"questions": {
"label": {
"type": "choice",
"instructions": "Which intent best describes this banking customer message?",
"criteria": {
"card_arrival": "A card that was ordered has not arrived yet",
"card_delivery_estimate": "How long delivery of a new card takes",
"card_not_working": "The card exists but does not work",
"card_swallowed": "An ATM kept or swallowed the card",
"lost_or_stolen_card": "The card was lost or stolen",
"compromised_card": "Someone else may have the card details (fraud risk)",
"card_payment_wrong_exchange_rate": "A card payment used the wrong exchange rate",
"card_payment_not_recognised": "A card payment the customer does not recognise",
"declined_card_payment": "A card payment was declined",
"activate_my_card": "How to activate a card",
"card_about_to_expire": "The card is about to expire",
"getting_spare_card": "Ordering an additional or spare card"
}
}
}
}'
# Automate when data.answers.label.confidence >= 0.50;
# send everything in between to a person.