Eval

Banking77 · card intents: 8 decision models compared

96 labeled rows, one zero-shot question, every model given the same one-line definition per label. Source: mteb/banking77 test split: 12 easily confused card intents × 8.

Run this on your own data

Question and definitions

Which intent best describes this banking customer message?

card_arrival: A card that was ordered has not arrived yet
card_delivery_estimate: How long delivery of a new card takes
card_not_working: The card exists but does not work
card_swallowed: An ATM kept or swallowed the card
lost_or_stolen_card: The card was lost or stolen
compromised_card: Someone else may have the card details (fraud risk)
card_payment_wrong_exchange_rate: A card payment used the wrong exchange rate
card_payment_not_recognised: A card payment the customer does not recognise
declined_card_payment: A card payment was declined
activate_my_card: How to activate a card
card_about_to_expire: The card is about to expire
getting_spare_card: Ordering an additional or spare card

Target error rate

Recommendation

Use Clef-flash: automate answers with confidence ≥ 0.50. That covers 97% of rows with a 2.2% error rate; send the rest to a person.

Leaderboard

Model	Accuracy	Macro F1	ECE	Auto-handled	Latency p50 / p90	$ / 1k	Credits
Clef-flash Cloudflareresults from 2026-10-07	97.9%93%–99%	0.978	0.080	97%≥ 0.50	594 / 733 ms	$0.0373	×1
Clef Cloudflareresults from 2026-10-07	94.8%88%–98%	0.947	0.096	95%≥ 0.50	649 / 1227 ms	$0.0994	×2
Luna Decisions OpenAI1 row failedresults from 2026-10-07	90.5%83%–95%	0.902	0.032	86%≥ 0.67	486 / 525 ms	$0.0296	×1
Decider Perplexityresults from 2026-10-25	89.6%82%–94%	0.894	0.045	86%≥ 0.58	528 / 639 ms	$0.0053	×1
Decision-1 Microsoftresults from 2026-10-16	88.5%81%–93%	0.884	0.060	79%≥ 0.73	474 / 526 ms	$0.0083	×1
d1-ultra Liquid AIresults from 2026-10-16	87.5%79%–93%	0.870	0.035	78%≥ 0.72	525 / 597 ms	$0.0067	×2
Jev TypeSaferesults from 2026-10-16	86.5%78%–92%	0.863	0.065	81%≥ 0.86	378 / 649 ms	$0.0235	×1
d1 Liquid AIresults from 2026-10-16	86.5%78%–92%	0.863	0.065	86%≥ 0.68	548 / 672 ms	$0.0032	×1

Clef-flash Cloudflare97.9%
Macro F1
0.978
ECE
0.080
Latency p50 / p90
594 ms
$ / 1k
$0.0373
Auto-handled
97% (≥ 0.50)
results from 2026-10-07
Clef Cloudflare94.8%
Macro F1
0.947
ECE
0.096
Latency p50 / p90
649 ms
$ / 1k
$0.0994
Auto-handled
95% (≥ 0.50)
results from 2026-10-07
Luna Decisions OpenAI90.5%
Macro F1
0.902
ECE
0.032
Latency p50 / p90
486 ms
$ / 1k
$0.0296
Auto-handled
86% (≥ 0.67)
results from 2026-10-07
1 row failed
Decider Perplexity89.6%
Macro F1
0.894
ECE
0.045
Latency p50 / p90
528 ms
$ / 1k
$0.0053
Auto-handled
86% (≥ 0.58)
results from 2026-10-25
Decision-1 Microsoft88.5%
Macro F1
0.884
ECE
0.060
Latency p50 / p90
474 ms
$ / 1k
$0.0083
Auto-handled
79% (≥ 0.73)
results from 2026-10-16
d1-ultra Liquid AI87.5%
Macro F1
0.870
ECE
0.035
Latency p50 / p90
525 ms
$ / 1k
$0.0067
Auto-handled
78% (≥ 0.72)
results from 2026-10-16
Jev TypeSafe86.5%
Macro F1
0.863
ECE
0.065
Latency p50 / p90
378 ms
$ / 1k
$0.0235
Auto-handled
81% (≥ 0.86)
results from 2026-10-16
d1 Liquid AI86.5%
Macro F1
0.863
ECE
0.065
Latency p50 / p90
548 ms
$ / 1k
$0.0032
Auto-handled
86% (≥ 0.68)
results from 2026-10-16

Calibration · Confusion matrix

Model

Bars show how often the model was right at each confidence level. On the dashed line, "90% sure" means right 90% of the time.

true ↓ / predicted →	1	2	3	4	5	6	7	8	9	10	11	12
1. activate_my_card	8	0	0	0	0	0	0	0	0	0	0	0
2. card_about_to_expire	0	8	0	0	0	0	0	0	0	0	0	0
3. card_arrival	0	0	6	1	0	0	0	0	0	0	0	1
4. card_delivery_estimate	0	0	0	8	0	0	0	0	0	0	0	0
5. card_not_working	0	0	0	0	8	0	0	0	0	0	0	0
6. card_payment_not_recognised	0	0	0	0	0	8	0	0	0	0	0	0
7. card_payment_wrong_exchange_rate	0	0	0	0	0	0	8	0	0	0	0	0
8. card_swallowed	0	0	0	0	0	0	0	8	0	0	0	0
9. compromised_card	0	0	0	0	0	0	0	0	8	0	0	0
10. declined_card_payment	0	0	0	0	0	0	0	0	0	8	0	0
11. getting_spare_card	0	0	0	0	0	0	0	0	0	0	8	0
12. lost_or_stolen_card	0	0	0	0	0	0	0	0	0	0	0	8

Rows are the true labels, columns the predicted ones. Green is correct; red marks the pairs the model confuses.

Rows models got wrong

18 rows have at least one wrong answer. Rows every model misses often point to an unclear definition or a wrong label.

#	Text	Label	Clef-flash	Clef	Luna Decisions	Decider	Decision-1	d1-ultra	Jev	d1
1	How do I locate my card?	card_arrival	lost_or_stolen_card78%	lost_or_stolen_card68%	lost_or_stolen_card60%	lost_or_stolen_card64%	lost_or_stolen_card77%	lost_or_stolen_card91%	lost_or_stolen_card93%	lost_or_stolen_card89%
2	When will I get my card?	card_arrival	card_delivery_estimate63%	card_delivery_estimate53%	card_delivery_estimate71%	card_delivery_estimate70%	card_delivery_estimate67%	card_delivery_estimate79%	card_delivery_estimate63%	card_delivery_estimate55%
9	I need my card now!	card_delivery_estimate	card_delivery_estimate49%	card_arrival35%	card_arrival48%	card_arrival66%	card_arrival73%	card_arrival35%	card_arrival92%	card_arrival80%
43	What do I do to stop unauthorized transactions on my card? I was never in the place from which the transactions on my bill were made and I never made them. Please help!	compromised_card	compromised_card59%	card_payment_not_recognised53%	card_payment_not_recognised97%	card_payment_not_recognised82%	card_payment_not_recognised61%	card_payment_not_recognised93%	card_payment_not_recognised47%	card_payment_not_recognised47%
14	Send my card as soon as you are able to.	card_delivery_estimate	card_delivery_estimate78%	card_delivery_estimate43%	card_arrival40%	card_arrival38%	card_arrival75%	card_arrival27%	card_arrival85%	card_arrival59%
85	going to need a new card what are the fees and time it takes	card_about_to_expire	card_about_to_expire73%	card_about_to_expire85%	card_delivery_estimate57%	getting_spare_card38%	card_delivery_estimate53%	card_delivery_estimate59%	card_delivery_estimate91%	card_delivery_estimate74%
41	I think someone is using my card to make transactions I don't remember.	compromised_card	compromised_card47%	card_payment_not_recognised40%	card_payment_not_recognised89%	card_payment_not_recognised47%	compromised_card55%	card_payment_not_recognised72%	compromised_card48%	card_payment_not_recognised67%
81	Am I able to order a new card and have it sent to me in China?	card_about_to_expire	card_about_to_expire81%	card_about_to_expire75%	getting_spare_card67%	card_about_to_expire39%	getting_spare_card60%	getting_spare_card71%	getting_spare_card77%	getting_spare_card77%
88	What would the price be for an expired card replacement and how long will it take me to get it?	card_about_to_expire	card_about_to_expire63%	card_about_to_expire86%	card_delivery_estimate66%	card_about_to_expire69%	card_delivery_estimate47%	card_delivery_estimate45%	card_delivery_estimate73%	card_delivery_estimate56%
15	I need my card to come as soon as possible.	card_delivery_estimate	card_delivery_estimate82%	card_delivery_estimate53%	card_delivery_estimate79%	card_delivery_estimate49%	card_arrival66%	card_delivery_estimate47%	card_arrival85%	card_arrival55%
72	My card didn't work in one of the shops.	declined_card_payment	declined_card_payment86%	declined_card_payment57%	declined_card_payment51%	card_not_working49%	declined_card_payment52%	declined_card_payment69%	card_not_working52%	card_not_working52%
10	Can you get me my card fast?	card_delivery_estimate	card_delivery_estimate88%	card_delivery_estimate64%	card_delivery_estimate39%	card_arrival58%	card_delivery_estimate63%	card_delivery_estimate68%	card_arrival41%	card_delivery_estimate46%
19	How can I get my physical card to work?	card_not_working	card_not_working96%	card_not_working80%	card_not_working87%	card_not_working52%	activate_my_card88%	activate_my_card71%	card_not_working54%	card_not_working76%
46	How do I freeze my account?	compromised_card	compromised_card72%	compromised_card78%	—	compromised_card54%	compromised_card56%	lost_or_stolen_card67%	compromised_card60%	lost_or_stolen_card62%
61	Please put a freeze on my card, I am worried there has been some payments on it and I don't know what for.	card_payment_not_recognised	card_payment_not_recognised97%	card_payment_not_recognised90%	card_payment_not_recognised90%	compromised_card46%	card_payment_not_recognised64%	card_payment_not_recognised59%	compromised_card48%	card_payment_not_recognised81%
63	I see a charge on my account that I don't recall making. I feel like my account may have been compromised.	card_payment_not_recognised	card_payment_not_recognised47%	card_payment_not_recognised44%	card_payment_not_recognised80%	card_payment_not_recognised48%	compromised_card73%	card_payment_not_recognised78%	compromised_card62%	card_payment_not_recognised73%
45	It seems someone used my card! There are a few transactions from a small town in the middle of nowhere that I definitely have not made! Please prevent them from using it immediately!	compromised_card	compromised_card53%	compromised_card43%	compromised_card33%	compromised_card51%	compromised_card65%	card_payment_not_recognised44%	compromised_card75%	compromised_card49%
74	How long will it take to activate my new card?	activate_my_card	activate_my_card91%	activate_my_card59%	activate_my_card96%	activate_my_card92%	activate_my_card83%	activate_my_card75%	activate_my_card100%	card_delivery_estimate45%

Call it from your code

The question you wrote, the selected model and its threshold, as an API call.

curl -X POST https://jevstation.com/api/v1/systemone \
  -H "Authorization: Bearer $JEVSTATION_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
      "model": "clef-flash",
      "state": "<the text to classify>",
      "questions": {
        "label": {
          "type": "choice",
          "instructions": "Which intent best describes this banking customer message?",
          "criteria": {
            "card_arrival": "A card that was ordered has not arrived yet",
            "card_delivery_estimate": "How long delivery of a new card takes",
            "card_not_working": "The card exists but does not work",
            "card_swallowed": "An ATM kept or swallowed the card",
            "lost_or_stolen_card": "The card was lost or stolen",
            "compromised_card": "Someone else may have the card details (fraud risk)",
            "card_payment_wrong_exchange_rate": "A card payment used the wrong exchange rate",
            "card_payment_not_recognised": "A card payment the customer does not recognise",
            "declined_card_payment": "A card payment was declined",
            "activate_my_card": "How to activate a card",
            "card_about_to_expire": "The card is about to expire",
            "getting_spare_card": "Ordering an additional or spare card"
          }
        }
      }
    }'

# Automate when data.answers.label.confidence >= 0.50;
# send everything in between to a person.