Citation check with Jev: is the claim supported by its source?

What this citation check tests

The example above is a claim that looks close to its source. The source says a 2023 survey of 1,200 IT leaders found 38% planned to increase spending on observability tools and 12% planned cuts. The claim says most IT leaders planned to increase observability spending. Jev answers two questions about it in one call:

The claim gets the direction right — more leaders planned increases than cuts — but "most" is a stronger word than 38% supports. This is the typical failure in research summaries and write-ups: not a fabrication, but a finding stretched a little in the retelling. So the outputs to read are how the supported probability splits between partially and unsupported, and how high overstated goes. A plain supported / not-supported answer would hide that difference.

Two questions on a short state are a standard evaluation: 1 credit on JevStation.

How to write claim-check questions that work

One claim per state. If the claim field holds three sentences, a single probability has to cover all of them. Split compound claims in code and check each one against the passage it cites.

Send the passage, not the paper. Pass the paragraph the citation points to. TypeSafe notes that unrelated material in the state costs accuracy, so trimming the source is part of the check.

Describe each level of support. The supported Choice defines partially as "The source supports part of the claim only". Jev reads literally, so if your editorial standard treats a hedged claim as supported, say so in the criteria.

Give overstatement its own criteria. A Noul with a true and false line makes the boundary explicit:

{
  "type": "noul",
  "instructions": "Does `claim` overstate what `source` says?",
  "criteria": {
    "true": "The claim uses stronger wording, a larger scope or more certainty than the source.",
    "false": "The claim says the same as the source, or less."
  }
}

For more on phrasing, see Designing Questions Jev Can Answer.

From one claim to a review step

1. Extract claim–source pairs. For each citation in a draft, pair the sentence with the passage it cites. That pairing is ordinary code, not a Jev question.
2. Run them together. Put the pairs in a CSV and run the same two questions over every row with batch evaluation.
3. Sort, do not auto-reject. Pass claims that are clearly supported and not overstated, and send the rest to an editor with the probabilities attached so the most doubtful are read first.
4. Record the model version. Every response carries a model field. Log it with each result and re-check your thresholds when it changes — JevStation pins the model at deployment, so you do not choose it per request.
5. Call it from your tools. The same evaluation runs through the System One API with an API key and charges the same credits as the playground.

Check it before you trust it

• Use your own verified claims. Collect 50 to 200 claims a person has already checked and see where Jev’s probabilities separate them.
• Test the wording. TypeSafe’s notes show that a Noul and an equivalent Choice can disagree. Do not carry a threshold tuned on overstated over to a Choice asking the same thing.
• Keep a person on the final call. Calibration is measured across groups of answers, not per answer, so treat each result as a priority for review rather than a verdict.

Where Jev is the wrong claim checker

• The claim turns on arithmetic. Whether "most" fairly describes 38% is a question about wording; whether 38% of 1,200 is "over 450" is arithmetic, which TypeSafe lists as a weakness. Extract figures and compare them in code, then pass the result into the state.
• The claim compares dates. "Before the policy changed" against a dated source is a date comparison, another documented weakness.
• The source is very long. JevStation caps the state at 24,000 characters. For a full report, find the relevant passage first.
• You are checking a RAG answer end to end. Use RAG evaluation, which also asks whether retrieval or generation failed.
• You need a written explanation for the author or the reader.

The evidence behind these limits is set out in Designing Questions Jev Can Answer, and Jev as a Judge covers how a single yes/no check holds up on a real grading task.