AI fact checker: is the claim supported by its source?

{
  "source": "The 2023 survey of 1,200 IT leaders found that 38% planned to increase spending on observability tools, while 12% planned cuts.",
  "claim": "According to the survey, most IT leaders planned to increase observability spending."
}

Three kinds of AI fact checker

Tools that come up for "AI fact checker" answer different questions, and it helps to know which one you need:

This tool is the third kind. It never searches: you give it the passage a claim cites, and it judges only that pairing. That is narrower than a web search, but it is the question that matters for a citation — does the source say what the text says it does? — and the answer is checkable, because the evidence is right there in the state.

What this citation check tests

The example above is a claim that looks close to its source. The source says a 2023 survey of 1,200 IT leaders found 38% planned to increase spending on observability tools and 12% planned cuts. The claim says most IT leaders planned to increase observability spending. Jev answers two questions about it in one call:

The claim gets the direction right — more leaders planned increases than cuts — but "most" is a stronger word than 38% supports. This is the typical failure in research summaries and write-ups: not a fabrication, but a finding stretched a little in the retelling. So the outputs to read are how the supported probability splits between partially and unsupported, and how high overstated goes. A plain supported / not-supported answer would hide that difference.

Two questions on a short state are a standard evaluation: 1 credit on JevStation.

How to write claim-check questions that work

One claim per state. If the claim field holds three sentences, a single probability has to cover all of them. Split compound claims in code and check each one against the passage it cites.

Send the passage, not the paper. Pass the paragraph the citation points to. TypeSafe notes that unrelated material in the state costs accuracy, so trimming the source is part of the check.

Describe each level of support. The supported Choice defines partially as "The source supports part of the claim only". Jev reads literally, so if your editorial standard treats a hedged claim as supported, say so in the criteria.

Give overstatement its own criteria. A Noul with a true and false line makes the boundary explicit:

{
  "type": "noul",
  "instructions": "Does `claim` overstate what `source` says?",
  "criteria": {
    "true": "The claim uses stronger wording, a larger scope or more certainty than the source.",
    "false": "The claim says the same as the source, or less."
  }
}

For more on phrasing, see Designing Questions Jev Can Answer.

Fact-checking AI-generated text

University library guides on fact-checking AI output give consistent advice: check that a cited source exists, then read it and confirm it says what the AI claimed. The second step is where this tool fits.

1. Check the source exists. Language models invent references. Confirm each cited URL, DOI or title resolves before anything else — that is a lookup, not a judgement, so do it in code or by hand.
2. Find the passage. Open the source and pull the paragraph the claim relies on. If no passage supports the claim at all, you have your answer already.
3. Check claim against passage. Paste both here, or run every pair from a draft through the batch page. unsupported and a high overstated probability go to the top of the review queue.
4. Handle claims with no citation. Find a source first — a search-based checker or a manual search can help — then check the claim against it here.

For answers produced by your own retrieval pipeline, RAG evaluation runs the same kind of check across the whole answer and also tells you whether retrieval or generation was at fault.

From one claim to a review step

1. Extract claim–source pairs. For each citation in a draft, pair the sentence with the passage it cites. That pairing is ordinary code, not a Jev question.
2. Run them together. Put the pairs in a CSV and run the same two questions over every row with batch evaluation.
3. Sort, do not auto-reject. Pass claims that are clearly supported and not overstated, and send the rest to an editor with the probabilities attached so the most doubtful are read first.
4. Record the model version. Every response carries a model field. Log it with each result and re-check your thresholds when it changes — JevStation pins the model at deployment, so you do not choose it per request.
5. Call it from your tools. The same evaluation runs through the System One API with an API key and charges the same credits as the playground.

Check it before you trust it

• Use your own verified claims. Collect 50 to 200 claims a person has already checked and see where Jev’s probabilities separate them.
• Test the wording. TypeSafe’s notes show that a Noul and an equivalent Choice can disagree. Do not carry a threshold tuned on overstated over to a Choice asking the same thing.
• Keep a person on the final call. Calibration is measured across groups of answers, not per answer, so treat each result as a priority for review rather than a verdict.

Where Jev is the wrong claim checker

• The claim turns on arithmetic. Whether "most" fairly describes 38% is a question about wording; whether 38% of 1,200 is "over 450" is arithmetic, which TypeSafe lists as a weakness. Extract figures and compare them in code, then pass the result into the state.
• The claim compares dates. "Before the policy changed" against a dated source is a date comparison, another documented weakness.
• The source is very long. JevStation caps the state at 24,000 characters. For a full report, find the relevant passage first.
• You are checking a RAG answer end to end. Use RAG evaluation, which also asks whether retrieval or generation failed.
• You need a written explanation for the author or the reader.

The evidence behind these limits is set out in Designing Questions Jev Can Answer, and Jev as a Judge covers how a single yes/no check holds up on a real grading task.