RAG evaluation with Jev: grounded answers and where the pipeline failed

What this RAG evaluation checks

The example above is a refund question. The retrieved context says refunds are issued within 2 business days of approval and that card issuers usually post them 5–10 business days after that. The answer says refunds are instant and arrive the same day. Jev answers four questions about it in one call:

This example is useful because the two stages disagree. The retriever did its job — the timing is right there in the context — and the model then contradicted it. So the outputs worth reading are how low grounded goes, how high context_relevant stays, and whether failure leans towards generation rather than retrieval. A single "is this answer good?" score would flag the answer but not tell you which half of the pipeline to fix.

Four questions on a short state are a standard evaluation, 1 credit on JevStation. The questions are answered in parallel against the same state, so asking about both stages adds almost no latency.

How to write RAG questions that work

Keep groundedness and correctness apart. An answer can be grounded in a stale document and still be wrong for the user, or correct from general knowledge and not grounded at all. grounded asks only about support in the context. If you also need correctness against a reference, add a separate question — see Jev as a judge.

Ask about the context on its own. context_relevant is the retrieval check. When it is low, the generator had nothing to work with, and a better prompt will not help. Give it criteria so Jev knows where the line is:

{
  "type": "noul",
  "instructions": "Does `retrieved_context` contain the information needed to answer `question`?",
  "criteria": {
    "true": "A careful reader could answer the question using only the context.",
    "false": "The context is off-topic, incomplete or silent on what the question asks."
  }
}

Describe every failure option. The failure Choice works because each option has a one-line description. Jev reads literally, so vague options such as "bad retrieval" leave it to guess your boundaries.

Do not expect the questions to agree arithmetically. TypeSafe notes that separate questions are independent and need not add up — a Noul and an equivalent Choice can give different numbers. Read failure alongside context_relevant rather than deriving one from the other.

For more on phrasing, see Designing Questions Jev Can Answer.

From one check to an evaluation step

1. Build a labelled set. 50 to 200 real question–context–answer triples, each marked by a person as grounded or not and, where wrong, which stage failed.
2. Run the set in one go. Put the triples in a CSV and run the same question set over every row with batch evaluation. Compare the probabilities to your labels.
3. Pick bands, not one threshold. Pass answers that are clearly grounded, regenerate or escalate the middle band, and block the rest.
4. Record the model version. Every response carries a model field. Log it with each result and re-check your thresholds when it changes — JevStation pins the model at deployment, so you do not choose it per request.
5. Call it from your pipeline. The same evaluation runs through the System One API with an API key and charges the same credits as the playground.

Check it before you trust it

• Measure against your own labels. A threshold that separates good from bad on the refund example is not a threshold for your corpus.
• Watch for injected text in retrieved documents. TypeSafe’s notes say content written to steer the model can move the answer. If your corpus includes untrusted pages, screen them with the prompt injection detector as well.
• Test non-English content separately. TypeSafe documents English as the language where accuracy is best.

Where Jev is the wrong RAG evaluator

• The answer turns on numbers or dates. Whether "within 7 days" matches "5–10 business days" is arithmetic, and TypeSafe lists math and date comparison as weaknesses. Extract and compare those values in code.
• You pass the whole corpus as context. TypeSafe warns that unrelated material in the state costs accuracy. Send only the passages the retriever returned; JevStation caps the state at 24,000 characters.
• You need to know which sentence is unsupported. Split the answer into claims and run the citation check on each one against its passage.
• You need a written critique for a reviewer or as feedback to the generator.

The evidence behind grading with a single yes/no question — and where it breaks — is laid out in Jev as a Judge: What One Yes/No Question Catches, and What It Misses.