Report · offline evaluation
Phase 6 — Evidence ranking
JEV versus lexical on the same 40 Real-World v1 cases. This phase measures candidate ordering, not generated answer quality.
Scope. 51 passages, 40 cases, 200 candidate records. Relevance judgments are developer-authored and have not received independent human review. The 8-case
exposed_test split is not blind.Overall results
| Metric | JEV | Lexical | Difference |
|---|---|---|---|
| MAP | 0.943 | 0.777 | +0.166 |
| MRR | 0.967 | 0.795 | +0.172 |
| Precision@1 | 0.950 | 0.650 | +0.300 |
| nDCG@5 | 0.964 | 0.842 | +0.122 |
| Recall@5 | 1.000 | 1.000 | 0.000 |
Paired bootstrap 95% intervals (40 cases): MAP +0.166 [0.089, 0.250], MRR +0.172 [0.092, 0.261], P@1 +0.300 [0.175, 0.450]. Recall@5 is saturated on both arms, so the gain is about ranking position, not whether a good passage appears.
JEV scoring cost
76,184 measured input tokens across 200 calls, 0 failures. Typesafe Jev 1.14 list price (docs.typesafe.ai/models, retrieved 2026-11-06): $0.042 per million input tokens, output free.
estimated_usd = (76,184 / 1,000,000) × 0.042 = $0.003199728 — an estimate, not an invoice (invoiced_usd = null).
Limitations
- Developer-authored judgments, not independently reviewed.
- The exposed split is not blind.
- Forty cases only; intervals are wide.
- No answer-quality measurement in this phase.