Evaluation · Benchmarks
Cost & Performance
Three distinct experiments answer different questions. Their results are shown separately; estimates and limitations are kept with the measurements.
Shared evaluation context
Real-World v1 · fixed inputs
The experiments use 51 passages, 40 evaluation cases, and 200 candidate records. The relevance judgments were written by the developer and have not received independent human review. The 8-case exposed test split was disclosed previously; it is not a blind or independent scientific validation set. The frozen synthetic benchmark and historical result files are separate and unchanged.
Estimate · projection
Token and cost calculator
Applies the token reduction measured on Real-World v1 (40 cases) to your own volume, then estimates cost from published list prices. It does not promise an answer-quality gain.
Assumptions: reduction 22.0% (scenario, family proxy); JEV at $0.042/M input tokens × 2.0 per baseline token; 70% input split; prices editable in the module, list prices at the time of writing. Measured reduction range across the paired runs is 21.1%–24.0%.
Estimates, not invoices. Token counts are a projection from measured rates and will differ with your prompt distribution. Quality is not guaranteed by these numbers.
Phase 6 · Evidence ranking
Real-World Benchmark
JEV-ranked evidence was compared with the frozen Lexical baseline. This phase measures candidate ordering, not generated answer quality or total service cost.
| Metric | JEV | Lexical | Difference |
|---|---|---|---|
| MAP | 0.943 | 0.777 | +0.166 |
| MRR | 0.967 | 0.795 | +0.172 |
| Precision@1 | 0.950 | 0.650 | +0.300 |
| Recall@5 | 1.000 | 1.000 | 0.000 |
Descriptive performance against developer-authored labels; the exposed split is not blind and the result does not establish general superiority.
Phase 7 · Answer-generation comparison
End-to-End RAG Benchmark
Traditional RAG passed the top five lexical candidates. Evidence Layer passed the top three candidates selected using saved Phase 6 JEV scores. The intended route difference was evidence selection and context size. Phase 7 made no new JEV calls; JEV API cost was unavailable and total system cost cannot be determined from this phase. A separate paired Haiku run selects with the lexical scorer only and makes no JEV call at all; it appears below next to the JEV-estimated Anthropic rows.
| Provider | Total tokens Traditional → EL | LLM cost Traditional → EL | E2E latency Traditional → EL | Evidence coverage | Quality readout |
|---|---|---|---|---|---|
| OpenAI | 507.75 → 386.10 −23.96% | $0.001741 → $0.001472 −15.48% | 2,264 → 3,061 ms +796 ms estimated | 1.000 → 0.979 | Correctness unchanged; completeness lower; no general improvement established |
| DeepSeek | 728.95 → 570.52 −21.73% | $0.000823 → $0.000716 −13.05% | 4,974 → 5,094 ms +121 ms estimated | 1.000 → 0.979 | Small rubric changes vary by metric; no general improvement established |
Phase 7 costs are LLM usage estimates only; three earlier incomplete DeepSeek responses lacked usage fields. JEV charges were unknown. Latency adds historical Phase 6 JEV timing and is not a fresh live end-to-end measurement. Automated answer judgments are not independent human review.
Anthropic — Evidence Layer with and without JEV
| Model and variant | Tokens / case | Measured LLM cost | System cost | Quality readout |
|---|---|---|---|---|
| Haiku · EL + lexical No JEV | 912.25 → 701.00 −23.16 % | $0.007876 → $0.006698 −14.96 % | $0.007876 → $0.006698 −14.96 % | Correctness 2.000 → 1.975 ; complétude 1.950 → 1.725 (IC −0.40 ; −0.075) ; groundedness 1.975 → 2.000. No quality gain established. |
| Haiku · EL + estimated JEV | 906.15 → 710.43 −21.60 % | $0.007754 → $0.006765 | $0.007754 → $0.009965 +$0.002210 | No quality gain established. |
| Sonnet · EL + estimated JEV | 822.75 → 631.68 −23.22 % | $0.121724 → $0.103798 | $0.121724 → $0.106998 | No quality gain established. |
| Opus · EL + estimated JEV | 915.75 → 723.00 −21.05 % | $0.317848 → $0.280656 | $0.317848 → $0.283856 | No quality gain established. |
The “no JEV” variant made no JEV call: system cost equals measured LLM cost. It removes the Haiku cost increase seen with the JEV estimate, but completeness falls by 0.225 and gold-passage recall from 1.000 to 0.871. The “estimated JEV” rows use list prices, not an invoice.
Phase 7B · Live cost validation
Fresh JEV in the end-to-end path
Phase 7B reran all four provider/route arms and made fresh JEV scoring requests within each Evidence Layer arm. The 80 paired automated answer judgments are exploratory. Values below compare the same fixed 40 cases; system cost is estimated from provider usage and published rates, not from account invoices.
| Measure | OpenAI | DeepSeek |
|---|---|---|
| Mean total tokens per answer | 510.35 → 384.05 −24.75% | 798.125 → 567.25 −28.93% |
| Estimated total API cost per 40-case arm | $0.070700 → $0.061188 −13.45% | $0.029040 → $0.025995 −10.49% |
| Mean end-to-end latency | 2,329 → 3,005 ms +676 ms | 4,144 → 4,159 ms +16 ms |
| Evidence coverage | 1.0000 → 0.9792 | 1.0000 → 0.9792 |
| Mean correctness score / 2 | 2.00 → 2.00 | 1.95 → 2.00 |
| Paired total-cost interval | Reduction interval excludes zero | Includes zero; reduction is not statistically conclusive on this fixed set |
No general answer-quality improvement was established. Coverage decreased by 0.0208 in both providers. OpenAI’s estimated system-cost reduction is supported by the paired interval for this dataset; the DeepSeek interval includes zero. OpenAI mean latency rose by about 676 ms; DeepSeek’s 16 ms increase is not conclusive. Estimated costs are not invoices and do not prove future savings.
Method and limits
- All three phases used the fixed Real-World v1 questions and developer-authored relevance labels. Those labels have not been independently human-reviewed.
- The eight-case exposed test split was disclosed earlier and is not a blind test. It should not be used as blind evidence for future comparisons.
- Phase 7 and Phase 7B used an automated OpenAI judge for answer rubrics. Automated scores are not independent human judgments and the judge provider overlaps with a tested generator.
- Phase 6 measures evidence ranking. Phase 7 measures answer generation with cached Phase 6 JEV output. Phase 7B measures live JEV plus answer generation. Their cost and latency values answer different questions and must not be pooled.
- The result files, manifests, source hashes, and synthetic freeze artifacts remain separate historical records. Reproduction commands and input/output fingerprints are in each linked report or manifest.