Report · answer generation
Phase 7 — Broad RAG versus Evidence Layer
Same question and same model per pair. Arm A passes the five best lexical passages; arm B passes three selected passages (JEV, or lexical only in the Haiku no-JEV run).
Two Anthropic variants. The historical Haiku run used cached JEV scores (estimated cost). The Haiku EL + lexical, no JEV run measures the same setup without any JEV scoring.
OpenAI and DeepSeek (historical)
| Provider | Total tokens A → B | LLM cost A → B | Latency A → B | Quality |
|---|---|---|---|---|
| OpenAI gpt-6.1-sol | 507.75 → 386.10 −23.96% | $0.069660 → $0.058880 | 2,264 → 3,061 ms | Correctness unchanged; completeness lower; no general improvement established. |
| DeepSeek V4 Pro | 728.95 → 570.52 −21.73% | $0.032920 → $0.028622 | 4,974 → 5,094 ms | Small metric-dependent changes; no general improvement established. |
Anthropic — with estimated JEV (historical) and without JEV (new)
| Model · variant | Tokens A → B | Measured LLM cost A → B | System cost A → B | Quality readout |
|---|---|---|---|---|
| Claude Haiku 5.5 · EL + lexical, no JEV | 912.25 → 701.00 −23.16% | $0.007876 → $0.006698 | $0.007876 → $0.006698 −14.96% | Correctness 2.000 → 1.975; completeness 1.950 → 1.725 (CI −0.40, −0.075); groundedness 1.975 → 2.000. No quality gain established. |
| Claude Haiku 5.5 · EL + estimated JEV | 906.15 → 710.43 −21.60% | $0.007754 → $0.006765 | $0.007754 → $0.009965 +$0.002210 | No quality gain established. |
| Claude Sonnet 5.5 · EL + estimated JEV | 822.75 → 631.68 −23.22% | $0.121724 → $0.103798 | $0.121724 → $0.106998 | No quality gain established. |
| Claude Opus 5.5 · EL + estimated JEV | 915.75 → 723.00 −21.05% | $0.317848 → $0.280656 | $0.317848 → $0.283856 | No quality gain established. |
JEV cost on the “estimated JEV” rows is a list-price estimate, not an invoice. The “no JEV” row made no JEV call, so its system cost equals measured LLM cost.
Limitations
- Developer-authored relevance judgments and required facts; automated OpenAI judge, not human review.
- The exposed split is not blind.
- Anthropic profiles use adaptive thinking / medium effort / 8192 cap, unlike Phase 7's 800 total output cap.
- JEV costs are estimates; LLM costs are measured from provider usage.