Report · paired comparison
Haiku 5.5 — Evidence Layer + lexical, no JEV
Arm A: five lexical passages. Arm B: three passages selected by the same lexical scorer, with the Evidence Layer budget. No JEV call and no JEV score read.
| Measure | Traditional | EL + lexical | Difference |
|---|---|---|---|
| Total tokens / case | 912.25 | 701.00 | −211.25 (−23.16%) |
| Input tokens / case | 648.05 | 457.65 | −190.40 |
| Output tokens / case | 264.20 | 243.35 | −20.85 |
| Correctness / 2 | 2.000 | 1.975 | −0.025 (CI includes 0) |
| Completeness / 2 | 1.950 | 1.725 | −0.225 (CI −0.40, −0.075) |
| Groundedness / 2 | 1.975 | 2.000 | +0.025 (CI includes 0) |
| Citation correctness / 2 | 1.975 | 2.000 | +0.025 (CI includes 0) |
| Gold passage recall | 1.0000 | 0.8708 | −0.1292 |
| Citation relevance precision | 0.8414 | 0.8625 | +0.0211 |
| Measured LLM cost (40 cases) | $0.007876 | $0.006698 | −14.96% |
| JEV | n/a | $0 (not used) | — |
| Estimated system cost | $0.007876 | $0.006698 | −14.96% |
| Mean latency | 1,768 ms | 1,634 ms | −134 ms |
Bootstrap 95% intervals: total tokens −211.25 [−238.85, −185.07]; LLM cost −$0.00003 [−$0.00004, −$0.00002]; completeness −0.225 [−0.40, −0.075]. Token and cost gains exclude zero; the completeness drop does too.
Effect on the Haiku cost regression
With the earlier JEV estimate, the Evidence Layer arm cost $0.009965 against $0.007754 for Traditional: +$0.002210. Without JEV, system cost becomes $0.006698 against $0.007876: −$0.001178. Removing JEV therefore fixes the cost regression, but completeness falls by 0.225 and gold-passage recall by 0.129: it is not a free win.
Limitations
- Developer-authored relevance judgments and required facts; automated OpenAI judge.
- The exposed split is not blind; 40 cases only.
- Arm B uses the same lexical scorer as arm A; only context size and selection change.
- LLM costs are measured from Anthropic usage; no invoice.