update / state snapshot
SEAM — Where We Are
Snapshot 2026-07-10 · HISTORY#374 · 374 entries across 85 days of development. Measurement integrity landed; the score picture is honest; the remaining lever is answerer strategy.
cat1 / cat3 → target 0.80
What happens to 82 holdout questions
Evidence classifier (PR#371) on the corrected holdout baseline — the first honest inventory of where answers land.
Key insight: the old v1 classifier claimed 33 answerer_miss cases. evidence/1 finds zero — every non-correct case had gold scattered across turns or was open-domain.
cat1/cat3 → 0.80 program
Evidence classifier (evidence/1) — fixed crude context_recall precision + the ≥0.5 failure classifier. v1 fields preserved byte-identical.
Judge/2 prompt fixes alias/abbrev under-scoring + groundedness axis. Replay harness for 82 stored answers (~$0.007 max). Judge version plumbing fixed in #374.
The actual lever. All prior answerer work was diagnostics. This is where the score has to move.
2 of 3 PRs landed · zero paid spend so far · all paid calls operator-gated
Tried & exhausted vs still open
| Lever | Result | Verdict |
|---|---|---|
| Retrieval tuning | recall barely moves compact→broad | EXHAUSTED |
| Prompt v1 (include all) | +0.041 cat1 / +0.027 cat3 dev; over-gen on holdout | MIXED |
| Stronger answerer (DeepSeek-v4-pro) | cat1 0.689 vs anchor 0.705 — noise margin | WASH |
| Prompt v2 (precision) | 0/17 cases changed at all | NULL |
| Incomplete-search fix | Only confirmed fixable pattern (2/11 cases) | OPEN |
4 of 5 meaningful levers exhausted. Only incomplete-search + the strategic 0.80-achievability question remain.
Holdout + project stats
Post-measurement correction
What we thought (v1)
- 33 answerer_miss cases — answerer failing to use available evidence
- cat3 is “generation-bound” — retrieval fine, answerer weak
- Fix answerer prompt/model → scores go up
- Context_recall ≥ 0.5 = gold is available
What we know now (evidence/1)
- Zero clean answerer misses — gold scattered across turns
- Gold-label incompleteness is real
- Context_recall false-positives on generic tokens
- Multi-turn entity coreference: same person → different entity ID per turn
Classifier shift (of 82 cases)
The “answerer-bound” framing rested on classifier false confidence. Real bottleneck: scattered evidence, gold incompleteness, task-design tension.
What’s next
A. Complete the 3-PR program
Finish PR#136 merge → paid rejudge with judge/2 (~$0.007) → design PR 3 answerer strategy from corrected evidence. Highest EV, lowest cost.
B. Manual classify remaining 44
Only 11/44 rigorously hand-checked. Completing this sharpens the fixable-error rate and ceiling before more spend. ~2–4h operator time, $0.
C. Incomplete-search pattern fix
Only confirmed fixable pattern (2 of 2 real errors): multi-pass/re-scan or structural entity-grounded scoring. Unscoped, uncosted.
D. Fix failure_class threshold
context_recall ≥ 0.5 is too loose. Affects trust in miss counts on every future run. Small, free, defensive.
E. Is 0.80 achievable here?
Given gold incompleteness and multi-turn scatter, can answer-quality tuning alone reach 0.80 — or does the path need per-turn entity resolution first? Operator call.
HISTORY entries · last 30 days
2026-07-08 spike = paid diagnostic + DeepSeek infra + run-record + numpy cache in one session. Steady 3–5 entry/day otherwise.
What we’ve been working on
Top topics by HISTORY entry count. Recent shift toward judge / answerer / measurement.