handoff / adjudication / state snapshot
SEAM — Handoff Adjudication
HISTORY#377 · offline cat1/cat3 adjudication complete · PR #139 merged · PR #140 pending · $0.00 audit spend · 43-case private table · context handoff at 82.8% utilization
Offline audit · 29 classified cases
Private 43-case table (SHA-256 25336886…b13cc5). Each failure classified into one primary root cause. Nothing shipped — pure measurement.
Ceiling projections + session meter
cat1 · entity QA
All confirmed fixes reach 0.778689. All confirmed + all mixed reach 0.803279 with zero miss tolerance.
cat3 · inference
6 defensible inference targets, 8 defective/underspecified. All defensible fixes reach 0.690476.
Handoff triggered at 82.8% utilization. Successor has ~60,700 tokens of headroom.
14 inference targets · classified
Each cat3 failure was adjudicated for whether the expected inference is genuinely supported by the retrieved evidence or reflects an underspecified gold label.
Even if all 6 defensible targets were fixed perfectly, cat3 caps at 0.690. The remaining 8 are gold-label issues — not fixable via answerer or retrieval.
Merge state + next actions
PR #139 · handoff infrastructure
MERGED06140ead on main · context-handoff directory + auto-trigger wiring.
PR #140 · cat1/cat3 adjudication
PENDINGBranch agent/cat13-offline-adjudication at d18a594. CI running. CodeRabbit rate-limited (~44 min to free window). HISTORY#377 + aggregate audit doc.
@coderabbitai review
→
⬤ Squash-merge if clean
Next · handoff registry
PLANNEDNew branch after PR #140 merges. Tracked docs/handoffs/INDEX.md with latest pointer + supersedes chain + validation.
Never use auto-merge. Poll checks → plain --squash only when review + CI are clean.
Recommended minimal tracked protocol
Canonical superseding handoff location — found during normal startup, each handoff explicitly supersedes the prior one. Tracked in repo, not dated files scattered across directories.
📁 docs/handoffs/INDEX.md
- Explicit
latestpointer - Newest-first chain table
- Columns: handoff_id, path, supersedes, history#, status
🔗 Per-handoff metadata
- Every handoff carries
handoff_id+supersedes - Verifier rejects: missing targets, cycles, multiple live heads, index/latest mismatch
- Focused tests for the chain invariants
🧭 Startup routing
- AGENTS.md reads
docs/handoffs/INDEX.mdat session start - DATA_ROUTING.md routes handoff reads
- REPO_LEDGER.md records the protocol
📋 History contract
- HISTORY.md stays append-only
- New HISTORY entry for this protocol change
- First indexed handoff = successor to 2026-07-11 cat1/cat3 chain
First indexed handoff points at the exact next operator decision: product-correct raw+adjudicated reporting (recommended), benchmark-specific guessing, or a measurement-first adjudicated overlay.
What was adjudicated
Private table at /media/terrabyte/T7/Proprietary/DATA/20260711-locomo-cat1cat3-judge2-offline-adjudication.md. No paid calls authorized. Pure offline measurement — classify every failure, compute ceiling, surface the real bottleneck.
What the audit revealed
- Judge/gold is the dominant cat1 category — 13 of 29 cases. Gold-label issues, not answerer failures.
- Answerer fixes alone cap at 0.779 — even with perfect retrieval.
- cat3 is structurally capped at 0.690 — 8 of 14 targets are defective/underspecified in the gold labels themselves.
- Retrieval is NOT the bottleneck — only 5 retrieval-root-cause cases in cat1.
Next fork point
The handoff registry's first indexed entry will point at one of these three paths:
A. Product-correct reporting
Raw + adjudicated results, honest ceilings, no benchmark gaming. Recommended.
B. Benchmark-specific guessing
Target the known gold-label patterns. Moves scores but teaches to the test.
C. Measurement-first overlay
Build adjudicated classification into the eval harness itself. Deferred but durable.