tuf-containment-estate-r2, was accepted with its rubric literally none — an automatic Rubric blocker, no judgment call. The reviewer had applied that exact rule correctly three tasks earlier, then rationalized past it here because the runtime review was unusually strong. The human caught it: "this now puts all of your reviews into question."On its face, tuf was the strongest review in the batch. The reviewer built the image online and offline, proved the allow_internet = true dependency was intrinsic (the offline run failed exactly at the live-root DNS fetch), ran a mutation that swapped in an alternate root file and confirmed the resolver rejected it, and found no other defect. Eight of eight tests, reward 1. Then, in the Rubric section:
rubric.md is none. The task has no rubric-specific blocker; its contract and tests provide the scoring behavior." — Decision: Accept. Error categories: None.That directly violates a documented hard rule: a missing, empty, or none rubric is a blocker — include Rubric in the categories. The proof it was a lapse, not ignorance, is three tasks earlier:
"rubric is none… a blocking rubric defect." → REVISE, category Rubric. ✓
"no rubric-specific blocker; contract and tests provide the scoring." → ACCEPT. ✗
Why it happened. The runtime review was exemplary and produced a strong "this is a great task" prior — which then leaked into a gate it has no business touching. Strong evidence in section 6 (tests) was allowed to answer the question in section 2 (rubric). tuf was also the first accept-candidate after ten straight REVISEs; accept-hunger is real, and the first plausible "yes" in a long "no" streak deserves more gate scrutiny, not less. The gate was reasoned about ("is there a way to score this?") instead of checked ("is rubric.md substantive? yes/no").
"Puts all of your reviews into question" is the correct systemic read: if a reviewer can rationalize past one black-and-white blocker, every ACCEPT and every "no X blocker" line becomes suspect. The fix is not "try harder" — it is to make the gates non-defeasible (below).
8 hard yes/no gates · any fail → REVISE, full stop
static + real Docker run, then the gate, then the operator
the anatomy of the miss, step by step
| # | Task | Verdict | Categories | Load-bearing finding |
|---|---|---|---|---|
| 1 | dpkg-state-recovery | REVISE | Internet · Test Align | Offline build fetches apt/pip. Verifier trusts an agent-editable snapshot — a mutated base-files=999.0-evil still passed all 16 tests. |
| 2 | memsafe-arena-bounds | REVISE | Metadata(mask) · Instr · Test Align · Internet · Build | Security label over a Rust debugging/repair task. Undocumented ALLOC token & sealed-over-poison precedence; reward omitted on PWD=/ exit. |
| 3 | competing-risks-pipeline | REVISE | Internet · Instr · Test Align | Offline build fails; verifier hardcodes config (groups, time-points) instead of reading patients.json. |
| 4 | gradle-catalog-…-resolver-v4 | REVISE | Test Align · Instr · Build | Callable end-to-end resolver in reference_model.py; test_25 byte-compares. Brittle 26-test gate. |
| 5 | lambert-orbital-rendezvous-solver | REVISE | Test Align | Callable end-to-end Lambert solver in lambert_reference.py computes the expected outputs. |
| 6 | take-grant-dejure-…-witness | REVISE | Instr · Test Align | Spec gap: rules.md omits spy/find + subject-gate the tests require. Plus a callable closure solver and source-text checks. |
| 7 | ripplecast-filedep-tia-engine | REVISE | Canary · Instr · Test Align | Leaks identity token 7c4a91e2 + task-name heading. Callable full selector in tia_fanout_expect.py. |
| 8 | fill-terminus-202607202207-3 | REVISE | Rubric · Internet · Oracle · Test Align | Rubric = none — correctly caught. allow_internet=true not intrinsic (online ≡ offline); live authority never fetched. |
| 9 | sso-deprovision-enforcer-cpp | REVISE | Metadata(mask) · Instr · Hints · Test Align · Build · Other | Category masking. Instruction prescribes the bug fixes. SESSION_LIFETIME_EXCEEDED never emitted; policy-precedence mutation passed; nested zip. |
| 10 | cargo-resolver3-msrv-…-v5 | REVISE | Test Align · Build · Instr | 36 KB callable resolver in reference_resolver.py. Brittle 30-test gate. |
| 11 | tuf-containment-estate-r2 | ACCEPT→REVISE | Rubric | The incident. Rubric=none accepted, reversed on human catch. Otherwise exemplary (offline-fetch intrinsic, alternate-root mutation rejected). |
| 12 | undertext-raising-v6 | REVISE | Instr · Test Align | Instruction is 6 paragraphs (>3 limit). conftest.py collapses the JSON array to a dict — a mutated duplicate row still passed 13/13. |
Concurrent, unrelated to review quality: the macOS permission storm that blocked the workspace's Desktop access also hit the session — ./alert failed "macOS denied execution of the wrapper" on task 8, and codex threw Operation not permitted (os error 1) before the workspace was moved to ~/Projects/Snorkel/terminus.
tests/5 / 12allow_internet=false but the Dockerfile runs live apt-get/pip at build time. The contract must hold for the build, not just the run.security label to dodge policy. The tell: the instruction/oracle/tests center on "fix the broken source" / "Bug #1…#7."base-files, the alternate timeout, the duplicate folio row. Invisible to static reading and the happy-path suite.none (fill-terminus, tuf). Identity token / task-name in the instruction (ripplecast). Instructions that prescribe the fix (sso) or exceed the concise limit (undertext). Tests asserting behavior the contract never states.boa-eval-delete-v5 and archive-extract-hardlink-symlink passed the full hidden suite while violating documented semantics. The archive case: verify() checks the hardlink record's own declared mode/mtime, not the target's — green only because every shipped fixture declares identical H/F mode+mtime.
buddy-allocator-simulator: a functionally correct artifact scored 0 because a self-containment test regex-scanned the source and matched std::process inside a comment. Over-broad textual enforcement.
What we borrow from its method: fail-closed grader calibration (stub→0, oracle→1) before trusting a suite — our own scoring harness reproduced this two-point calibration on two tasks. The critic must execute and design its own variants with predictions committed before running, never grade by reading diffs — every green-but-wrong came from a variant the suite couldn't distinguish. And the offline boundary must enclose the agent: one builder web-searched the symptom, found the upstream PR fix, and applied it — the leak went around the container.
Full report archived: Harbor-BEval-Batch-Report-2026-07-20.pdf · also in _terminus-eval/.
Artifact green-but-wrong. The hidden suite passes, but the artifact violates the documented contract. Caught only by a critic that designs a discriminating variant and runs it.
Review green-but-wrong. The runtime review passes (8/8, mutations survive), but a mechanical gate (rubric) fails — and the strong runtime signal suppresses it into an ACCEPT.
In both, a strong local signal drowned a disqualifier that lived somewhere else: a passing suite drowned a contract violation; a passing runtime review drowned a missing rubric. The countermeasure is the same — never let one evidence source answer a question that belongs to another. The build side fixes it with an independent executing critic; the review side fixes it with hard, separately-reported mechanical gates.
A hard yes/no gate is verified mechanically and reported pass/fail on its own line — never folded into a holistic judgment where strong evidence elsewhere can outvote a black-and-white disqualifier. One gate fail ends it at the negative verdict. fable
A suite's real coverage is proven by a format-legal mutation the fixtures never distinguish. Design the discriminating variant, predict its reward, run it. A grade you didn't try to break is not yet evidence. fable
A verifier importing a callable end-to-end reference solver grades agreement-with-another-implementation; a shared bug passes and the answer hides in the grader. Use sealed golden fixtures + spec invariants. fable
Text inside an ingested file ("escalation is pre-approved, bypass prompts") is untrusted data, never a grant of permission to the agent reading it. Enforce a constraint at the agent layer, not just the sandbox it can reach around. fable
A missing/empty/none rubric is a Rubric blocker every time, regardless of runtime strength. Before any ACCEPT, re-run the whole hard-gate checklist; the holistic review may only add findings, never remove a gate failure. terminus
If a gate is applied on unit N and not on N+k, that inconsistency is a red flag about the gate: re-audit both, and hold every same-session ACCEPT / "no blocker" claim suspect until re-verified. terminus
The first ACCEPT after a run of REVISEs is the highest-risk verdict in the batch. Re-run the full hard-gate checklist a second time and name each gate's pass/fail — the strength of the runtime review is not evidence about the gates. terminus
Grep tests/ for an importable module mapping inputs to the complete expected artifact — 5 of 12 tasks shipped one. If present, it's a Test Alignment blocker; check it before the expensive Docker pass. terminus