feat(bench): composite guards — efficiency, consistency, canary - #693
Conversation
vouch scored a mean over per-category means and nothing else. a retrieval strategy can score well on category means while being right by luck: brittle under paraphrase, or dragging in half the kb to reach the answer. the koth ladder measures strategies against each other, so an unguarded score is exactly what a ladder optimises into. the compatibility problem the issue raises is real: changing the composite formula makes every recorded score and ladder entry incomparable, which is a season-boundary decision, not a merge-day one. so the guards are reported *beside* the composite rather than inside it. `composite` keeps its exact formula and meaning; `composite_guarded` is the new measurement; and `bench_version` on every report is what stops the two being compared by accident. the ladder adopts the guarded number when a maintainer decides, not when this merges. run_seeds also degrades cleanly on a version-1 report with no guard block. efficiency is a bounded penalty on how much budget the answers consumed, floored so a wasteful run is penalised and never zeroed — it is a tiebreak, not a verdict. consistency is metamorphic: each pack-graded case is re-asked through a template paraphrase and must grade the same. template-driven, never llm-written, so a run stays reproducible from the seed alone. canary plants a coined value that answers no question; any pack surfacing it halves the composite. the wording took two passes. the first draft read "the retired access code was ..." and tripped on "what was the project codename before it changed?" — lexical overlap on code/retired, not a dump, so the guard was measuring the generator's word choice. a test now asserts the bait shares no vocabulary with any generated question. the binary multiplier is joined by a leak rate, because "one stray pack in twenty-six" and "half of them" are different engineering problems and a 0.5 that cannot tell them apart is not actionable. measured over seeds 1-3: efficiency 0.88, consistency 1.00, canary 0.50 — tripped on every seed, leak 0.08-0.24. a ten-item pack over this corpus carries about a quarter of it, which is the lever the guard exists to expose. consistency at 1.00 is a floor rather than proof: prefix-form paraphrases keep every content word, so they do not move lexical retrieval. both numbers are in the reference table rather than tuned away. Closes vouchdev#616
github-actions
Bot
added
docs
|
reporting the guards beside the composite rather than folding them in is the right resolution of the compatibility problem, and the canary is the finding and you are right to lead with it. a 10-item pack carrying ~24% of a 38-claim corpus is a measurement of the corpus as much as the ranker, and saying so rather than shipping 0.5 as a quality signal is the honest read. the two self-criticisms are worth more than the numbers. consistency at 1.00 being a floor because prefix-wrapper paraphrases leave every content word intact is a real limitation, and the reasoning for not fixing it with a content-rewriting paraphrase — that it would measure the rewriter instead of the ranker — is correct. same for the one thing i would want stated explicitly is the exit condition.
|
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # CHANGELOG.md
vouchdev#686 added coerce_numeric and routed capture.py's two boolean fields through coerce_bool, but left min_observations and dedup_window_seconds on bare int()/float(). a typo'd value raised out of load_config instead of falling back to the default, which is the exact case the helper's own docstring cites (`min_observations: "three"`), and the resulting unused import tripped ruff F401. surfaced by merging test into this branch: the branch-push workflows on test don't run pytest/mypy/ruff, so the gate never ran on the merge that landed it. Co-authored-by: Cursor <cursoragent@cursor.com>
changelog union only; no source conflict.
|
diff coverage: n/a — this PR changes no python under |
2 participants
Closes #616
vouch scored a mean over per-category means and nothing else. a strategy can score well on category means while being right by luck — brittle under paraphrase, or dragging in half the KB to reach the answer. the ladder measures strategies against each other, so an unguarded score is exactly what a ladder optimises into.
the compatibility problem, solved by not forcing the decision
the issue is blunt that this is the hard part:
season 1 is live (#573), and forcing a boundary is a maintainer call. so the guards are reported beside the composite rather than inside it:
compositekeeps its exact definition, so no recorded score or ladder entry becomes incomparable andpaired_verdictkeeps working untouched.bench_versionis what stops a guarded score being compared to a legacy one by accident. the ladder adoptscomposite_guardedwhen you decide, not when this merges.run_seedsalso degrades cleanly on a version-1 report with no guard block — pinned by a test, because that is the exact scenario the stamp exists for.test_composite_keeps_its_original_meaningrecomputes the plain mean and asserts equality, so a later refactor cannot quietly fold the guards in.the numbers, reported rather than tuned
over seeds 1-3:
the canary is the finding. the planted line reaches a pack because a 10-item pack over this corpus carries ~24% of it (9 of 38 live claims, measured). that is the lever the guard exists to expose, and the reason
composite_guardedis not yet the ladder's number.consistency at 1.00 is a floor, not proof. my paraphrases are prefix wrappers (
"quick one — {q}"), so every content word survives and lexical retrieval is unmoved. a content-rewriting paraphrase would test harder but risks changing the question, which would make the guard measure the rewriter instead of the ranker. i took the safe form and said so in the reference table rather than claiming robustness i had not demonstrated.the canary took two passes, and the second is a test
the first draft read "the retired access code was ..." and tripped on "what was the project codename before it changed?" — lexical overlap on code/retired, not a dump. the guard was measuring my word choice.
test_canary_shares_no_vocabulary_with_any_questionnow asserts the bait shares no 4+ letter token with any generated question, so that class of miscalibration cannot come back.the binary multiplier is joined by
canary_leak_ratebecause "one stray pack in twenty-six" and "half of them" are different engineering problems, and a 0.5 that cannot tell them apart is not actionable.what did not change
categories 1-10 still generate byte-identically — the canary is planted on the derived rng established in #634, and that property is still pinned.
compositeover seeds 1,2,3 measures 0.6429, exactly its pre-change value.tests
18 new cases in
tests/test_bench.py: the version stamp,compositekeeping its formula,composite_guardedbeing the product, guards reported separately; efficiency ordering/bounds/degenerate inputs; paraphrase determinism and the consistency arithmetic including a forced failure path (stock retrieval never disagrees with itself, so the drop is forced with a destructive rewrite); canary distinctness per seed, presence-in-corpus-but-answers-nothing, the vocabulary guard, and the leak rate agreeing with the binary trip; andrun_seedsreporting both composites plus tolerating a legacy report.verification
the reference baseline table in the module docstring carries both numbers and both caveats.