fix(sampling): use an unbiased, seedable shuffle for random question selection - #86
Open
Agnik47 wants to merge 1 commit into
Open
fix(sampling): use an unbiased, seedable shuffle for random question selection#86Agnik47 wants to merge 1 commit into
Agnik47 wants to merge 1 commit into
Conversation
…selection Random sampling shuffled with `[...questions].sort(() => Math.random() - 0.5)`. The comparator ignores its arguments, which violates the consistency `sort` requires, so the permutation is engine-dependent and leaves elements near their original positions far more often than chance. Since the sampler then takes the first N per category, that bias lands exactly on the selected slice: over 12,000 trials on a 6-element array, one element held its original position 25.3% of the time against an expected 16.7%. A "random sample of 10 per category" kept favouring the same front-of-dataset questions, so a sample's accuracy was not an unbiased estimate of the full run. Replace it with Fisher-Yates driven by a seedable mulberry32 generator. A seed can be supplied with `--seed`; when it is omitted one is generated, logged, and recorded in the checkpoint (and in the compare manifest), so a sampled run is reproducible after the fact instead of being neither uniform nor repeatable. The selection logic was duplicated verbatim in the orchestrator and the batch compare path — fixing one would have left the other biased — so it now lives in `src/orchestrator/sampling.ts` and both call it. Fixes supermemoryai#72 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nwg8d2HEScsHTWpBgWewWX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #72.
Problem
Random sampling shuffled with an inconsistent comparator:
The comparator ignores its arguments, so it violates the consistency
Array.prototype.sortrequires and the resulting permutation is engine-dependent. In practice elements stay near their original positions far more often than chance — and since the sampler then takes the first N of the shuffled array, the bias lands squarely on the selected slice.Measured over 12,000 trials on a 6-element array (expected 2,000 per element/position cell):
sort(() => Math.random() - 0.5)One element held its original position 25.3% of the time against an expected 16.7%. So
--sample-type randomkept picking the same front-of-dataset questions, and a sample's accuracy was not an unbiased estimate of the full-run accuracy it stands in for. There was no seed either, so the selection was neither uniform nor reproducible.The same code existed in two places — the single-run orchestrator and the batch compare path — so fixing one would have left the other biased.
Fix
src/orchestrator/sampling.tswith a Fisher–Yatesshuffledriven by a seedable mulberry32 PRNG, plus the sharedselectQuestionsBySampling. Both call sites now use it, removing the duplicated logic.--seed Nonrunandcomparereproduces a previous selection. Invalid seeds are rejected at parse time rather than surfacing mid-run.Random sampling seed: 1234 (reuse with --seed 1234)) and stored inSamplingConfig.seed— in the run checkpoint and in the compare manifest — so a past sampled run can be re-selected question-for-question.Map, so integer-like question types keep insertion order instead of being hoisted to the front.Behaviour is unchanged for
mode: "full",mode: "limit"andsampleType: "consecutive".Tests
New
src/orchestrator/sampling.test.ts(16 tests,bun test):sort(() => Math.random() - 0.5)it replacesfull/limit/consecutiveselection, undersized categories, empty and single-element inputs, PRNG range and reproducibilityReverting
shuffleto the old expression fails 3 of them; all 16 pass with the fix.bunx tsc --noEmitis clean and both new files are Prettier-clean. The five touched files already failbun run format:checkonmain(as do ~69 others), so I left their pre-existing formatting alone to keep the diff scoped.