Skip to content

fix(sampling): use an unbiased, seedable shuffle for random question selection - #86

Open
Agnik47 wants to merge 1 commit into
supermemoryai:mainfrom
Agnik47:fix/uniform-random-sampling
Open

fix(sampling): use an unbiased, seedable shuffle for random question selection#86
Agnik47 wants to merge 1 commit into
supermemoryai:mainfrom
Agnik47:fix/uniform-random-sampling

Conversation

@Agnik47

@Agnik47 Agnik47 commented Aug 14, 2026

Copy link
Copy Markdown

Fixes #72.

Problem

Random sampling shuffled with an inconsistent comparator:

// src/orchestrator/index.ts:56 — and again, verbatim, in src/orchestrator/batch.ts:74
const shuffled = [...questions].sort(() => Math.random() - 0.5)
selected.push(...shuffled.slice(0, sampling.perCategory).map((q) => q.questionId))

The comparator ignores its arguments, so it violates the consistency Array.prototype.sort requires and the resulting permutation is engine-dependent. In practice elements stay near their original positions far more often than chance — and since the sampler then takes the first N of the shuffled array, the bias lands squarely on the selected slice.

Measured over 12,000 trials on a 6-element array (expected 2,000 per element/position cell):

min cell max cell
sort(() => Math.random() - 0.5) 1,250 3,041
Fisher–Yates 1,929 2,078

One element held its original position 25.3% of the time against an expected 16.7%. So --sample-type random kept picking the same front-of-dataset questions, and a sample's accuracy was not an unbiased estimate of the full-run accuracy it stands in for. There was no seed either, so the selection was neither uniform nor reproducible.

The same code existed in two places — the single-run orchestrator and the batch compare path — so fixing one would have left the other biased.

Fix

  • New src/orchestrator/sampling.ts with a Fisher–Yates shuffle driven by a seedable mulberry32 PRNG, plus the shared selectQuestionsBySampling. Both call sites now use it, removing the duplicated logic.
  • --seed N on run and compare reproduces a previous selection. Invalid seeds are rejected at parse time rather than surfacing mid-run.
  • When no seed is given, one is generated, logged (Random sampling seed: 1234 (reuse with --seed 1234)) and stored in SamplingConfig.seed — in the run checkpoint and in the compare manifest — so a past sampled run can be re-selected question-for-question.
  • Category buckets moved from a plain object to a Map, so integer-like question types keep insertion order instead of being hoisted to the front.

Behaviour is unchanged for mode: "full", mode: "limit" and sampleType: "consecutive".

Tests

New src/orchestrator/sampling.test.ts (16 tests, bun test):

  • every element lands in every position equally often (max deviation < 150 over 12,000 seeded trials, ~3.7σ — deterministic, so it cannot flake)
  • a characterisation test pinning the bias of the sort(() => Math.random() - 0.5) it replaces
  • same seed reproduces the same subset in the same order; different seeds diverge
  • a run given no seed reports one that replays its exact selection
  • across 200 seeds every question in the pool is reachable — the front-loading is gone
  • full / limit / consecutive selection, undersized categories, empty and single-element inputs, PRNG range and reproducibility

Reverting shuffle to the old expression fails 3 of them; all 16 pass with the fix.

bunx tsc --noEmit is clean and both new files are Prettier-clean. The five touched files already fail bun run format:check on main (as do ~69 others), so I left their pre-existing formatting alone to keep the diff scoped.

…selection

Random sampling shuffled with `[...questions].sort(() => Math.random() - 0.5)`.
The comparator ignores its arguments, which violates the consistency `sort`
requires, so the permutation is engine-dependent and leaves elements near their
original positions far more often than chance. Since the sampler then takes the
first N per category, that bias lands exactly on the selected slice: over 12,000
trials on a 6-element array, one element held its original position 25.3% of the
time against an expected 16.7%. A "random sample of 10 per category" kept
favouring the same front-of-dataset questions, so a sample's accuracy was not an
unbiased estimate of the full run.

Replace it with Fisher-Yates driven by a seedable mulberry32 generator. A seed
can be supplied with `--seed`; when it is omitted one is generated, logged, and
recorded in the checkpoint (and in the compare manifest), so a sampled run is
reproducible after the fact instead of being neither uniform nor repeatable.

The selection logic was duplicated verbatim in the orchestrator and the batch
compare path — fixing one would have left the other biased — so it now lives in
`src/orchestrator/sampling.ts` and both call it.

Fixes supermemoryai#72

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nwg8d2HEScsHTWpBgWewWX
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Random sampling uses sort(() => Math.random() - 0.5), which is a biased shuffle — sampled runs are not representative

1 participant