You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
flaky: usage-projection e2e rebuilt-history test times out its 60 s wait on a slow runner #1720
e2e/usage-projection.test.ts, "a history.db whose projection table lost a column while the daemon was stopped is rebuilt at the next start, a lease is granted meanwhile, and simlock status prints the rebuilt line" fails with Error: waitFor timed out after 60000ms (history rebuilt) on a slow runner. On PR #1704 (run 44299714535) it failed twice, then passed on the third attempt with no code change; #1716, which holds the same code, passed it. On main CI the test already takes 44 to 47 s of its 60 s wait (runs 44289531652, 44287129321, 44277559942); the failing run took 81 s.
Confirmed. The bug is in the test; the product is correct. There is no new test and no bug/1720-repro branch: the reproduction is the existing test, "a history.db whose projection table lost a column … prints the rebuilt line" (e2e/usage-projection.test.ts:158), run on a loaded host. It failed twice on unchanged code while this Mac's load average was about 700. It failed once more under taskpolicy -b (background priority).
pnpm test:e2e e2e/usage-projection.test.ts -t "lost a column"
Error: waitFor timed out after 60000ms (history rebuilt)
❯ e2e/usage-projection.test.ts:185:5
CI run 44299714535, attempts 1 and 2: same error, test took 83.7 s / 81.3 s
Root cause
Defect in the test. e2e/usage-projection.test.ts:12 seeds SEEDED = 300_000 events so the rebuild is still running when the second lease is granted. e2e/usage-projection.test.ts:187 then gives the whole rebuild a fixed 60 s on the real clock. The rebuild is CPU-bound and scales with the runner. Here it takes about 11 s, on main CI 44 to 47 s, and over 60 s on a loaded shard. The rebuild itself is fine: logged per batch, it grows only from about 40 ms to 55 ms, not quadratic. The other side of the race has a wide margin: the meanwhile status comes about 0.5 s after the start.
Simplest fix
e2e/usage-projection.test.ts:12: SEEDED = 100_000. Keep the 60 s wait.
SEEDED=100_000, full file, twice: Tests 4 passed (4); this test 4.5 s / 4.2 s (was 18.3 s)
taskpolicy -b, this test, twice: 1 passed (69.2 s / 73.1 s)
quiet runs: meanwhile 0.45–0.5 s after start, rebuilt 1.2–2.0 s after start
Alternatives rejected
Raise the 60 s wait: it adds CI time and still has no bound on a loaded shard.
A test-only hook that holds the rebuild until the lease is granted: it removes the race, but it is a new daemon seam for one test.
Risk
The headroom moves to the other side. On a fast host the rebuild ends about 1 to 1.5 s after the meanwhile status. A spawn of the CLI that is slower than that would fail not.toContain("History: figures were rebuilt").
It is still a real-time race in an e2e test (testing rule 5). The fix gives more headroom; it does not make the test deterministic.
Fast track
Defect: yes
Fix tried: yes
Nothing public added or broken: yes
No ADR or rule changed: no — the test still races the real clock (testing rule 5)
Maintainer decision on the triage report (2026-11-20):
Do not lower the seed to 100,000. Make the test deterministic instead: add a test-only hook that holds the history rebuild until the second lease is granted, and seed a small count (about 1,000 events). This adds one daemon test seam, which is accepted.
Activity
V3RON commentedon Nov 18, 2026
Reproduction
Confirmed. The bug is in the test; the product is correct. There is no new test and no
bug/1720-reprobranch: the reproduction is the existing test, "a history.db whose projection table lost a column … prints the rebuilt line" (e2e/usage-projection.test.ts:158), run on a loaded host. It failed twice on unchanged code while this Mac's load average was about 700. It failed once more undertaskpolicy -b(background priority).Root cause
Defect in the test.
e2e/usage-projection.test.ts:12seedsSEEDED = 300_000events so the rebuild is still running when the second lease is granted.e2e/usage-projection.test.ts:187then gives the whole rebuild a fixed 60 s on the real clock. The rebuild is CPU-bound and scales with the runner. Here it takes about 11 s, onmainCI 44 to 47 s, and over 60 s on a loaded shard. The rebuild itself is fine: logged per batch, it grows only from about 40 ms to 55 ms, not quadratic. The other side of the race has a wide margin: themeanwhilestatus comes about 0.5 s after the start.Simplest fix
e2e/usage-projection.test.ts:12:SEEDED = 100_000. Keep the 60 s wait.Alternatives rejected
Risk
meanwhilestatus. A spawn of the CLI that is slower than that would failnot.toContain("History: figures were rebuilt").Fast track
Written by an agent.
V3RON commentedon Nov 20, 2026
Maintainer decision on the triage report (2026-11-20):
Do not lower the seed to 100,000. Make the test deterministic instead: add a test-only hook that holds the history rebuild until the second lease is granted, and seed a small count (about 1,000 events). This adds one daemon test seam, which is accepted.
Written by an agent.
fix(daemon): the history rebuild hold ends when the history is stopped (
V3RON commentedon Nov 20, 2026
Delivery run
Merged: #1792 (#1720)
Parked: none
Waiting: none
Decided: none
Written by an agent.
fix(history): hold the history rebuild in the usage-projection e2e in…