Skip to content

flaky: usage-projection e2e rebuilt-history test times out its 60 s wait on a slow runner #1720

Closed

Description

@V3RON

e2e/usage-projection.test.ts, "a history.db whose projection table lost a column while the daemon was stopped is rebuilt at the next start, a lease is granted meanwhile, and simlock status prints the rebuilt line" fails with Error: waitFor timed out after 60000ms (history rebuilt) on a slow runner. On PR #1704 (run 44299714535) it failed twice, then passed on the third attempt with no code change; #1716, which holds the same code, passed it. On main CI the test already takes 44 to 47 s of its 60 s wait (runs 44289531652, 44287129321, 44277559942); the failing run took 81 s.

Written by an agent.

Activity

  1. added
    bug:newReported. An agent may triage it (rule 8); the fast track may set it ready (rule 17).
    flaky-testNames a test that fails without a code change. Delivery skips its base-commit check.
    on Nov 18, 2026
  2. self-assigned this
    on Nov 18, 2026
  3. V3RON commented on Nov 18, 2026

    @V3RON
    ContributorAuthor

    Reproduction

    Confirmed. The bug is in the test; the product is correct. There is no new test and no bug/1720-repro branch: the reproduction is the existing test, "a history.db whose projection table lost a column … prints the rebuilt line" (e2e/usage-projection.test.ts:158), run on a loaded host. It failed twice on unchanged code while this Mac's load average was about 700. It failed once more under taskpolicy -b (background priority).

    pnpm test:e2e e2e/usage-projection.test.ts -t "lost a column"
    Error: waitFor timed out after 60000ms (history rebuilt)
     ❯ e2e/usage-projection.test.ts:185:5
    CI run 44299714535, attempts 1 and 2: same error, test took 83.7 s / 81.3 s
    

    Root cause

    Defect in the test. e2e/usage-projection.test.ts:12 seeds SEEDED = 300_000 events so the rebuild is still running when the second lease is granted. e2e/usage-projection.test.ts:187 then gives the whole rebuild a fixed 60 s on the real clock. The rebuild is CPU-bound and scales with the runner. Here it takes about 11 s, on main CI 44 to 47 s, and over 60 s on a loaded shard. The rebuild itself is fine: logged per batch, it grows only from about 40 ms to 55 ms, not quadratic. The other side of the race has a wide margin: the meanwhile status comes about 0.5 s after the start.

    Simplest fix

    e2e/usage-projection.test.ts:12: SEEDED = 100_000. Keep the 60 s wait.

    SEEDED=100_000, full file, twice: Tests 4 passed (4); this test 4.5 s / 4.2 s (was 18.3 s)
    taskpolicy -b, this test, twice: 1 passed (69.2 s / 73.1 s)
    quiet runs: meanwhile 0.45–0.5 s after start, rebuilt 1.2–2.0 s after start
    

    Alternatives rejected

    • Raise the 60 s wait: it adds CI time and still has no bound on a loaded shard.
    • A test-only hook that holds the rebuild until the lease is granted: it removes the race, but it is a new daemon seam for one test.

    Risk

    • The headroom moves to the other side. On a fast host the rebuild ends about 1 to 1.5 s after the meanwhile status. A spawn of the CLI that is slower than that would fail not.toContain("History: figures were rebuilt").
    • It is still a real-time race in an e2e test (testing rule 5). The fix gives more headroom; it does not make the test deterministic.

    Fast track

    • Defect: yes
    • Fix tried: yes
    • Nothing public added or broken: yes
    • No ADR or rule changed: no — the test still races the real clock (testing rule 5)
    • One fix: yes

    Written by an agent.

  4. removed their assignment
    on Nov 18, 2026
  5. V3RON commented on Nov 20, 2026

    @V3RON
    ContributorAuthor

    Maintainer decision on the triage report (2026-11-20):

    Do not lower the seed to 100,000. Make the test deterministic instead: add a test-only hook that holds the history rebuild until the second lease is granted, and seed a small count (about 1,000 events). This adds one daemon test seam, which is accepted.

    Written by an agent.

  6. removed
    bug:newReported. An agent may triage it (rule 8); the fast track may set it ready (rule 17).
    on Nov 20, 2026
  7. self-assigned this
    on Nov 20, 2026
  8. V3RON commented on Nov 20, 2026

    @V3RON
    ContributorAuthor

    Delivery run

    Merged: #1792 (#1720)
    Parked: none
    Waiting: none
    Decided: none

    Written by an agent.

  9. added a commit that references this issue on Nov 22, 2026
    2addc64
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bug:readyAn agent may fix it.flaky-testNames a test that fails without a code change. Delivery skips its base-commit check.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions