Skip to content

test(orchestrate): prove Orca retry fencing across two processes (T14494) - #2828

Open
kryptobaseddev wants to merge 3 commits into
mainfrom
task/T14494
Open

kryptobaseddev wants to merge 3 commits into
mainfrom
task/T14494

Conversation

@kryptobaseddev

@kryptobaseddev kryptobaseddev commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Adds a two-process test for the Orca retry reservation. Until now only an in-process race covered it (orca-worker-custody.test.ts), as the OpenCode review of #2803 pointed out. The test turned up one real defect (below), which this PR fixes.

Test

packages/core/src/orchestrate/__tests__/orca-retry-multiprocess.test.ts starts two separate node processes, each with its own SQLite connection to one temporary CLEO store. Both run the compiled retryOrcaWorker on the same failed Dispatch, released together by a barrier file. Each child injects an Orca runner that answers from recorded fixture replies and appends every worker-start to a shared launch log, so no real Orca worker ever starts. Every wait is bounded: 20 s per barrier, 45 s per child, 90 s per test.

  • Staged race (the proof): a new test-only seam, setRetryReservationProbe, runs inside recordWorker's reservation transaction, after the predecessor check and immediately before the retiring UPDATE (the metadataJson + stoppedAt CAS). Both processes park there and are released together. The test asserts exactly one UPDATE winner, one launch from the winner's pid, one open successor and a retired predecessor. The loser must get exactly Orca retry reservation lost; no new launch is authorized.
  • Free races (3 rounds): both processes retry at the same moment. The loser must get one of the documented custody refusals, now including settlement's Orca resource observation changed; reload the exact worker. (review point 2). Exactly one launch is still required.

Defect found and fixed

With both processes parked at the CAS, the loser did not get a documented refusal. Its UPDATE failed with a raw Failed query: update "tasks_agent_instances"… error, caused by ERR_SQLITE_ERROR 517 database is locked (SQLITE_BUSY_SNAPSHOT). The reservation is a deferred transaction, and the loser's read snapshot predates the other process's write, so SQLite refuses the stale writer before the CAS predicate is evaluated. recordWorker now reports that conflict on the retry path as the documented Orca retry reservation lost refusal, keeping the original error as cause. The transaction itself is unchanged, and exactly one retry still launches.

Falsifier results (each change made temporarily in a scratch copy, then restored)

  • CAS predicate removed (metadataJson + stoppedAt; UPDATE keyed by id only): all 4 tests still pass. Across processes the predicate is unreachable, because WAL snapshot isolation refuses the stale writer first. Both processes can never hold the same pre-state at the CAS through the retry entry point, since each observation bumps the stored revision. The predicate is defence in depth; the cross-process fence is the transaction. The test file's header documents this.
  • CAS predicate and the in-transaction predecessor check both removed: all 4 tests still pass, for the same reason.
  • Snapshot-conflict mapping removed: the staged test fails, because the loser gets the raw SQLite error. So the test detects the fix.

Per-AC evidence

  • AC1 (a test runs two separate processes retrying one dispatch against one store and proves exactly one reservation wins): the staged test parks both processes at the reservation UPDATE and asserts exactly one wins; the free races cover the earlier fences. Locally pnpm exec vitest run packages/core/src/orchestrate/__tests__/orca-retry-multiprocess.test.ts gave Tests 4 passed (4) on 3 consecutive runs. orca-worker-custody.test.ts (31) and worker-records.test.ts (9) also pass. Gates 28, 35, 38 and 43 and the changeset lint pass.

Notes: local runs used a copy of the main checkout's core dist with worker-records.js transpiled from this branch's source, because no local build was allowed. CI runs against a fresh build. The suite is skipped with a reason when the dist is missing, as task-claims-multiprocess.test.ts is.

Closes: T14494 AC1

🤖 Generated with Claude Code

https://claude.ai/code/session_015tRfRTUirSrHNNTW4C8Hy9

kryptobaseddev and others added 2 commits November 12, 2026 00:28
…494)

Two separate node processes retry one failed Dispatch against one temporary
store with an injected Orca runner that logs launches to a shared file.
Free races and a staged race (both observe, then reserve together) each end
with exactly one reservation, one worker-start, and a fence refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015tRfRTUirSrHNNTW4C8Hy9
Add a test-only reservation probe that runs inside recordWorker's
transaction right before the predecessor-retiring UPDATE. The staged race
now parks both processes there and releases them together; exactly one
UPDATE wins. The stale one was refused by SQLite (SQLITE_BUSY_SNAPSHOT)
as a raw 'database is locked' error, so recordWorker now maps that to the
documented 'Orca retry reservation lost' refusal. Free races accept every
documented custody refusal, including settlement's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015tRfRTUirSrHNNTW4C8Hy9
Replace the inline asserted error type in isSqliteSnapshotConflict with
'errcode' in / typeof narrowing; behaviour unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015tRfRTUirSrHNNTW4C8Hy9

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant