Repository navigation
test(orchestrate): prove Orca retry fencing across two processes (T14494) - #2828
Open
kryptobaseddev wants to merge 3 commits into
Open
kryptobaseddev wants to merge 3 commits into
kryptobaseddev wants to merge 3 commits into
Conversation
…494) Two separate node processes retry one failed Dispatch against one temporary store with an injected Orca runner that logs launches to a shared file. Free races and a staged race (both observe, then reserve together) each end with exactly one reservation, one worker-start, and a fence refusal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015tRfRTUirSrHNNTW4C8Hy9
Add a test-only reservation probe that runs inside recordWorker's transaction right before the predecessor-retiring UPDATE. The staged race now parks both processes there and releases them together; exactly one UPDATE wins. The stale one was refused by SQLite (SQLITE_BUSY_SNAPSHOT) as a raw 'database is locked' error, so recordWorker now maps that to the documented 'Orca retry reservation lost' refusal. Free races accept every documented custody refusal, including settlement's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015tRfRTUirSrHNNTW4C8Hy9
kryptobaseddev
force-pushed
the
task/T14494
branch
from
November 12, 2026 08:28
4061079 to
5ef293a
Compare
Replace the inline asserted error type in isSqliteSnapshotConflict with 'errcode' in / typeof narrowing; behaviour unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015tRfRTUirSrHNNTW4C8Hy9
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a two-process test for the Orca retry reservation. Until now only an in-process race covered it (
orca-worker-custody.test.ts), as the OpenCode review of #2803 pointed out. The test turned up one real defect (below), which this PR fixes.Test
packages/core/src/orchestrate/__tests__/orca-retry-multiprocess.test.tsstarts two separatenodeprocesses, each with its own SQLite connection to one temporary CLEO store. Both run the compiledretryOrcaWorkeron the same failed Dispatch, released together by a barrier file. Each child injects an Orca runner that answers from recorded fixture replies and appends everyworker-startto a shared launch log, so no real Orca worker ever starts. Every wait is bounded: 20 s per barrier, 45 s per child, 90 s per test.setRetryReservationProbe, runs insiderecordWorker's reservation transaction, after the predecessor check and immediately before the retiringUPDATE(themetadataJson+stoppedAtCAS). Both processes park there and are released together. The test asserts exactly oneUPDATEwinner, one launch from the winner's pid, one open successor and a retired predecessor. The loser must get exactlyOrca retry reservation lost; no new launch is authorized.Orca resource observation changed; reload the exact worker.(review point 2). Exactly one launch is still required.Defect found and fixed
With both processes parked at the CAS, the loser did not get a documented refusal. Its
UPDATEfailed with a rawFailed query: update "tasks_agent_instances"…error, caused byERR_SQLITE_ERROR 517 database is locked(SQLITE_BUSY_SNAPSHOT). The reservation is a deferred transaction, and the loser's read snapshot predates the other process's write, so SQLite refuses the stale writer before the CAS predicate is evaluated.recordWorkernow reports that conflict on the retry path as the documentedOrca retry reservation lostrefusal, keeping the original error ascause. The transaction itself is unchanged, and exactly one retry still launches.Falsifier results (each change made temporarily in a scratch copy, then restored)
metadataJson+stoppedAt;UPDATEkeyed by id only): all 4 tests still pass. Across processes the predicate is unreachable, because WAL snapshot isolation refuses the stale writer first. Both processes can never hold the same pre-state at the CAS through the retry entry point, since each observation bumps the stored revision. The predicate is defence in depth; the cross-process fence is the transaction. The test file's header documents this.Per-AC evidence
UPDATEand asserts exactly one wins; the free races cover the earlier fences. Locallypnpm exec vitest run packages/core/src/orchestrate/__tests__/orca-retry-multiprocess.test.tsgaveTests 4 passed (4)on 3 consecutive runs.orca-worker-custody.test.ts(31) andworker-records.test.ts(9) also pass. Gates 28, 35, 38 and 43 and the changeset lint pass.Notes: local runs used a copy of the main checkout's core dist with
worker-records.jstranspiled from this branch's source, because no local build was allowed. CI runs against a fresh build. The suite is skipped with a reason when the dist is missing, astask-claims-multiprocess.test.tsis.Closes: T14494 AC1
🤖 Generated with Claude Code
https://claude.ai/code/session_015tRfRTUirSrHNNTW4C8Hy9