fix(ocap-kernel): keep the kernel's account of a vat's life in one piece - #1019
fix(ocap-kernel): keep the kernel's account of a vat's life in one piece#1019sirtimid wants to merge 7 commits into
Conversation
…t as gone `restartVat` keeps the vat's c-list and takes the vat out of the kernel's vat table for as long as launching a worker and negotiating with it takes. Absence from that table was the only signal available, so a crank landing in the window resolved a live vat as a dead one: a message went splat, a `notify` or `bringOutYourDead` took the run loop down, and a garbage-collection action released the kernel's side of entries the returning incarnation still holds. The vat's flux is now recorded rather than guarded against. `provideVat` waits on that record, so a crank arriving mid-restart delivers to the new incarnation, and the kernel's endpoint lookup is asynchronous to let it wait. The crank waits for the vat, rather than the restart waiting for the run loop — which is the same direction SwingSet takes it, where a delivery to an evicted vat awaits `ensureVatOnline` and eviction is routine. Inverted the other way, as a lock the restart holds while the loop stands still, whatever holds it must never await anything the loop has to deliver, and `runVat` is exactly that kind of await. The wait for the crank in flight stays ahead of the record, which is load-bearing: record first and wait after, and a crank that is already running reaches its endpoint lookup, finds the record, and waits for a restart that is waiting for that crank to end. What the ordering leaves open is a crank the run loop starts in the turn between the wait resolving and the record appearing — it takes the outgoing handle and can still be mid-delivery when the worker goes down. Closing that needs the restart to happen inside a crank, the way `processUpgradeVat` does upstream, where the vat is idle by construction and nothing mutates kernel state from outside the run loop. A relaunch that fails now marks the vat terminated. It previously left a vat with no worker that the store still counted among the living, which nothing revisits: `cleanupTerminatedVat` only walks vats that are marked. The GC action guard for a vat that is absent but not terminated stays, now as an assertion rather than a live path, with its reasoning corrected: aborting the crank does preserve the action, since `rollbackCrank` restores the cached GC set, but nothing about the vat changes between cranks, so the action would be re-selected and re-aborted forever with no delivery to wait on. Also shortens this PR's CHANGELOG entries, which had grown to carry rationale that belongs in these messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…for endpoints that are gone Restarting a vat alongside a running run loop cannot be made safe by ordering alone. The previous approach recorded the vat as mid-flux so a delivery would wait for the new incarnation, and the record had to be installed *after* waiting out the crank in flight — install it before, and a crank that is already running reaches its endpoint lookup, finds the record, and waits for a restart that is waiting for that crank to end. That ordering left a turn of its own: a crank the run loop starts between the wait resolving and the record appearing takes the outgoing handle, and can be mid-delivery when the worker goes down. So the restart is now the run loop's own work, as a queued `restartVat` item, the way SwingSet queues `upgrade-vat` for `processUpgradeVat`. In a crank of its own there is no window to close: the run loop is the only thing that delivers, and it is here instead, so the vat is idle by construction. `Kernel.restartVat` settles when the crank has done it, and refuses outright if the run loop is dead, since nothing would ever carry the request out. Termination keeps the flux record, because it cannot be queued: `reset` and `clearStorage` tear vats down on kernels whose run loop has died. Both of its steps now live inside `#trackFlux`, in the order that does not deadlock, so a caller does not sequence them and cannot get them wrong — with a test that hangs if the order is reversed. Two more, found in review of the previous round: `#deliverNotify` and `#deliverBringOutYourDead` awaited the endpoint with no handling for one that has vanished, so a crank landing during a termination took the rejection into the run loop and killed it. This predates the wait — the lookup used to throw synchronously in the same case — but the wait is what makes it routine. All three of notify, reap, and GC-action delivery now go through `#resolveEndpoint`, which drops the work for an endpoint that is gone for good (a terminated vat, or a remote) and propagates anything else. The notify resolves its endpoint before translating, which would otherwise mint c-list entries for an endpoint with no way to hear about them. A relaunch that failed marked the vat terminated but left its root pinned: `stopVat` releases that pin only when it is the one ending the vat, and it had been told the vat was coming back, while vat cleanup does not touch pins at all. The pin, and the root's refcount, were held for the life of the kernel. Both paths now release it through one helper. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The send path caught every endpoint lookup failure and treated it as a splat, which its own TODO called out: an error that is not "this endpoint is gone" silently discarded a deliverable message and rejected its result with ENDPOINT_UNREACHABLE. It is now the last of the four delivery paths to go through `resolveEndpoint`, so a splat happens where the endpoint will not be back — a terminated vat, or a remote — and anything else propagates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nd rollback Four ways to kill or wedge the kernel, found reviewing this branch. `rollbackCrank` emptied `maybeFreeKrefs` rather than restoring it. The set is not per-crank — only `collectGarbage` empties it, at the end of a crank that had an item — so a candidate created while the run loop was idle, as `terminateVat` unpinning a root creates one, was owed a collection that any later crank's rollback silently cancelled. Savepoints now carry the set as it stood when they were taken. The audit cannot see this one: the counts stay self-consistent at 0. A restart that could not relaunch its vat threw, and the run loop's catch rolls back on any throw — undoing the termination records `performVatRestart` had just written and returning the request to the run queue. Every subsequent process start dequeued it and failed the same way. It now terminates the vat and reports through the waiter, so the crank commits and the request is spent. The comment claiming the throw preserved those records had the causality backwards. Terminating a vat left a queued restart for it to be carried out against a vat that no longer existed; `#restartVatWorker` is the one item type that does not go through `#resolveEndpoint`, so the resulting `VatNotFoundError` propagated. Restart-then-terminate is reachable from RPC. The waiter is now rejected when the vat is terminated and the request dropped when the crank reaches it. `cleanupTerminatedVat` ends by *unmarking* the vat it finished, so work outliving it — a `bringOutYourDead` scheduled before it died, which nothing purges from the reap queue — arrived at an endpoint that was neither present nor terminated, which `#resolveEndpoint` reserves its throw for. It now asks whether the store has a live record of the vat at all. Also fixed, from the same review: - `getImporters` counted only vats, so retiring an object deleted it without telling a remote importer, leaving a c-list entry naming nothing — which the audit reports as dangling, taking the run loop with it. Adds `getRemoteIds`. - `#deliverGCAction` computed the live kref set before awaiting the endpoint and used it after. A remote re-handshaking in that window clears its c-list without waiting for the crank, and `krefsToErefs` throws rather than returning short. - `#endVat` marks the vat terminated in a `finally`. A teardown that threw left it unmarked, which is the state above, and falsified `#trackFlux`'s stated invariant that waiters can read "gone" as terminated. - Comments that no longer described the code: `provideVat` waiting on restarts (only teardown is recorded), `stopVat` tearing down "only the worker" (it releases the root pin, as of this branch), `clearStorage` terminating vats, the audit standing in for the disabled `retireExport` assert, and a stale `(1, 1)` baseline rationale. `#vatsInFlux` narrows to `Promise<void>`, which removes a branch of `provideVat` that could not be reached. Tests: each fix has a regression test that fails against the code without it. Closes the two coverage gaps the review named — the splat path charging the run queue item's own target when routing went through a promise, and `ko6.refCount` in the control-panel e2e, restored as three per-checkpoint values rather than dropped as nondeterministic. Full unit suite, kernel-test with auditing on every crank, and `test:e2e:ci` at 17/17. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only `deleteVat` removes `vatConfig.<vatId>`, and `cleanupTerminatedVat`
sweeps `${vatId}.`-prefixed keys, which never match it. The writes making up
a vat's death were interleaved with awaits across `VatManager.stopVat`,
`#endVat`'s `finally`, `VatHandle.terminate` and a lambda in `Kernel.ts`, so a
throw part-way left the vat marked terminated with its config alive — which
reads as *active* again as soon as cleanup drops the mark, killing the run
loop over the disagreement and resurrecting the vat on the next process start.
`VatManager.#retireVat` now makes all four writes with no await between them,
modelled on SwingSet's synchronous prelude in `kernel.js` `terminateVat`;
worker teardown follows and is best-effort. `#endVat` and `#abandonVat` go as
duplicates of it, and `VatHandle.terminate` is left with only its own channel
to close.
A vat whose stream fails is retired by the manager, via a new
`onCriticalFailure`, rather than tearing itself down: that left the handle in
the manager and the vat live in the store, so the next delivery went to a
worker that could not answer and, the vat RPC client having no timeout, the
crank never completed while the run loop still reported itself running.
`makeGCAndFinalize` drains the queues before sweeping, since a pending
continuation still holds its closure's objects, so a vat reports its dropped
imports on the `bringOutYourDead` that provoked them rather than a later one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lifecycle work was split out of #1010 into its own stacked PR, so the eight entries that describe it were citing a PR that no longer contains them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ebe3756. Configure here.
…plit Both entries describe work #1010 started and #1019 finished. The clause each gained — the relaunch-failure pin release, and restoring the collection candidates rather than emptying them — arrived with the lifecycle commits, so it belongs to #1019 even though the entry it hangs off is #1010's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coverage Report
File Coverage
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Replaced by #1023, which is this branch rebased onto the split stack (#1020 → #1021 → #1022). Closing now that its replacement is up — it stayed open until then so this work always had a live pointer. Two conflicts surfaced by the rebase were real disagreements rather than textual noise, and are resolved in their own commit:
Also worth noting: #1023 closes #1015, which #1022 could only advance. The branch |

Stacked on #1010. Base is
sirtimid/clist-import-refcount; review that first. Split out of itat
988411e4e— the refcount and GC accounting stays there, the vat lifecycle work is here.The defect
Only
kernelStore.deleteVat()removesvatConfig.<vatId>.cleanupTerminatedVatsweeps${vatId}.-prefixed keys, which never matchvatConfig.<vatId>. So the two writes that togethermean "this vat is dead" —
deleteVatandmarkVatAsTerminated— both had to land, and nothingmade them.
They were spread across four functions with awaits between them:
VatManager.stopVat,#endVat'sfinally,VatHandle.terminate, and a lambda inKernel.ts. A throw part-way left the vatmarked terminated with its config alive. That reads as active again the moment
cleanupTerminatedVatcallsforgetTerminatedVat, at which pointKernelRouter.#resolveEndpointsees a vat the store calls live and the kernel has no handle for, rethrows, and kills the run
loop.
initializeAllVatsthen resurrects the vat on the next process start.Two more, found alongside it:
A vat whose stream failed tore itself down.
VatHandle.#init's drain catch calledthis.terminate(true, …), which diddeleteVatbut nevermarkVatAsTerminatedand neverremoved the handle from
VatManager.#vats. The router went on resolving the handlesuccessfully, the write went nowhere, and since the vat RPC client has no timeout the crank
never completed — the run loop hung while
getRunLoopStatus()still reportedrunning.Pre-existing, but newly load-bearing now that
auditRefCountsis on for everykernel-testkernel.
A vat reported its dropped imports a
bringOutYourDeadlate.makeGCAndFinalizecalledgc()without draining the queues first. A pending continuation still holds its closure'sobjects, so a sweep with work outstanding finds them reachable and the vat reports nothing on
the BOYD that provoked it. This is what made
garbage-collection.test.ts > an object shared by two importers > survives until both importers let gofail under full-suite load withexpected [ 'v2', 'v3' ] to strictly equal [ 'v3' ].Approach
VatManager.#retireVat— the store side of a vat's death, in one synchronous step. Reject thepromises it was deciding, unpin its root,
deleteVat,markVatAsTerminated, with no awaitbetween them, so the half-written state cannot arise. Modelled on SwingSet's
terminateVat(
kernel.js:345-412) and its comment at:348about the "synchronous prelude". Killing theworker is deliberately not part of it: that can fail, and a store that says the vat is dead is
worth more than a store still waiting to find out.
#endVatand#abandonVatwere both partial copies of this and are gone.Kernel.ts's lambdaand
launchVatlose their trailingmarkVatAsTerminated.VatHandle.terminateis left with onlyits own channel to close, and rejects its pending RPCs ahead of ending the stream rather than
after, so a stream that will not close does not strand callers on a worker that is already dead.
stopVat(vatId, true, …)tolerates a missing handle. A restart needs a live handle to readits config from; an ending vat does not, and must not — the vat may be one the store still lists
while the kernel has lost its handle, which is exactly what
SubclusterManager.terminateSubclustercan hand it, since it iterates the store's own vat list and previously aborted its loop on the
first such vat. It still refuses a vat neither the kernel nor the store knows about.
A fatal stream error is routed through the manager, via a new required
onCriticalFailureprop on
VatHandle. Only the manager can put the vat's death on record and drop the handle.makeGCAndFinalizedrains the queues before sweeping — twoawait delay(0), matchingSwingSet's two
setImmediates atagoric-sdk/packages/internal/src/lib-nodejs/gc-and-finalize.js:82-85.reapAndSettlein the GC test waits for the outcome, not just an empty action set. An emptyset is also what "the vat has not told us anything yet" looks like, so it was returning on round
zero. It now takes a
done: () => boolean, re-reaps each round, and requires both.Two things reviewers should push on
#retireVatis synchronous, which is not the same as atomic. Its doc comment says "With noawait between them, that state cannot arise" — true of interleaving, false of crank rollback. When
#retireVatruns inside a crank (theKernel.tslambda path) and anything later in that crankthrows, the run loop's catch calls
rollbackCrank('start')and reverts all four writes, whileresolvePromises'immediatedefault has already resolved JS subscriptions in RAM, which norollback touches. The run loop is dying in that scenario anyway, so it is not a live bug — but the
state left on disk is "vat alive, no worker", which
initializeAllVatsresurrects. Naming it hererather than fixing it, because the fix is the queued-teardown change below.
onCriticalFailurewrites to the store from outside any crank. It fires from a.catch()onthe drain promise at an arbitrary microtask. If the run loop is mid-crank on some other vat and
that crank aborts, the rollback reverts the four store writes but not the RAM
#vats.delete—leaving store-says-active with no handle, which is the same disagreement
#resolveEndpointkillsthe run loop over. It also never rejects the dead handle's pending RPCs, so anyone parked on a
sendVatCommandfor that vat waits forever. Both are strictly better than the hang this replaces,and both close properly only once teardown is queued work.
Known and deliberately not fixed
waitForCrank()does not serialize. AfterendCrank()the run loop re-entersstartCrank→#getNextRunQueueItem→deliver(item)synchronously in the same microtask(
KernelQueue.ts:127-178), while thewaitForCrank()continuation is only a queued microtask(
store/methods/crank.ts:113-133). A queuedrestartVattherefore runs its#vats.has(vatId)guard before a concurrent
terminateVathas done anything, and both are RPCs.mainhas the samehole in a different shape, so this is not a regression — but the comments at
VatManager.ts:41-58and
270-273claim a guarantee the mechanism does not provide.The structural fix is to port three more SwingSet properties, in order: route
terminateVatthrough the run queue with a separate non-crank teardown path for
reset/clearStorage/shutdownthat runs only when the loop is already stopped; report lifecycle outcomes by resolving a kernel
promise carried on the queue item rather than a RAM waiter map, as SwingSet does via
vatAdminMethargs(kernel.js:1043-1055), queued after the rollback so the report survives theunwind; then delete
#vatsInFlux,#trackFlux,provideVat,#restartWaiters,#awaitRestart,the async-ness of
Kernel.#getEndpoint, and#resolveEndpoint's three-way store-vs-RAM test.Net effect is a deletion of roughly 200 lines of the hardest-to-reason-about code in the kernel.
The prerequisite savepoint work —
consumeMessageplus a second savepoint — is #1012.Worth filing separately: the vat RPC client has no timeout (
RpcClient.#createMessage), so anylost write hangs a crank forever with
getRunLoopStatus()still reportingrunning.onCriticalFailurecovers the stream-error case; a timeout would cover the rest.Verification
yarn lintclean;@metamask/ocap-kerneland@ocap/kernel-testsuites pass; fullyarn test:dev:quietexit 0988411e4everified independently green — lint and both suites — so the split boundaryholds on its own
yarn test:e2e:ci,extension: 16 passed, 1 flaky across two runs. A different test flakedeach time, both timing out on
loadExtension's 10s "Subcluster s1 - 3 Vats" readiness gate, soit is startup flake rather than anything lifecycle-specific
yarn test:e2e:ci,kernel-node-runtime: 68/69. The one failure —remote-comms > Intentional Disconnect > handles remote intentional disconnect without reconnecting,URL redemption timed out after 8000ms— reproduces identically on988411e4eand on
origin/main, so it is pre-existing and unrelated to this PRVatManagertests fail against the old interleaving:records all of it even when the worker refuses to go(the promise rejections anddeleteVatnever ran),
records it for a vat the store still lists but the kernel has lost(oldstopVatopened with
getVat, which throws), andrecords it when a vat's stream fails under it(
onCriticalFailuredid not exist). The other two are regression guards on preserved behaviour🤖 Generated with Claude Code
Note
High Risk
Changes core run-loop delivery, vat termination/restart, GC rollback, and store/router invariants; regressions can hang or kill the run loop or resurrect half-dead vats.
Overview
This PR tightens vat lifecycle so the kernel’s view of a vat stays consistent under teardown, restart, and delivery.
Vat death is recorded in one synchronous step via
VatManager.#retireVat(reject decider promises, unpin root,deleteVat,markVatAsTerminated), instead of spreading those writes acrossstopVat,VatHandle.terminate, andKernelcallbacks with awaits in between. Termination can run without a live handle (e.g. subcluster teardown), and fatal stream errors go throughonCriticalFailureso the manager retires the vat instead of leaving a zombie handle that hangs cranks.Restarts are queued as
restartVatrun-queue items (enqueueRestartVat/performVatRestart), so no crank sees a vat mid-replacement. Failed relaunch terminates the vat and reports to the caller without aborting the crank (avoiding infinite replay). Termination supersedes pending restart waiters.Delivery routing uses async
provideVat/#resolveEndpointto wait out teardown or drop work only when an endpoint is gone for good;notify,bringOutYourDead, and GC actions follow the same rules.rollbackCrankrestores GC candidate sets from savepoints instead of clearing them.getImportersincludes remotes;gc-and-finalizedrains the event loop before sweeping so BOYD reports drops on the right crank.Reviewed by Cursor Bugbot for commit b3c121f. Bugbot is set up for automated code reviews on this repo. Configure here.