Skip to content

fix(ocap-kernel): keep the kernel's account of a vat's life in one piece - #1019

Closed
sirtimid wants to merge 7 commits into
sirtimid/clist-import-refcountfrom
sirtimid/vat-lifecycle-consistency
Closed

fix(ocap-kernel): keep the kernel's account of a vat's life in one piece#1019
sirtimid wants to merge 7 commits into
sirtimid/clist-import-refcountfrom
sirtimid/vat-lifecycle-consistency

Conversation

@sirtimid

@sirtimid sirtimid commented Aug 12, 2026

Copy link
Copy Markdown
Member

Stacked on #1010. Base is sirtimid/clist-import-refcount; review that first. Split out of it
at 988411e4e — the refcount and GC accounting stays there, the vat lifecycle work is here.

The defect

Only kernelStore.deleteVat() removes vatConfig.<vatId>. cleanupTerminatedVat sweeps
${vatId}.-prefixed keys, which never match vatConfig.<vatId>. So the two writes that together
mean "this vat is dead" — deleteVat and markVatAsTerminated — both had to land, and nothing
made them.

They were spread across four functions with awaits between them: VatManager.stopVat, #endVat's
finally, VatHandle.terminate, and a lambda in Kernel.ts. A throw part-way left the vat
marked terminated with its config alive. That reads as active again the moment
cleanupTerminatedVat calls forgetTerminatedVat, at which point KernelRouter.#resolveEndpoint
sees a vat the store calls live and the kernel has no handle for, rethrows, and kills the run
loop. initializeAllVats then resurrects the vat on the next process start.

Two more, found alongside it:

  • A vat whose stream failed tore itself down. VatHandle.#init's drain catch called
    this.terminate(true, …), which did deleteVat but never markVatAsTerminated and never
    removed the handle from VatManager.#vats. The router went on resolving the handle
    successfully, the write went nowhere, and since the vat RPC client has no timeout the crank
    never completed — the run loop hung while getRunLoopStatus() still reported running.
    Pre-existing, but newly load-bearing now that auditRefCounts is on for every kernel-test
    kernel.

  • A vat reported its dropped imports a bringOutYourDead late. makeGCAndFinalize called
    gc() without draining the queues first. A pending continuation still holds its closure's
    objects, so a sweep with work outstanding finds them reachable and the vat reports nothing on
    the BOYD that provoked it. This is what made
    garbage-collection.test.ts > an object shared by two importers > survives until both importers let go fail under full-suite load with expected [ 'v2', 'v3' ] to strictly equal [ 'v3' ].

Approach

VatManager.#retireVat — the store side of a vat's death, in one synchronous step. Reject the
promises it was deciding, unpin its root, deleteVat, markVatAsTerminated, with no await
between them, so the half-written state cannot arise. Modelled on SwingSet's terminateVat
(kernel.js:345-412) and its comment at :348 about the "synchronous prelude". Killing the
worker is deliberately not part of it: that can fail, and a store that says the vat is dead is
worth more than a store still waiting to find out.

#endVat and #abandonVat were both partial copies of this and are gone. Kernel.ts's lambda
and launchVat lose their trailing markVatAsTerminated. VatHandle.terminate is left with only
its own channel to close, and rejects its pending RPCs ahead of ending the stream rather than
after, so a stream that will not close does not strand callers on a worker that is already dead.

stopVat(vatId, true, …) tolerates a missing handle. A restart needs a live handle to read
its config from; an ending vat does not, and must not — the vat may be one the store still lists
while the kernel has lost its handle, which is exactly what SubclusterManager.terminateSubcluster
can hand it, since it iterates the store's own vat list and previously aborted its loop on the
first such vat. It still refuses a vat neither the kernel nor the store knows about.

A fatal stream error is routed through the manager, via a new required onCriticalFailure
prop on VatHandle. Only the manager can put the vat's death on record and drop the handle.

makeGCAndFinalize drains the queues before sweeping — two await delay(0), matching
SwingSet's two setImmediates at
agoric-sdk/packages/internal/src/lib-nodejs/gc-and-finalize.js:82-85.

reapAndSettle in the GC test waits for the outcome, not just an empty action set. An empty
set is also what "the vat has not told us anything yet" looks like, so it was returning on round
zero. It now takes a done: () => boolean, re-reaps each round, and requires both.

Two things reviewers should push on

#retireVat is synchronous, which is not the same as atomic. Its doc comment says "With no
await between them, that state cannot arise" — true of interleaving, false of crank rollback. When
#retireVat runs inside a crank (the Kernel.ts lambda path) and anything later in that crank
throws, the run loop's catch calls rollbackCrank('start') and reverts all four writes, while
resolvePromises' immediate default has already resolved JS subscriptions in RAM, which no
rollback touches. The run loop is dying in that scenario anyway, so it is not a live bug — but the
state left on disk is "vat alive, no worker", which initializeAllVats resurrects. Naming it here
rather than fixing it, because the fix is the queued-teardown change below.

onCriticalFailure writes to the store from outside any crank. It fires from a .catch() on
the drain promise at an arbitrary microtask. If the run loop is mid-crank on some other vat and
that crank aborts, the rollback reverts the four store writes but not the RAM #vats.delete
leaving store-says-active with no handle, which is the same disagreement #resolveEndpoint kills
the run loop over. It also never rejects the dead handle's pending RPCs, so anyone parked on a
sendVatCommand for that vat waits forever. Both are strictly better than the hang this replaces,
and both close properly only once teardown is queued work.

Known and deliberately not fixed

waitForCrank() does not serialize. After endCrank() the run loop re-enters startCrank
#getNextRunQueueItemdeliver(item) synchronously in the same microtask
(KernelQueue.ts:127-178), while the waitForCrank() continuation is only a queued microtask
(store/methods/crank.ts:113-133). A queued restartVat therefore runs its #vats.has(vatId)
guard before a concurrent terminateVat has done anything, and both are RPCs. main has the same
hole in a different shape, so this is not a regression — but the comments at VatManager.ts:41-58
and 270-273 claim a guarantee the mechanism does not provide.

The structural fix is to port three more SwingSet properties, in order: route terminateVat
through the run queue with a separate non-crank teardown path for reset/clearStorage/shutdown
that runs only when the loop is already stopped; report lifecycle outcomes by resolving a kernel
promise carried on the queue item rather than a RAM waiter map, as SwingSet does via
vatAdminMethargs (kernel.js:1043-1055), queued after the rollback so the report survives the
unwind; then delete #vatsInFlux, #trackFlux, provideVat, #restartWaiters, #awaitRestart,
the async-ness of Kernel.#getEndpoint, and #resolveEndpoint's three-way store-vs-RAM test.
Net effect is a deletion of roughly 200 lines of the hardest-to-reason-about code in the kernel.
The prerequisite savepoint work — consumeMessage plus a second savepoint — is #1012.

Worth filing separately: the vat RPC client has no timeout (RpcClient.#createMessage), so any
lost write hangs a crank forever with getRunLoopStatus() still reporting running.
onCriticalFailure covers the stream-error case; a timeout would cover the rest.

Verification

  • Monorepo yarn lint clean; @metamask/ocap-kernel and @ocap/kernel-test suites pass; full
    yarn test:dev:quiet exit 0
  • Base 988411e4e verified independently green — lint and both suites — so the split boundary
    holds on its own
  • yarn test:e2e:ci, extension: 16 passed, 1 flaky across two runs. A different test flaked
    each time, both timing out on loadExtension's 10s "Subcluster s1 - 3 Vats" readiness gate, so
    it is startup flake rather than anything lifecycle-specific
  • yarn test:e2e:ci, kernel-node-runtime: 68/69. The one failure —
    remote-comms > Intentional Disconnect > handles remote intentional disconnect without reconnecting, URL redemption timed out after 8000ms — reproduces identically on 988411e4e
    and on origin/main, so it is pre-existing and unrelated to this PR
  • Three of the five new VatManager tests fail against the old interleaving:
    records all of it even when the worker refuses to go (the promise rejections and deleteVat
    never ran), records it for a vat the store still lists but the kernel has lost (old stopVat
    opened with getVat, which throws), and records it when a vat's stream fails under it
    (onCriticalFailure did not exist). The other two are regression guards on preserved behaviour

🤖 Generated with Claude Code


Note

High Risk
Changes core run-loop delivery, vat termination/restart, GC rollback, and store/router invariants; regressions can hang or kill the run loop or resurrect half-dead vats.

Overview
This PR tightens vat lifecycle so the kernel’s view of a vat stays consistent under teardown, restart, and delivery.

Vat death is recorded in one synchronous step via VatManager.#retireVat (reject decider promises, unpin root, deleteVat, markVatAsTerminated), instead of spreading those writes across stopVat, VatHandle.terminate, and Kernel callbacks with awaits in between. Termination can run without a live handle (e.g. subcluster teardown), and fatal stream errors go through onCriticalFailure so the manager retires the vat instead of leaving a zombie handle that hangs cranks.

Restarts are queued as restartVat run-queue items (enqueueRestartVat / performVatRestart), so no crank sees a vat mid-replacement. Failed relaunch terminates the vat and reports to the caller without aborting the crank (avoiding infinite replay). Termination supersedes pending restart waiters.

Delivery routing uses async provideVat / #resolveEndpoint to wait out teardown or drop work only when an endpoint is gone for good; notify, bringOutYourDead, and GC actions follow the same rules. rollbackCrank restores GC candidate sets from savepoints instead of clearing them. getImporters includes remotes; gc-and-finalize drains the event loop before sweeping so BOYD reports drops on the right crank.

Reviewed by Cursor Bugbot for commit b3c121f. Bugbot is set up for automated code reviews on this repo. Configure here.

sirtimid and others added 5 commits August 10, 2026 13:56
…t as gone

`restartVat` keeps the vat's c-list and takes the vat out of the kernel's vat
table for as long as launching a worker and negotiating with it takes. Absence
from that table was the only signal available, so a crank landing in the window
resolved a live vat as a dead one: a message went splat, a `notify` or
`bringOutYourDead` took the run loop down, and a garbage-collection action
released the kernel's side of entries the returning incarnation still holds.

The vat's flux is now recorded rather than guarded against. `provideVat` waits on
that record, so a crank arriving mid-restart delivers to the new incarnation, and
the kernel's endpoint lookup is asynchronous to let it wait. The crank waits for
the vat, rather than the restart waiting for the run loop — which is the same
direction SwingSet takes it, where a delivery to an evicted vat awaits
`ensureVatOnline` and eviction is routine. Inverted the other way, as a lock the
restart holds while the loop stands still, whatever holds it must never await
anything the loop has to deliver, and `runVat` is exactly that kind of await.

The wait for the crank in flight stays ahead of the record, which is load-bearing:
record first and wait after, and a crank that is already running reaches its
endpoint lookup, finds the record, and waits for a restart that is waiting for
that crank to end. What the ordering leaves open is a crank the run loop starts in
the turn between the wait resolving and the record appearing — it takes the
outgoing handle and can still be mid-delivery when the worker goes down. Closing
that needs the restart to happen inside a crank, the way `processUpgradeVat` does
upstream, where the vat is idle by construction and nothing mutates kernel state
from outside the run loop.

A relaunch that fails now marks the vat terminated. It previously left a vat with
no worker that the store still counted among the living, which nothing revisits:
`cleanupTerminatedVat` only walks vats that are marked.

The GC action guard for a vat that is absent but not terminated stays, now as an
assertion rather than a live path, with its reasoning corrected: aborting the
crank does preserve the action, since `rollbackCrank` restores the cached GC set,
but nothing about the vat changes between cranks, so the action would be
re-selected and re-aborted forever with no delivery to wait on.

Also shortens this PR's CHANGELOG entries, which had grown to carry rationale
that belongs in these messages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…for endpoints that are gone

Restarting a vat alongside a running run loop cannot be made safe by ordering
alone. The previous approach recorded the vat as mid-flux so a delivery would wait
for the new incarnation, and the record had to be installed *after* waiting out
the crank in flight — install it before, and a crank that is already running
reaches its endpoint lookup, finds the record, and waits for a restart that is
waiting for that crank to end. That ordering left a turn of its own: a crank the
run loop starts between the wait resolving and the record appearing takes the
outgoing handle, and can be mid-delivery when the worker goes down.

So the restart is now the run loop's own work, as a queued `restartVat` item, the
way SwingSet queues `upgrade-vat` for `processUpgradeVat`. In a crank of its own
there is no window to close: the run loop is the only thing that delivers, and it
is here instead, so the vat is idle by construction. `Kernel.restartVat` settles
when the crank has done it, and refuses outright if the run loop is dead, since
nothing would ever carry the request out.

Termination keeps the flux record, because it cannot be queued: `reset` and
`clearStorage` tear vats down on kernels whose run loop has died. Both of its
steps now live inside `#trackFlux`, in the order that does not deadlock, so a
caller does not sequence them and cannot get them wrong — with a test that hangs
if the order is reversed.

Two more, found in review of the previous round:

`#deliverNotify` and `#deliverBringOutYourDead` awaited the endpoint with no
handling for one that has vanished, so a crank landing during a termination took
the rejection into the run loop and killed it. This predates the wait — the lookup
used to throw synchronously in the same case — but the wait is what makes it
routine. All three of notify, reap, and GC-action delivery now go through
`#resolveEndpoint`, which drops the work for an endpoint that is gone for good
(a terminated vat, or a remote) and propagates anything else. The notify resolves
its endpoint before translating, which would otherwise mint c-list entries for an
endpoint with no way to hear about them.

A relaunch that failed marked the vat terminated but left its root pinned:
`stopVat` releases that pin only when it is the one ending the vat, and it had
been told the vat was coming back, while vat cleanup does not touch pins at all.
The pin, and the root's refcount, were held for the life of the kernel. Both
paths now release it through one helper.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The send path caught every endpoint lookup failure and treated it as a splat,
which its own TODO called out: an error that is not "this endpoint is gone"
silently discarded a deliverable message and rejected its result with
ENDPOINT_UNREACHABLE. It is now the last of the four delivery paths to go through
`resolveEndpoint`, so a splat happens where the endpoint will not be back — a
terminated vat, or a remote — and anything else propagates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nd rollback

Four ways to kill or wedge the kernel, found reviewing this branch.

`rollbackCrank` emptied `maybeFreeKrefs` rather than restoring it. The set is
not per-crank — only `collectGarbage` empties it, at the end of a crank that had
an item — so a candidate created while the run loop was idle, as `terminateVat`
unpinning a root creates one, was owed a collection that any later crank's
rollback silently cancelled. Savepoints now carry the set as it stood when they
were taken. The audit cannot see this one: the counts stay self-consistent at 0.

A restart that could not relaunch its vat threw, and the run loop's catch rolls
back on any throw — undoing the termination records `performVatRestart` had just
written and returning the request to the run queue. Every subsequent process
start dequeued it and failed the same way. It now terminates the vat and reports
through the waiter, so the crank commits and the request is spent. The comment
claiming the throw preserved those records had the causality backwards.

Terminating a vat left a queued restart for it to be carried out against a vat
that no longer existed; `#restartVatWorker` is the one item type that does not
go through `#resolveEndpoint`, so the resulting `VatNotFoundError` propagated.
Restart-then-terminate is reachable from RPC. The waiter is now rejected when
the vat is terminated and the request dropped when the crank reaches it.

`cleanupTerminatedVat` ends by *unmarking* the vat it finished, so work
outliving it — a `bringOutYourDead` scheduled before it died, which nothing
purges from the reap queue — arrived at an endpoint that was neither present nor
terminated, which `#resolveEndpoint` reserves its throw for. It now asks whether
the store has a live record of the vat at all.

Also fixed, from the same review:

- `getImporters` counted only vats, so retiring an object deleted it without
  telling a remote importer, leaving a c-list entry naming nothing — which the
  audit reports as dangling, taking the run loop with it. Adds `getRemoteIds`.
- `#deliverGCAction` computed the live kref set before awaiting the endpoint and
  used it after. A remote re-handshaking in that window clears its c-list
  without waiting for the crank, and `krefsToErefs` throws rather than returning
  short.
- `#endVat` marks the vat terminated in a `finally`. A teardown that threw left
  it unmarked, which is the state above, and falsified `#trackFlux`'s stated
  invariant that waiters can read "gone" as terminated.
- Comments that no longer described the code: `provideVat` waiting on restarts
  (only teardown is recorded), `stopVat` tearing down "only the worker" (it
  releases the root pin, as of this branch), `clearStorage` terminating vats,
  the audit standing in for the disabled `retireExport` assert, and a stale
  `(1, 1)` baseline rationale. `#vatsInFlux` narrows to `Promise<void>`, which
  removes a branch of `provideVat` that could not be reached.

Tests: each fix has a regression test that fails against the code without it.
Closes the two coverage gaps the review named — the splat path charging the run
queue item's own target when routing went through a promise, and `ko6.refCount`
in the control-panel e2e, restored as three per-checkpoint values rather than
dropped as nondeterministic. Full unit suite, kernel-test with auditing on every
crank, and `test:e2e:ci` at 17/17.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only `deleteVat` removes `vatConfig.<vatId>`, and `cleanupTerminatedVat`
sweeps `${vatId}.`-prefixed keys, which never match it. The writes making up
a vat's death were interleaved with awaits across `VatManager.stopVat`,
`#endVat`'s `finally`, `VatHandle.terminate` and a lambda in `Kernel.ts`, so a
throw part-way left the vat marked terminated with its config alive — which
reads as *active* again as soon as cleanup drops the mark, killing the run
loop over the disagreement and resurrecting the vat on the next process start.

`VatManager.#retireVat` now makes all four writes with no await between them,
modelled on SwingSet's synchronous prelude in `kernel.js` `terminateVat`;
worker teardown follows and is best-effort. `#endVat` and `#abandonVat` go as
duplicates of it, and `VatHandle.terminate` is left with only its own channel
to close.

A vat whose stream fails is retired by the manager, via a new
`onCriticalFailure`, rather than tearing itself down: that left the handle in
the manager and the vat live in the store, so the next delivery went to a
worker that could not answer and, the vat RPC client having no timeout, the
crank never completed while the run loop still reported itself running.

`makeGCAndFinalize` drains the queues before sweeping, since a pending
continuation still holds its closure's objects, so a vat reports its dropped
imports on the `bringOutYourDead` that provoked them rather than a later one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sirtimid
sirtimid requested a review from a team as a code owner August 12, 2026 17:03
The lifecycle work was split out of #1010 into its own stacked PR, so the
eight entries that describe it were citing a PR that no longer contains them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ebe3756. Configure here.

Comment thread packages/ocap-kernel/src/vats/VatManager.ts
…plit

Both entries describe work #1010 started and #1019 finished. The clause each
gained — the relaunch-failure pin release, and restoring the collection
candidates rather than emptying them — arrived with the lifecycle commits, so
it belongs to #1019 even though the entry it hangs off is #1010's.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 72.26%
⬆️ +0.21%
9360 / 12952
🔵 Statements 72.1%
⬆️ +0.21%
9511 / 13191
🔵 Functions 73.04%
⬆️ +0.17%
2225 / 3046
🔵 Branches 66.12%
⬆️ +0.32%
3802 / 5750
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
packages/ocap-kernel/src/Kernel.ts 89.84%
⬆️ +0.08%
79.54%
⬆️ +0.97%
85.41%
🟰 ±0%
89.84%
⬆️ +0.08%
331-333, 404, 428, 503-513, 601, 675, 751-754, 767, 777-778, 831, 854
packages/ocap-kernel/src/KernelQueue.ts 98.58%
⬆️ +0.02%
90.27%
🟰 ±0%
100%
🟰 ±0%
98.58%
⬆️ +0.02%
148, 539
packages/ocap-kernel/src/KernelRouter.ts 94.73%
⬆️ +0.80%
85.26%
⬆️ +6.80%
100%
🟰 ±0%
94.7%
⬆️ +0.77%
127, 190, 207, 282, 337, 397, 424, 427, 541
packages/ocap-kernel/src/KernelServiceManager.ts 98.52%
⬆️ +2.94%
92.3%
⬆️ +7.69%
100%
🟰 ±0%
98.52%
⬆️ +2.94%
311
packages/ocap-kernel/src/types.ts 100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
packages/ocap-kernel/src/garbage-collection/gc-finalize.ts 77.27%
⬆️ +2.27%
75%
🟰 ±0%
100%
🟰 ±0%
77.27%
⬆️ +2.27%
27-31, 72-76
packages/ocap-kernel/src/store/types.ts 100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
packages/ocap-kernel/src/store/methods/crank.ts 100%
🟰 ±0%
93.75%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
packages/ocap-kernel/src/store/methods/remote.ts 98.48%
⬆️ +0.05%
100%
🟰 ±0%
100%
🟰 ±0%
98.48%
⬆️ +0.05%
105-109
packages/ocap-kernel/src/store/methods/vat.ts 98.44%
⬆️ +1.15%
89.47%
⬆️ +7.66%
100%
🟰 ±0%
98.43%
⬆️ +1.16%
298-299
packages/ocap-kernel/src/vats/VatHandle.ts 90.76%
⬆️ +0.62%
86.66%
🟰 ±0%
100%
🟰 ±0%
90.76%
⬆️ +0.62%
379-384, 393-399
packages/ocap-kernel/src/vats/VatManager.ts 96.4%
⬇️ -3.60%
89.47%
⬇️ -10.53%
100%
🟰 ±0%
96.37%
⬇️ -3.63%
181-193
Generated in workflow #4630 for commit b3c121f by the Vitest Coverage Report Action

@sirtimid

Copy link
Copy Markdown
Member Author

Replaced by #1023, which is this branch rebased onto the split stack (#1020#1021#1022). Closing now that its replacement is up — it stayed open until then so this work always had a live pointer.

Two conflicts surfaced by the rebase were real disagreements rather than textual noise, and are resolved in their own commit:

Also worth noting: #1023 closes #1015, which #1022 could only advance. getImporters now counts remotes, so retiring an object queues a retireImport for a remote importer instead of deleting the object and leaving the remote's c-list entry naming nothing.

The branch sirtimid/vat-lifecycle-consistency is left on origin.

@sirtimid sirtimid closed this Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant