Skip to content

fix(vmm): persist guest events off the executor and outside the global lock - #1271

Open
kvinwang wants to merge 1 commit into
nextfrom
fix/vmm-event-io
Open

kvinwang wants to merge 1 commit into
nextfrom
fix/vmm-event-io

Conversation

@kvinwang

Copy link
Copy Markdown
Collaborator

Problem

HostApi.Notify persisted guest events on an async worker while holding the global VMM state mutex. An instance.info file/directory fsync could therefore stall unrelated VMs' control-plane operations.

Fix

  • Keep in-memory event updates under short state locks; move persistence to spawn_blocking.
  • Serialize writes with the existing per-VM lifecycle lock, owned by the blocking task even after request cancellation.
  • Reject notifications targeting a removed/replaced VM. Ordinary progress events do not wait for lifecycle I/O.

Fixes #1269.

Validation

Linux x86_64, Rust 1.92.0, debug tests; no QEMU/TDX VM or storage-throughput benchmark:

  • cargo test -p dstack-vmm: 212 passed, 3 existing root/network-namespace tests ignored.
  • cargo fmt -p dstack-vmm -- --check and CI-style production clippy passed.
  • A single-thread Tokio test stalls persistence on a channel (5-second failure guard), confirms other events/state access stay responsive, cancels the request, and checks lifecycle exclusion plus ordered subsequent writes.
  • Temporary-directory tests check persisted contents, shutdown markers, error propagation, and stale-VM rejection.

This removes executor/global-lock occupancy by persistence; it does not make fsync itself faster or claim a production speedup percentage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vmm: guest event persistence holds the global state lock and blocks the async executor

1 participant