Skip to content

fix(os): ID-map the Sysbox rootfs overlay instead of cloning it - #1489

Merged
kvinwang merged 1 commit into
feat/kernel-ovl-idmapfrom
feat/sysbox-idmap-rootfs
Oct 27, 2026
Merged

kvinwang merged 1 commit into
feat/kernel-ovl-idmapfrom
feat/sysbox-idmap-rootfs

Conversation

@kvinwang

@kvinwang kvinwang commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

sysbox-mgr runs with --disable-ovfs-on-idmapped-mount, which forces Sysbox's chown fallback. That fallback clones the container rootfs into /var/lib/sysbox/rootfs/<id> and deletes the clone when Sysbox stops. Every write made inside a Sysbox container is lost on a Sysbox restart or CVM reboot; volumes are unaffected.

Dropping the flag alone is not enough. Stock Sysbox then ID-maps the overlay lower layers but chowns the upper layer, and never reverts that chown when a container fails to start. A retry sees a non-root rootfs owner, skips ID shifting, and image files show up as nobody:nogroup (reproduced with Sysbox 0.7.1 on both kernels: start a container whose command is missing, add the command, start again).

Fix

Patch Sysbox to ID-map the merged overlay mount as a whole when the kernel supports it (#1488). Nothing on disk is chowned, so a failed start leaves nothing behind, and writes stay in Docker's upper directory on the persistent data disk. Then drop the flag.

  • sysbox-libs: new IDMappedOverlayfs shift type and a probe that ID-maps a test overlay mount.
  • sysbox-ipc/sysbox-mgr: probe at startup and pass the result to sysbox-runc; the stop/pause/resume chown bookkeeping does not apply to the new mode.
  • sysbox-runc: prefer the new mode for overlay rootfs; skip both chowns.

Kernels without the backport keep the existing lower-layer mode. The four patches live next to the Yocto recipe and are applied with --fuzz=0 by both backends; sysboxMgrProtobuf.pb.go is regenerated with the generator that produced the current file (byte-identical output for the unmodified proto).

Verification

Full dev image built from this stack with repro-build.sh (mkosi, FLAVORS=dev), booted as local KVM guests with simulated TDX and a TPM-sealed data disk, once with storage_fs=zfs and once with ext4. Outer Docker uses overlay2 on both (on ZFS 2.4.0 too); sysbox-mgr reports ID-mapped overlayfs support on both and containers use the new mode (root mount idmapped overlay).

Each check passed on both filesystems:

  • Nested Docker 29.7.1 / runc 1.3.6 inside the Sysbox container.
  • Missing command → failed start → command added → retry of the same container: image files stay 0:0 and root can write. Stock 0.7.1 without the flag gives 65534:65534 here.
  • Upper-layer files keep container-native owners (0:0, 1000:1000) while running and stopped; no shifted IDs, no -sysbox workdir, no chown entries in the sysbox-mgr log.
  • Container restart, pause/unpause, stop → Sysbox restart → start, and CVM reboot with automatic restart: file hashes, owner/mode, a package installed after creation, and inner Docker data all survive.
  • docker commit records the correct owners.
  • Upstream overlayfs idmapped_mounts selftests: 6/6 pass.

Repeated on real Intel TDX hardware (same image, no_tee=false, guest detects TDX, key_provider=tpm) with both ZFS and ext4: all of the above pass. The reboot case is a cold restart of the TD with the same data disk and TPM state, since a TD exits QEMU on reboot.

Scope

  • Restarting Sysbox while its containers are running is still unsupported: stop the containers first.
  • Recreating a container (docker rm, compose or image changes) still discards its writable layer by design.
  • Nested Docker 29 should use "storage-driver": "overlay2"; the containerd image store is not compatible with Sysbox's snapshotter mount handling.

Stacked on #1488.

@kvinwang
kvinwang added this pull request to stack #1490 October 26, 2026 11:34
@kvinwang
kvinwang force-pushed the feat/sysbox-idmap-rootfs branch from 62eb17e to 7a4cdba Compare October 27, 2026 03:33
@kvinwang
kvinwang removed this pull request from stack #1490 October 27, 2026 03:34
@kvinwang
kvinwang added this pull request to stack #1491 October 27, 2026 03:34
@kvinwang kvinwang changed the title fix(os): stop forcing Sysbox rootfs cloning fix(os): ID-map the Sysbox rootfs overlay instead of cloning it Oct 27, 2026
sysbox-mgr ran with --disable-ovfs-on-idmapped-mount, which forced
Sysbox's chown fallback. That fallback clones the container rootfs into
/var/lib/sysbox/rootfs/<id> and deletes the clone when Sysbox stops,
losing every write made inside a Sysbox container.

Dropping the flag alone is not enough. Stock Sysbox then ID-maps only the
overlay lower layers and chowns the upper layer. If a container fails to
start, that chown is never reverted. The retry sees a non-root rootfs
owner, skips ID shifting, and image files show up as nobody:nogroup.

Patch Sysbox to ID-map the merged overlay mount as a whole when the
kernel supports it (the overlayfs backport in this stack). Nothing on
disk is chowned, so no ownership state can be left behind, and writes
stay in Docker's upper directory on the persistent data disk.
sysbox-mgr probes the feature at startup; kernels without it keep the
lower-layer mode.

The patches are shared with the Yocto recipe, and the mgr protobuf file
is regenerated for the new config field.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
kvinwang force-pushed the feat/sysbox-idmap-rootfs branch from 7a4cdba to 2ce985b Compare October 27, 2026 04:13
@kvinwang
kvinwang merged commit a466dfd into next Oct 27, 2026
9 checks passed
@kvinwang
kvinwang deleted the feat/sysbox-idmap-rootfs branch October 27, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant