fix(os): harden the guest initramfs early-boot path

[kvinwang]

First: is os/yocto/.../dstack-initscript/init actually shipped?

Two of the three items below are about a file under os/yocto/, which CLAUDE.md
calls deprecated. That would normally be reason enough to close them. It is not,
and this is the finding that decides whether the rest of the PR exists.

os/mkosi/scripts/make-release-artifacts.sh:92 installs that exact file, verbatim,
as /init of the initramfs that goes into the shipped UKI:

install -m0755 "$ROOT/os/yocto/layers/meta-dstack/recipes-core/images/dstack-initscript/init" "$ird/init"

The surrounding block assembles the initramfs from the mkosi tree (busybox at
:56, veritysetup at :68), then ukify build --initrd=.../initramfs.cpio.gz
turns it into dstack-uki.efi. So this script is the early boot of both
backends and of every image dstack ships today. It is not dead code, and it is
the one piece of the boot path with no console and no recovery.

acceptance.sh now asserts that link, so if mkosi ever stops installing this
file the tests below stop claiming to cover a shipped artifact.

---

O-a — the initramfs mounts /proc, /sys, /run, /dev with no options

Real. The whole finding rested on a premise that had never been checked:
that systemd does not repair the options afterwards. It was checked.

Booted real systemd on the shipped base (Debian trixie, systemd 257.13-1~deb13u1)
with /run, /proc and /sys mounted bare first, exactly as the initramfs
leaves them, then read /proc/self/mountinfo once the system reported running:

### BEFORE systemd:
6481 12283 0:671 / /run  rw,relatime - tmpfs tmpfs rw,inode64
6484 12286 0:672 / /proc rw,relatime - proc  proc  rw
6485 12292 0:3980 / /sys rw,relatime - sysfs sysfs rw

### AFTER systemd started:
6481 12283 0:671 / /run  rw,relatime - tmpfs tmpfs rw,inode64
6484 12286 0:672 / /proc rw,relatime - proc  proc  rw
6485 12292 0:3980 / /sys rw,relatime - sysfs sysfs rw

Unchanged. mount_setup()'s mount_one() returns early for a path that is
already a mount point, and the initramfs mount --moves all four across
switch_root — verified that mount --move preserves options, so nothing in
the chain restores them.

The concrete consequence, on the same run:

$ stat -c '%A %a' /run      →  drwxrwxrwt 1777

A bare tmpfs takes the kernel default of 1777, not the 0755 systemd would
have used. So every CVM runs with /run world-writable and suid/dev
honoured, and /proc and /sys without nosuid,nodev,noexec — purely as an
accident of the custom initramfs. A stock Debian boot gets all of these.

This is the cheap-bounded-hardening case rather than a live exploit: the CVM is
single-tenant, so per the project's threat model file modes between workloads
are not a boundary. The argument for fixing it is that the image is less
hardened than the distro default for no reason, and nothing downstream can
correct it.

Failing test first

New os/tests/test-initramfs-early-mounts.sh rebuilds the initramfs layout
make-release-artifacts.sh assembles (same busybox, same applet symlinks, empty
/proc, /sys, /run, /dev), chroots into it, runs the real init, and
reads the options back out of the kernel. Against next:

FAIL: /ir/proc is missing 'nosuid' (mounted rw,relatime)
FAIL: /ir/proc is missing 'nodev' (mounted rw,relatime)
FAIL: /ir/proc is missing 'noexec' (mounted rw,relatime)
FAIL: /ir/sys is missing 'nosuid' (mounted rw,relatime)
FAIL: /ir/sys is missing 'nodev' (mounted rw,relatime)
FAIL: /ir/sys is missing 'noexec' (mounted rw,relatime)
FAIL: /ir/run is missing 'nosuid' (mounted rw,relatime)
FAIL: /ir/run is missing 'nodev' (mounted rw,relatime)
FAIL: /ir/dev is missing 'nosuid' (mounted rw,relatime)
FAIL: /run was not mounted with mode=0755 (kernel default for tmpfs is 1777): ... rw,relatime - tmpfs tmpfs rw,inode64
10 initramfs mount-option assertion(s) failed

After:

ok: /ir/proc rw,nosuid,nodev,noexec,relatime
ok: /ir/sys  rw,nosuid,nodev,noexec,relatime
ok: /ir/run  rw,nosuid,nodev,relatime
ok: /ir/dev  rw,nosuid
ok: /ir/run mode=755

It runs the script rather than grepping it, because the failure that would
actually ship is busybox silently dropping an option it does not parse. All
four option sets were confirmed accepted by busybox 1.37.0 — the version Debian
trixie ships as busybox-static, which is what mkosi.conf installs.

Fix

nosuid,nodev,noexec for /proc and /sys; nosuid,nodev,mode=0755 for
/run; nosuid,strictatime,mode=0755 for /dev — the same values systemd's
own mount_table uses. /dev keeps dev and exec because it exists to carry
device nodes.

Blast radius

Changes the initramfs, therefore the UKI, therefore the measurements — a new OS
image release and a KMS whitelist entry, as with any guest-OS change. Behaviour
risk is a workload that relied on setuid or device nodes on /run, or on exec
from /proc//sys; containers get their own /run in their own mount
namespace, so this is the host side of the CVM only. Deliberately not
changed: size=/nr_inodes= limits on /run. systemd would cap it at 20% of
RAM; adding a cap here is a real behaviour change on small CVMs and is a
separate decision from the option flags.

---

O-b — After=chronyd.service names a unit that may not exist

Refuted. The alias genuinely exists in the shipped image. Evidence:

1. Debian trixie's chrony package ships no chronyd.service. It ships
   chrony.service with:

   [Install]
   Alias=chronyd.service
   WantedBy=multi-user.target
   

2. [Install] aliases only materialise on enable — correct. But
   80-dstack.preset already carries enable chrony.service, and mkosi runs
   systemctl --root=/buildroot preset-all over the image tree
   (mkosi/__init__.py:3128, called from run_preset at :4022). Reproducing
   exactly that on trixie with this repo's preset files:

   $ systemctl preset-all
   $ ls -la /etc/systemd/system/ | grep chrony
   lrwxrwxrwx 1 root root 38 chronyd.service -> /usr/lib/systemd/system/chrony.service
   

3. The prod profile's RemoveFiles= does not touch /etc/systemd/system, so
   the symlink survives into the released rootfs.

So After=chronyd.service resolves, and chronyd does run. No rename.
Renaming to chrony.service would be churn on a measured file presented as a
fix.

What the investigation did surface is that the ordering is load-bearing on a
line in a different file, with nothing recording the coupling: delete
enable chrony.service from the preset and two things break silently at once
— chronyd never starts, and the After= resolves to nothing. acceptance.sh
now pins it, conditional on the After= still being there.

The related chronyc makestep claim — confirmed, and fixed

dstack-prepare.sh:100 was chronyc makestep || log "Warning: ...". Ran it
against a chronyd with no reachable source:

$ chronyc tracking | grep Leap
Leap status     : Not synchronised
$ chronyc makestep
200 OK
$ echo $?
0

makestep only tells chronyd it may step instead of slew. It answers 200 OK
and exits 0 whether or not a source has ever been reached, so the || log
branch is unreachable and the guard guards nothing. dstack-prepare.sh then
proceeds to dstack-util setup — the RA-TLS handshake to the KMS — on an
arbitrary clock, and both sides of that are wall-clock judgements: quote
collateral freshness and certificate validity windows.

Fix: keep makestep, add a bounded chronyc waitsync 30 0 0 1 after it.
Measured the bound rather than assuming it — waitsync 5 0 0 1 against an
unreachable source returned exit=1 after 4s, so the 30-try form costs ~29s
worst case and returns non-zero rather than hanging. The || log is kept, so an
NTP-less CVM still boots; it just no longer claims a clock it does not have.
Unbounded waiting was rejected: this unit has FailureAction=reboot.

---

O-c — dm-verity opened with no corruption policy

Real, and a policy call. Please overrule me cheaply if you disagree.

Today init opens the rootfs with no mode, which the kernel confirms:

mode [none]                     → table: ... sha256 <root> <salt>
mode [--panic-on-corruption]    → table: ... sha256 <root> <salt> 1 panic_on_corruption
mode [--restart-on-corruption]  → table: ... sha256 <root> <salt> 1 restart_on_corruption
mode [--ignore-corruption]      → table: ... sha256 <root> <salt> 1 ignore_corruption

With no mode, dm-verity returns EIO to the faulting reader and nothing else.
A tampered rootfs block surfaces late, scattered across whichever process
happened to read it, while the CVM keeps serving attested traffic from a rootfs
whose integrity guarantee has already failed.

The two alternatives, and what each gives the host

	what it does	what the host gains
--panic-on-corruption	panic() — stops every CPU at once	a reboot, from one flipped bit
--restart-on-corruption	kernel_restart() — runs reboot notifiers and device shutdown first	a reboot, from one flipped bit
(today)	EIO to the reader	a CVM that keeps running on a rootfs it can no longer vouch for

I chose --panic-on-corruption. Reasoning, in the order I weighted it:

1. The "a host that can flip one bit gains a reliable reboot" objection is real
   but not a new capability. The host already owns the VM lifecycle and can
   reset or kill the CVM outright. Halting concedes nothing it does not have.
2. panic=1 is already on the guest kernel command line
   (os/image/kernel-cmdline.sh), so both halting modes end in a reboot
   regardless. The choice is only how the kernel gets there.
3. Given that, panic() is the safer of the two: it stops immediately, where
   kernel_restart() first runs reboot notifiers and device shutdown paths —
   i.e. more kernel code reading the rootfs that is known to be corrupt.

To overrule: change --panic-on-corruption to --restart-on-corruption on
the single veritysetup open line in
os/yocto/layers/meta-dstack/recipes-core/images/dstack-initscript/init. The
test reads the flag out of that line rather than restating it, so it keeps
passing either way. What the test refuses is the two states that are not a halt:
no policy, and --ignore-corruption.

Failing test first

New os/tests/test-initramfs-verity-policy.sh. Against next:

FAIL: init opens the verity rootfs with no corruption policy.
      dm-verity then only returns EIO to the faulting reader, so a
      tampered rootfs never stops the guest. Expected one of
      --panic-on-corruption or --restart-on-corruption on the
      'veritysetup open' line of .../dstack-initscript/init

After:

ok: init opens the rootfs with --panic-on-corruption
ok: kernel dm-verity target reports panic_on_corruption

The runtime leg builds a real verity device and reads the mode back out of
dmsetup table, which is what catches a flag the shipped veritysetup rejects
or a kernel whose dm-verity lacks the mode — either of which would leave the
guest with no rootfs and no console to debug from. Confirmed against
cryptsetup-bin 2:2.7.5-2 (trixie, the version mkosi.conf installs) and
CONFIG_DM_VERITY=y on the 6.18.40 guest kernel.

It deliberately never corrupts a block: both halting modes take down the
kernel running the test, which in a container is the host.

Blast radius

Same measurement change as O-a. Behaviourally: a CVM that today would limp along
with scattered EIO now reboots. That is the intended change. There is no
performance cost — the mode only affects the error path.

---

Gates

• os/mkosi/tests/acceptance.sh passes, including the new assertions.
• Both new tests pass; both were confirmed to fail against next first.
• New acceptance.sh assertions were negative-tested — deleting
  enable chrony.service and deleting the waitsync line each make it fail
  with the intended message.
• prek run clean on every touched file (shellcheck, trailing whitespace,
  EOF, line endings).
• Both new tests skip cleanly (exit 0) where there is no usable docker, so
  they do not become a hard CI dependency. Verified with a docker-free PATH.

What I did not verify

No guest image was built. Everything above was verified against the exact
component versions the image pins — Debian trixie, systemd 257, busybox 1.37.0,
cryptsetup 2.7.5 — but assembled in containers, not in a real CVM. Specifically
not verified end to end:

• that a full make os-image still builds and boots with these options;
• the real panic_on_corruption firing (deliberately never triggered);
• the waitsync path against a real CVM's NTP configuration.

I did not start a multi-hour mkosi build. If you want that gate before merge,
say so and I will run it.

One /dev caveat the test comments record: devtmpfs is a singleton, so a
second mount inherits the first mount's mode= and the test cannot assert
mode=0755 for /dev. In the guest the initramfs is the first mount, so the
option does apply there; only nosuid is asserted, and that one is per-mount
and did fail before the fix.