feat(system-tests): boot nested VMs on the local backend by basvandijk · Pull Request #11265 · dfinity/ic

basvandijk · 2026-08-21T14:31:39Z

Stacked on #11270, which gives the local backend's ic-gateway a resolvable domain and a certificate that nodes trust. Review that one first.

Problem

Nested system-tests (rs/tests/nested) drive the whole node deployment chain: SetupOS installs HostOS onto a blank disk, HostOS boots a nested GuestOS VM, and that GuestOS registers with the NNS. All five non-bare-metal targets were pinned to Farm because setup_and_start_nested_vms had only a Farm code path.

What was missing

Three things in the driver:

the install target was built as a DiskImage::Url whose _URL variable run_systest.sh never sets under the local backend;
setup_and_start_nested_vms needed its Local arm;
NESTED_CONFIG_IMAGE_PATH claimed .zst for a file nothing compresses, which the local backend would have run unzstd over.

The fourth — the nested node's NNS URL — is what #11270 is for. A node that is not in the registry yet cannot pass the replicas' firewall, so it registers through the ic-gateway; that needed the gateway to be reachable by name and trusted, which it now is on both backends. nested_nns_url is consequently backend-agnostic.

Also: pad_to_request_alignment now refuses a GPT image rather than silently displacing its backup header, and the nested VM drops from 32 GiB to 16 GiB — the floor, since the driver takes 8 GiB off and HostOS then takes 4 more, leaving the GuestOS the same 4 GiB every other node VM gets. That reduction applies on Farm too; all 20 Farm nested targets pass with it.

The blocker: SetupOS halts because chrony can never synchronise

check-ntp.sh is the one SetupOS check not gated on ic.setupos.run_checks, so setupos-disable-checks does not skip it. It waits 60s for NTPSynchronized and then halts the installation forever — log_and_halt_installation_on_error ends in sleep infinity.

Serving NTP to the group ourselves does not fix it, and the first two commits here do exactly that before it is reverted. chrony's authselectmode defaults to mix, and sources.c then marks the authenticated sources required as soon as both authenticated and unauthenticated sources are configured — precisely what chrony.conf does in pairing fourteen nts servers with two plain pools. Sources configured from names that never resolve still count. So the clock can only ever be synchronised from one of those fourteen NTS servers; the driver's SNTP server was polled every 64s for a whole run and never selected, with chronyd saying so in the node journals:

Can't synchronise: no required source in selectable sources

Reproduced against chronyd 4.8 directly: given that config shape it polls a reachable plain server for 100s without selecting it, and selects it in 4s once authselectmode ignore is added. Satisfying the check for real would mean serving NTS under one of those fourteen hostnames with a certificate the guest trusts, which no test harness can arrange.

So the check is gated instead, like the four that also need connectivity, and the reasoning is recorded on check_ntp. Nothing is lost: every test VM takes its clock from the hypervisor via -rtc base=utc.

Scope

registration and both *_upgrade_smoke_test targets are un-pinned, along with the _head_nns variants system_test_nns generates from them — backend is per target definition. All six were run locally before dropping the pins.

The two *_from_latest_release_to_current targets stay pinned: they install from a prebuilt mainnet SetupOS image, of which setupos-disable-checks can only rewrite the grub boot args, so their check-ntp.sh predates this gate.

One thing for reviewers to weigh: this takes the non-manual nested _local targets from two to six, and system_tests.bzl:342 reserves CPU only — ceil(cpus / cpus_oversubscription_factor) = 9 of the 27 these declare — with no accounting for memory or disk. Three can therefore run concurrently on a 32-core runner, for ~84 GiB of nominal VM memory and ~60 GiB of qcow2 overlays. cpus_oversubscription_factor = 1 would serialise them if that turns out to matter; no target in the repo sets it today, so this is left at the default.

🤖 Generated with Claude Code

Copilot

Pull request overview

Enables nested VM system tests on the local QEMU backend while preserving Farm behavior.

Changes:

Adds local SetupOS boot, resource, networking, and NNS registration support.
Provides local DNS/SNTP services required by SetupOS.
Enables the registration test locally and adds safety around GPT image alignment.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`rs/tests/nested/src/util.rs`	Whitelists local nested-node registration traffic.
`rs/tests/nested/src/lib.rs`	Reduces local nested VM memory allocation.
`rs/tests/nested/BUILD.bazel`	Enables local registration testing and updates TODOs.
`rs/tests/driver/src/driver/test_env_api.rs`	Removes obsolete empty-disk URL helpers.
`rs/tests/driver/src/driver/serve_ntp_task.rs`	Implements and tests the local SNTP server.
`rs/tests/driver/src/driver/resource.rs`	Selects backend-appropriate empty disks.
`rs/tests/driver/src/driver/nested.rs`	Corrects the raw config-image extension.
`rs/tests/driver/src/driver/mod.rs`	Exposes the NTP task module.
`rs/tests/driver/src/driver/local_backend.rs`	Adds NTP DNS records and GPT-safe alignment.
`rs/tests/driver/src/driver/ic_images.rs`	Adds backend-aware SetupOS and empty-disk selection.
`rs/tests/driver/src/driver/group.rs`	Runs NTP as a supervised local task.
`rs/tests/driver/src/driver/bootstrap.rs`	Boots nested VMs locally and selects a reachable NNS URL.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (1)

rs/tests/driver/src/driver/bootstrap.rs:586

The Local branch panics before reaching the dummy fallback when no IC/NNS was created: TopologySnapshot::root_subnet() calls root_subnet_id(), which explicitly panics if the registry has no root subnet (test_env_api.rs:547-559). This breaks the existing //rs/tests/node:launch_single_host_local setup, which starts only NestedNodes and intentionally creates no IC infrastructure. Guard the topology lookup (for example with env.prep_dir("")) and use the dummy URL when no prep state exists.

        SystemTestBackend::Local => env
            .topology_snapshot()
            .root_subnet()
            .nodes()
            .next()

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Copilot

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

The nested system-tests install a node from SetupOS, and SetupOS refuses to install onto a machine whose clock is not synchronized. `check-ntp.sh` waits 60s for `timedatectl show -p NTPSynchronized` to become `yes`, then halts the installation forever -- `log_and_halt_installation_on_error` ends in `sleep infinity`. It is the one SetupOS check that is *not* gated on `ic.setupos.run_checks`, so the `setupos-disable-checks` pass that turns a dev image into a test image, which does skip the age, hardware, network and guestos-elected checks, leaves this one running. On the local backend chrony can never satisfy it. `chrony.conf` names fourteen public NTS servers and two NTP pools; the backend runs in a network namespace with no external connectivity and its `dnsmasq` runs `--no-resolv`, so every one of those names comes back REFUSED. No source, no sync, no installation. Gating the check like the other four would have been a one-line change, but it only reaches images we build ourselves: `guestos_upgrade_from_latest_release_to_current` and its HostOS sibling boot a prebuilt *mainnet* SetupOS image, and `create_test_img` can only rewrite that image's grub boot args. So serve NTP instead. That works with any image, and it removes a divergence rather than hiding one: HostOS and GuestOS chrony now report a synchronized clock on the local backend as they do on Farm, instead of running permanently unsynchronized behind a clock that only happens to be right because every VM reads the host's through `-rtc base=utc`. `serve_ntp_task` answers SNTP on UDP 123 at the group's gateway, and `start_dnsmasq` seeds its hosts-file with the two pool names pointing there. The gateway already carries the resolver addresses GuestOS is hard-coded to query, so this puts the group's NTP service on the one address every guest talks to anyway. A dedicated address was not an option: the `/64`'s subnet-id field is two bits wide and all four values are taken -- nodes, management, journald and the file server. The fourteen NTS names are deliberately left unresolvable. Answering them would mean terminating an NTS-KE TLS handshake on port 4460, and `prefer` only biases chrony's selection, so an unreachable preferred source does not stop it selecting the pool source; a REFUSED lookup is the cheaper outcome. Seeding the records in `start_dnsmasq` rather than through `add_dns_record` means they are there for the first guest to boot, with no SIGHUP to race, and `add_dns_record` appends, so the records tests register later are unaffected. Binding a privileged port needs no new capability work: the driver created the user namespace it runs in, so it holds `CAP_NET_BIND_SERVICE` in its effective set, and task subprocesses fork rather than exec. The task itself is modelled on `serve_files_task` down to the plan wiring, so the scheduler kills it with the subtree instead of treating its non-return as a failure. The two DNS records are seeded for *every* local group, not just the nested ones. That is deliberate -- nothing else in the local suite resolves these names, and `dnsmasq` already REFUSES every other external name -- and it is what makes the clock right for ordinary GuestOS nodes too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A nested test (`rs/tests/nested`) drives the whole node-deployment chain: SetupOS installs HostOS onto a blank disk, HostOS boots a nested GuestOS VM, and that GuestOS registers with the NNS. All five non-bare-metal targets were pinned to Farm because `setup_and_start_nested_vms` had only a Farm code path. Most of what the local backend needed was already there, and the parts that matched did so exactly rather than by luck. `allocate_resources` already had a `Local` arm. `write_nested_vm` derives the HostOS and GuestOS addresses by SLAAC'ing deterministic MACs into the allocated VM's `/64`, which the backend's `vm_mac` and `create_vm` feed unchanged. `create_setupos_config_image` writes `ipv6_gateway = <prefix>::1`, which is `group_gateway_ipv6` on the group bridge -- and HostOS pings that gateway through each candidate NIC before accepting it, so the address has to be real, not just consistent. Three MACs end up behind one TAP (SetupOS, then HostOS, which writes its derived MAC onto the outer NIC, then the inner GuestOS), which a Linux bridge learns without being told. And HostOS's nftables only accepts SSH from its `dfinity_dcs` set, which contains `2a00:fb01:400::/56` -- the prefix this backend moved to in 4d4ab2c, for unrelated reasons. Four things were actually missing. `get_resource_request_for_nested_nodes` built the install target as a `DiskImage::Url` unconditionally, and `run_systest.sh` never sets `ENV_DEPS__EMPTY_DISK_IMG_URL` under the local backend (it only hashes the file), so the URL was absent -- and `start_vm` panics on a URL primary image anyway. Both that image and the SetupOS image now go through `ic_images` getters shaped like `get_guestos_disk_image`, which resolve to a local path when there is one. `setup_and_start_nested_vms` gets its `Local` arm, mirroring the sibling `setup_and_start_vms`: attach the SetupOS image and the config image, then start. No boot-order plumbing is needed, because the shape the backend already produces is the one Farm relies on -- the primary disk carries `bootindex=1` but is all zeros, so the firmware falls through to SetupOS, and once `install-hostos.sh` has written HostOS onto it and created the `IC-OS` UEFI entry (persisted in the per-VM `OVMF_VARS.fd`), that entry wins. `NESTED_CONFIG_IMAGE_PATH` claimed to be `config.img.zst`, but `build-setupos-config-image.sh` writes a plain FAT image -- nothing compresses it. On Farm the name is only a multipart form field, so the lie was invisible; the local backend dispatches on the extension and would have run `unzstd` over a FAT filesystem. SetupOS finds the image by its `OVERRIDE` label either way, so the name is now honest. The nested node's NNS URL had to change. On Farm it is the ic-gateway, and that is not incidental: a nested node is not in the registry yet, so it cannot pass the replicas' firewall, while the gateway fronts an API boundary node that can. Neither of the gateway's properties survives locally -- it serves a self-signed certificate for a `.local` domain the group's `dnsmasq` knows nothing about, and a nested GuestOS has no knob for either, since `make_bootstrap_options` gives it no trust anchors the way a driver-managed node's config image does. So point it at the NNS node over plain HTTP, exactly as `create_config_disk_image` does for every other node in the group, and have the test open the group's range with `with_group_wide_firewall_whitelist` so an unregistered node can reach it. Keeping that opt-in per test preserves the deliberate narrowing of the default whitelist to the driver's own addresses. Two smaller things. `pad_to_request_alignment` ran over every extra disk, and its own doc-comment warned that boot disks must not be padded, because a GPT keeps its backup header in the last sector; the SetupOS installer disk is the first boot disk to be attached this way. Its partitions sit on 1 MiB boundaries, so no padding happens today -- but silently, so it now detects the GPT signature and refuses rather than corrupting a partition table. And the nested VM's memory comes down from the Farm default of 32 GiB to 16 GiB on the local backend, where all VMs share one host. 16 GiB is the floor, not a guess: the driver takes `HOSTOS_MEMORY_RESERVED_GIB` (8) off before writing `dev_vm_resources.memory`, and HostOS then takes `UPGRADE_VM_MEMORY_GIB` (4) off for the upgrade VM, so 16 leaves the GuestOS the same 4 GiB every other node VM gets, while 12 would size it 0 GiB and libvirt would refuse the domain. Only `registration` is un-pinned. The four upgrade targets have not been run on the local backend yet -- they additionally download an update image from the per-group file server and survive a reboot inside the test's timeouts -- so their TODOs now say that, rather than claiming the backend cannot boot a nested VM. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Empty commit to re-run CI now that the PR carries `CI_ALL_BAZEL_TARGETS`. The label sets `skip_long_tests=false`, which is what these nested targets need: they are tagged `long_test`, so the default PR run skips them entirely. It reaches both halves of the split. `ci-main` excludes `local_system_test`, so it runs the Farm `//rs/tests/nested:registration` -- the regression check that the new `SystemTestBackend::Local` arms did not disturb the Farm path they branch away from. `ci-rbe-evaluation` is the mirror image: it excludes `farm_system_test` and so runs `//rs/tests/nested:registration_local` on the Namespace cluster, which is the end-to-end verification of the change itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two review comments. `get_setupos_disk_image` had exactly one caller, in a branch that had already matched on `SystemTestBackend::Local` -- so it built a `DiskImage`, returned it through a Farm arm that could not be reached, and forced the caller to destructure it with a `let ... else { bail! }` for a case that could not happen. Read `ENV_DEPS__SETUPOS_DISK_IMG_PATH` at the call site instead, which is also what the existing `Local` arms in `resource.rs` and `universal_vm.rs` do. `get_empty_disk_image` stays: it is genuinely called for both backends, because `get_resource_request_for_nested_nodes` needs a `DiskImage` either way. The 16 GiB nested VM is now the default for Farm too, rather than a local-backend override, which deletes the override and its explanation. Farm was giving the nested GuestOS 20 GiB (32 minus the driver's 8 and HostOS's 4) where every other node VM in a test gets `DEFAULT_MEMORY_KIB_PER_VM`, 4 GiB. A nested node runs the same replica as its unnested peers, so there was nothing for the extra 16 GiB to do -- the figure looks inherited from production sizing, where a real node has 512 GiB. The only other consumer of the default is `//rs/tests/node:launch_single_host`; `rs/tests/nested/nns_recovery` sets both vcpus and memory explicitly (40 / 48 GiB) and is unaffected. While there: two `Default` impls in `rs/ic_os/config` claimed to be `(HOSTOS_VCPUS_PER_VM / 2, HOSTOS_MEMORY_KIB_PER_VM / 2)`. That was already wrong for vcpus (8 / 2 = 4, not the 16 they set) and wrong for both, since nested tests get `total - reserved` rather than half. Halving the memory constant would have made it wronger still, so say what those defaults actually are: a fallback for a `deployment.json` that omits the field, which nested tests never take. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

SetupOS output stops one second into `setupos.sh` and never resumes. What is lost is everything that matters: the per-check progress, and the "INTERNET COMPUTER - SETUP - FAILED" banner that `log_and_halt_installation_on_error` prints -- including the chrony service log and `chronyc sources` dump that `check-ntp.sh` attaches to it -- right before it goes to `sleep infinity`. The installer looks like it hung for no reason, and the VM has to be reverse-engineered from the outside. The cause is not the installer. `output-wrapper.sh` runs `setupos.sh 2>&1 | tee /dev/ttyS0`, so `tee` holds a single file descriptor on that tty for the whole install. `serial-getty@.service` carries `TTYVHangup=yes`, so systemd issues `TIOCVHANGUP` on /dev/ttyS0 when the getty on it starts; the kernel then swaps every *other* descriptor on that tty to `hung_up_tty_fops`, whose `write` returns EIO. `tee` drops the failing output, keeps writing to the one that still works, and the install runs on -- silently. Nor does tee's stdout help: `StandardOutput=tty` is /dev/console, which the boot args' trailing `console=tty0` points at the virtual console, and a VM with no VGA registers that as `Console: colour dummy device`. `Type=idle` was meant to prevent exactly this by keeping the unit behind the gettys, but it only defers while the job queue is busy. This unit waits for systemd-networkd-wait-online and serial-getty@ttyS0 waits for systemd-user-sessions; the network releases both at once and the installer wins by about a second, which is precisely the wrong side of the hangup. Seen at +1.0s in every run so far -- one local, three CI attempts. So order the unit after the getty, and let `tee` open the tty once the hangup is already behind it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`check-ntp.sh` is the one SetupOS check not gated on `ic.setupos.run_checks`, so the `setupos-disable-checks` pass that turns a dev image into a test image -- which does skip the age, hardware, network and guestos-elected checks -- leaves this one running. In a hermetic test environment it cannot pass, and failing it is fatal: `log_and_halt_installation_on_error` ends in `sleep infinity`, so the node is never installed. This is what blocks the nested system-tests on the local backend, whose network namespace has no external connectivity. The obvious remedy -- serving NTP to the test network -- does not work. chrony's `authselectmode` defaults to `mix`, and `sources.c` then marks the *authenticated* sources `require`d as soon as both authenticated and unauthenticated sources are configured, which is exactly what `chrony.conf` does in pairing fourteen `nts` servers with two plain pools. Sources configured from names that never resolve still count towards that. So the clock can only ever be synchronized from one of those fourteen NTS servers, and a test-local NTP server is never selected however correct its replies: chronyd reports "Can't synchronise: no required source in selectable sources" and `NTPSynchronized` stays `no`. Verified against chronyd 4.8 -- with that config shape it polls a reachable plain server for 100s without selecting it, and selects it in 4s once `authselectmode ignore` is added. Satisfying the check for real would mean serving NTS under one of those hostnames with a certificate the guest trusts, which no test harness can arrange. So gate it, like the four checks that also need connectivity. Nothing is lost: every test VM takes its clock from the hypervisor. `set_hwclock_utc` stays unconditional -- it needs no network. The two `*_from_latest_release_to_current` targets install from a prebuilt mainnet SetupOS image, of which `setupos-disable-checks` can only rewrite the grub boot args, so they keep hanging until this gate has shipped in a release. Their TODOs now say so rather than "not verified yet". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This reverts commit e9d4fba. `serve_ntp_task` cannot do what it was added for. It rested on `prefer` merely biasing chrony's selection, so that an unreachable preferred NTS source would not stop chrony selecting the plain pool source pointed at the driver. That much is true of `prefer` -- but it is not the whole story. `authselectmode` defaults to `mix`, under which chrony marks the authenticated sources `require`d as soon as both kinds are configured, so no unauthenticated source is ever selected. In `registration_local` the guests polled the driver's server every 64s for the entire run and stayed unsynchronized throughout, with chronyd reporting "Can't synchronise: no required source in selectable sources". The previous commit gates `check-ntp.sh` instead, which is what actually unblocks the nested tests, so none of this is needed: no local test requires a synchronized clock, and the two DNS records seeded here existed only to point chrony at the server. The comment on `check_ntp` now records why serving NTP is a dead end, so this is not attempted a second time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This reverts commit 117c1f7, which ordered `setupos.service` after `serial-getty@ttyS0.service` so that `output-wrapper.sh`'s `tee` would open /dev/ttyS0 only once the getty's hangup was behind it. Measured on `registration_local`, it changes nothing: the console still goes dark 1.02s after `setupos.sh - Start`, exactly as before, with the getty's terminal probe and issue banner as the last thing on the wire. Two reasons, either of which is fatal to the approach. `serial-getty@.service` is itself `Type=idle`, so its start job completes when systemd forks the child, well before that child waits out the idle pipe, sets the tty up and execs -- the ordering is satisfied within microseconds of the fork and says nothing about when the hangup happens. And the hangup is not only systemd's `TTYVHangup=yes`: agetty calls `vhangup()` on its own (`/dev/%s: vhangup() failed: %m` is in the binary), so no unit directive can prevent it either. Losing the install log is worth fixing -- it is what made the `check-ntp.sh` hang take a full investigation instead of one glance at a console -- but it has to be fixed where the descriptor lives, in `output-wrapper.sh`, not by ordering. Doing that means rewriting the wrapper every IC-OS install path depends on, so it is left for a change of its own rather than smuggled in here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ackend `guestos_upgrade_smoke_test` and `hostos_upgrade_smoke_test` were pinned to Farm pending verification: on top of installing a node from SetupOS they download an update image from the per-group file server and have to survive a reboot inside the test's timeouts. Both pass, so drop the pin. `backend` is per target definition, so this un-pins the `_head_nns` variants `system_test_nns` generates as well. All six were run locally, one at a time: //rs/tests/nested:registration_local PASSED in 892.4s //rs/tests/nested:registration_head_nns_local PASSED in 740.8s //rs/tests/nested:guestos_upgrade_smoke_test_local PASSED in 947.9s //rs/tests/nested:guestos_upgrade_smoke_test_head_nns_local PASSED in 835.9s //rs/tests/nested:hostos_upgrade_smoke_test_local PASSED in 1297.2s //rs/tests/nested:hostos_upgrade_smoke_test_head_nns_local PASSED in 1247.1s Each upgrade target is a real upgrade rather than a re-run of `registration`: the GuestOS one goes from version `0000...0000` to `0000...0000-test` after fetching the update image from `http://[<group files>]:8080/<hash>`, and the HostOS one additionally reaches the HostOS over SSH from the driver -- the first time anything has exercised that path on this backend, and it works unchanged because HostOS's nftables `dfinity_dcs` set already covers the group prefix. Against budgets of 30 min per test and 40 min overall there is comfortable headroom. Note this takes the non-manual nested `_local` targets from two to six, and `system_tests.bzl` reserves CPU only -- `ceil(cpus / cpus_oversubscription_factor)` = 9 of the 27 these declare -- with no accounting for memory or disk. Three can therefore run concurrently on a 32-core runner, for ~84 GiB of nominal VM memory and ~60 GiB of qcow2 overlays. If that starts biting, `cpus_oversubscription_factor = 1` on these targets serialises them. Still pinned are the two `*_from_latest_release_to_current` targets, which install from a prebuilt mainnet SetupOS image whose `check-ntp.sh` predates the `ic.setupos.run_checks` gate; see their TODOs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`nested_nns_url`'s `Local` arm reached the NNS node through `env.topology_snapshot().root_subnet()`, and both of those panic rather than returning an error: the first with "No no-name Internet Computer" when no IC was set up at all, the second with "root subnet id is not set" when the registry has one but no root subnet. Only the third step, `nodes().next()`, was fallible, so the `unwrap_or_else(dummy)` fallback was unreachable in the two cases that actually occur. The Farm arm has no such hole, which is why the asymmetry was easy to miss -- `get_deployed_ic_gateway` merely returns an `Err`. An NNS-less nested VM is a legitimate configuration, not a bug: `rs/tests/node/launch_single_host.rs` sets up no `InternetComputer` at all and only waits for HostOS to accept an SSH login. It is the documented way to bring a single HostOS up by hand (`bazel run //rs/tests/node:launch_single_host -- --keepalive`), and being `manual` it gets no CI signal, so nothing would have caught this. Add `TopologySnapshot::try_root_subnet`, which returns `None` instead of panicking -- shaped like the neighbouring `unassigned_nodes_config` -- and let `root_subnet` delegate to it so the two cannot drift. `nested_nns_url` then threads `safe_topology_snapshot().ok()` through to it and falls back to the dummy URL, as the Farm arm always did. Verified with `//rs/tests/node:launch_single_host_local`, which previously panicked in setup within seconds: Task setup PASSED in 15.06s [setup:bootstrap.rs:574] No NNS node or gateway found, using dummy URL SetupOS then installs and the VM reboots into HostOS unaided. That target still fails afterwards, for an unrelated reason that predates this change and is left alone: `await_can_login_as_admin_via_ssh` has no timeout parameter and takes the shared 500s `SSH_RETRY_TIMEOUT`, but its setup phase is only 15s here (there is no IC to build), so the whole ~5.5 min SetupOS install has to fit inside that budget and HostOS finishes booting with ~1.5 min to spare -- not enough for sshd. Giving that test a longer wait means either a new trait method or a `retry_with_msg!` around the call, in a target this PR does not otherwise touch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Now that the local backend's ic-gateway has a resolvable domain and a certificate issued by the CA every dev IC-OS image trusts, a nested node can register through it exactly as it does on Farm. `nested_nns_url` collapses to a single backend-agnostic body, and three workarounds go with it. `with_group_wide_firewall_whitelist()` was only needed because the node talked straight to a replica's `:8080` while still absent from the registry, so nothing whitelisted it. The gateway fronts an API boundary node that *is* in the registry, so the group's firewall can stay narrowed to the driver's own addresses, as `init_ic` intends. `try_root_subnet` existed only so the Local arm could tolerate a group with no NNS -- `rs/tests/node/launch_single_host.rs` stands up a nested VM and no IC at all. The gateway path gets that for free, because `get_deployed_ic_gateway` returns an `Err` rather than panicking, so `root_subnet` goes back to its original body. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions Bot added the feat label Aug 21, 2026

basvandijk requested a balanced review from Copilot August 21, 2026 14:31

Copilot started reviewing on behalf of basvandijk August 21, 2026 14:32 View session

basvandijk added the CI_ALL_BAZEL_TARGETS Runs all bazel targets label Aug 21, 2026

Copilot AI reviewed Aug 21, 2026

View reviewed changes

basvandijk commented Aug 21, 2026

View reviewed changes

Comment thread rs/tests/driver/src/driver/bootstrap.rs Outdated

basvandijk commented Aug 21, 2026

View reviewed changes

Comment thread rs/tests/nested/src/lib.rs Outdated

basvandijk commented Aug 22, 2026

View reviewed changes

Comment thread rs/tests/driver/src/driver/nested.rs Outdated

basvandijk commented Aug 22, 2026

View reviewed changes

Comment thread rs/tests/driver/src/driver/resource.rs Outdated

basvandijk requested a balanced review from Copilot August 22, 2026 15:49

Copilot started reviewing on behalf of basvandijk August 22, 2026 15:49 View session

Copilot AI reviewed Aug 22, 2026

View reviewed changes

Comment thread ic-os/components/setupos/check-ntp.sh

Comment thread rs/tests/driver/src/driver/bootstrap.rs Outdated

basvandijk requested a balanced review from Copilot August 22, 2026 17:27

Copilot started reviewing on behalf of basvandijk August 22, 2026 17:27 View session

Copilot AI reviewed Aug 22, 2026

View reviewed changes

basvandijk requested a balanced review from Copilot August 22, 2026 20:10

Copilot started reviewing on behalf of basvandijk August 22, 2026 20:11 View session

Copilot AI reviewed Aug 22, 2026

View reviewed changes

basvandijk mentioned this pull request Aug 22, 2026

feat(system-tests): make the local backend's ic-gateway reachable by name #11270

Draft

basvandijk changed the base branch from master to bas/local-backend-gateway-cert August 22, 2026 22:48

basvandijk force-pushed the bas/local-backend-nested branch from 2476795 to 688f7e0 Compare August 23, 2026 00:12

basvandijk requested a balanced review from Copilot August 23, 2026 00:12

Copilot started reviewing on behalf of basvandijk August 23, 2026 00:13 View session

Copilot AI reviewed Aug 23, 2026

View reviewed changes

basvandijk and others added 6 commits August 23, 2026 01:03

rename generic_workload_engine to unit_tests

0f4061a

Remove 12 GiB reference

e80d7e5

basvandijk and others added 8 commits August 23, 2026 01:03

rm comment

05cb4f2

basvandijk force-pushed the bas/local-backend-nested branch from 688f7e0 to c237df4 Compare August 23, 2026 01:04

Navigation Menu

Uh oh!

feat(system-tests): boot nested VMs on the local backend - #11265

feat(system-tests): boot nested VMs on the local backend#11265
basvandijk wants to merge 14 commits into
bas/local-backend-gateway-certfrom
bas/local-backend-nested

basvandijk commented Aug 21, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

basvandijk commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

What was missing

The blocker: SetupOS halts because chrony can never synchronise

Scope

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

basvandijk commented Aug 21, 2026 •

edited

Loading