Skip to content

DO NOT MERGE: diagnose the i686 triple fault on GitHub's AMD runners - #25

Closed
xaionaro wants to merge 13 commits into
mainfrom
qemu-multiarch-reset-diag
Closed

xaionaro wants to merge 13 commits into
mainfrom
qemu-multiarch-reset-diag

Conversation

@xaionaro

Copy link
Copy Markdown
Member

Diagnostics only, not for merge. Runs the i686 (and x86_64) QEMU tests, then tests/qemu/ci-diag-reset.sh, which records host KVM traces around the i686 guest's triple fault on the AMD runners and uploads them as the reset-diag artifact. Built on #24 (b9603e6). Will be closed after the run.

uefisettings failed to build for 32-bit targets such as i686 (E0308
at both ioctl calls, #4).
nix's ioctl_read! and ioctl_write_ptr! take a pointer to the type that
sizes the request, c_long here, and chattr passed an i64, which is
c_long only where long has 64 bits. The kernel itself reads and writes
an unsigned int (ioctl_getflags() and ioctl_setflags() in
fs/file_attr.c, Linux v6.17); only the request numbers,
_IOR('f', 1, long) and _IOW('f', 2, long), encode the size of a long.

The ioctls now take the request number built for a long, as before,
with an unsigned int buffer, and the flags are c_uint throughout, so
nothing converts or truncates them. The request numbers do not change.
clippy-i686 lints i686-unknown-linux-gnu, so a 32-bit build error fails
make lint.

This supersedes the unmerged fix on the bugfix/issue4 branch (36a405b),
which keeps a c_long buffer, converts it with as and makes the flags
i32: the kernel reads and writes the first 4 bytes of the 8-byte long,
which on big-endian 64-bit targets are its high half, not the flags,
and as hides truncation.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
@xaionaro
xaionaro changed the base branch from main to qemu-multiarch October 19, 2026 16:02
On i686-unknown-linux-gnu, off_t is signed and has 32 bits, so the
mmap fallback of extract-db rejected every page from 2 GiB on
("does not fit off_t"), and read_mapped_reads_above_4gib failed
there; no check ran the tests for a 32-bit target. copy_mapped_page
now maps through mmap64() with an off64_t offset, which the libc crate
provides on glibc and musl.

The new check test-i686 runs the tests for i686-unknown-linux-gnu on
the host, which needs gcc's 32-bit libraries; CI's lint-and-test job
installs gcc-multilib and runs it next to test.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
test_clear_attrs compared the flags after set_attrs with the initial
ones, so it never checked what clear_attrs left, and a new file on
tmpfs or btrfs has no other flag that clear_attrs could drop. With
FS_NODUMP_FL set, it now clears FS_IMMUTABLE_FL, which no new file has
or inherits, so that clearing it needs no capability, and checks that
FS_NODUMP_FL stays; its last check reads the flags after clear_attrs.

FS_NODUMP_FL is the kept flag as it is the only one that an owner may
set on both ZFS and tmpfs: OpenZFS takes only immutable, append-only,
nodump and projinherit, tmpfs no projinherit, and immutable and
append-only need CAP_LINUX_IMMUTABLE. So the set_attrs and
set_all_attrs tests cannot catch a call that drops the other flags,
which takes a second flag on the file.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
…iomem

The guest tests read the published HII database with read() of
/dev/mem, which fails with EFAULT on arm64, whose Linux maps neither
runtime services data nor reserved memory, and on 32-bit x86 above
the end of Linux's low memory (Linux 6.17: valid_phys_addr_range() in
arch/arm64/mm/mmap.c and arch/x86/mm/mmap.c). uefisettings then falls
back to mmap(), which the guest crate, free of unsafe code, cannot do.
So on those architectures the extract-db test would fail, and on
32-bit x86 also the premise of hii_devmem_guard, which expects EPERM.

publish-hiidb now also puts the CRC-32 of the database, from the
firmware's CalculateCrc32, in the volatile HiiDBCrc32 variable, and
the extract-db test compares the length and CRC-32 of the extracted
file with the published ones. The premise of hii_devmem_guard becomes
that the database is in a System RAM range of /proc/iomem, which
/dev/mem refuses under STRICT_DEVMEM.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
The guest's OVMF and busybox came from the host's apt, so an upgrade
of ovmf or busybox-static on the runner changed the guest with no
change here. apt cannot supply them for guests of other architectures
either: busybox-static of two architectures does not co-install, as it
has no Multi-Arch field, and Ubuntu 26.04 has no ovmf-ia32.

guest-packages-x86_64 now pins the kernel, OVMF and busybox by sha256,
as kernel.sha256 pinned the kernel, and fetch-packages.sh, which
replaces fetch-kernel.sh, downloads and unpacks .rpm and .deb packages
alike. It checks the whole list before any download, takes only HTTPS
URLs and redirects, and creates ROOT only once every package is
unpacked. Each package lists its mirrors with a permanent archive last,
and a mirror that fails or serves other bytes is skipped.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
run.sh read scenarios.tsv with `while read`, which fails on a last
line that lacks a newline, so the loop ended before that line: its
scenario never booted and run.sh still passed, even when the command
line named that scenario. The awk checks before the loop read the
line, so nothing reported it. The loop over the built executables had
the same shape; jq ends every line it prints, but run.sh no longer
relies on that.

Both loops now also take a last line without a newline, as
fetch-packages.sh does. The other reads in tests/qemu and scripts take
here-strings, which end with a newline, use mapfile, which keeps such
a line, or read a single line from /proc or a shebang.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
edk2-stable202508's BaseTools fail to build with gcc 15 and glibc 2.43,
as on Ubuntu 26.04: for C23, the default of gcc 15, glibc's strstr()
returns a const pointer for a const argument, and EfiRom.c:78 assigns
it to a CHAR8 *, which -Werror=discarded-qualifiers rejects. EDK2 fixed
that in 9af06ef3, first released in edk2-stable202602. The newest
release, edk2-stable202608, builds DriverSample on Ubuntu 24.04 and
26.04.

Of DriverSample, only 0d26d944 changes what the tests read: it drops
`default = TRUE` from the check box whose CHECKBOX_DEFAULT flag keeps
it checked, so the HII database is 6 bytes shorter, and every tested
question and default stays.

Cherry-picking 9af06ef3 onto edk2-stable202508 would make the script
fetch and apply a commit, and the pin would stop being a release tag.
EDK2 publishes no DriverSample.efi, and committing one would replace
the pinned source with a binary that cannot be audited.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
@xaionaro
xaionaro changed the base branch from qemu-multiarch to main October 24, 2026 10:03
The QEMU tests booted only x86_64 guests. run.sh now takes the
architecture, x86_64, i686 or aarch64, and finds its QEMU, firmware,
kernel, modules, terminals and EDK2 architecture in one table; make
test-e2e-qemu runs all three. i686 is 32-bit throughout, with Debian
12's 686-pae kernel, OVMF IA32 and an IA32 DriverSample, so
uefisettings meets 32-bit firmware and a 32-bit kernel's /dev/mem.
aarch64 runs under TCG on the same x86_64 host, with AAVMF, Fedora's
aarch64 kernel and a virtio console as the test terminal, which the
new kernel parameter uefisettings.tty= names. init loads the kernel
modules that run.sh copies, efivarfs on i686. run.sh drops the QEMU
and DRIVER_SAMPLE_DIR variables: one value cannot serve three
architectures, and PATH chooses the QEMU. Next to the cross gcc of
ubuntu-packages, which conflicts with gcc-multilib, test-i686 takes
libc6-dev-i386 and lib32gcc-<gcc version>-dev, as README.md now says.
Each boot also leaves bootN.qemu.log, how QEMU saw the guest end: a
poweroff makes a shutdown request, a reset none, and each CPU's last
KVM exit is 8 after a triple fault; run.sh prints it for a failed
boot.

CI runs both jobs on ubuntu-26.04. The QEMU job needs it: QEMU 9.0 and
newer fault unaligned loads from Device memory as arm64 does, so only
there can the aarch64 guests catch a regression of
#12; Ubuntu 24.04 has
QEMU 8.2. The lint job moves with it, for one image and one migration.
As gcc's default CPU varies, build-driver-sample.sh builds the X64
DriverSample for the baseline one, x86-64: the runners' gcc, Ubuntu's
amd64v3 build, defaults to x86-64-v3, whose AVX instructions fault in
the firmware, which runs with CR4.OSXSAVE clear.

The i686 guests boot with pti=on. This is containment: it keeps them
clear of a defect of GitHub's AMD runners and does not fix it. The
runners run KVM nested under Hyper-V on Ubuntu's linux-azure kernel,
whose workaround for Hyper-V's CR3 check
(https://bugs.launchpad.net/bugs/2106673) clears the guest CR3's low
12 bits whenever KVM loads an NPT root for a vCPU without CR4.PCIDE,
as a memory slot's deletion or a register write-back makes it do. In
32-bit PAE paging bits 5-11 address the page directory pointer table,
so a guest whose table does not start a page then walks whatever lies
at the start of the page, and typically triple-faults (CI runs
39767911331 and 39936216534<; diag-5: "confirmed by a kvm-amd.ko
without the mask in PAE paging, CI run N" or the alternative>).
Debian 12's kernel takes these tables from a 32-byte slab; with pti=on
it takes whole pages, which the mask leaves intact: such guests
survived both triggers (CI run 39936216534) and every boot of the
suite (CI run 40037090941).

publish-hiidb puts the database one byte into its pages, so
uefisettings reads the first page from an unaligned address on every
architecture. Two tests check that the guests keep exercising what
they should: read() of /dev/mem serves the database only on x86_64,
so on i686 and aarch64 the extract test passes only through the mmap()
fallback of uefisettings; and on aarch64 a 64-bit load from the
unaligned address dies of SIGBUS, which QEMU before 9.0 does not
raise.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
On the ubuntu-26.04 runners (AMD EPYC 7763, 9V74 and 9V45, KVM nested
under Azure), the i686 guest of the hii scenario triple-faults (KVM exit
8 on vCPU 1, run.sh's qemu.log) during the first test of hii_write,
deterministically, while x86_64 and aarch64 pass; locally (Intel, KVM
and TCG with AMD CPU models, pinned to CPU 1 too, Ubuntu's amd64 and
amd64v3 QEMU builds) it passes. This runs the x86_64 and i686 QEMU
tests, then boots their guests in several ways, mostly as run.sh does (a
hii boot, then the write boot, on one fresh variable store): baselines,
a fresh-store write, a guest that marks each step of a write on its
terminal (a new variable and a Timeout update through efivarfs, then
uefisettings with debug logs for its write path), unpinned and pinned
with the unbound workqueues to CPU 0 or 1, -smp 1, panic=0,
mitigations=off, Ubuntu's plain amd64 QEMU build, the i686 guest under
qemu-system-x86_64, the x86_64 kernel and initrd on the i686 firmware
and EFI drivers (EFI mixed mode), TCG and, on AMD, kvm_amd npt=0. Each
logs QEMU's KVM exits from a verbose guest kernel; the base,
CPU-1-pinned and npt=0 writes also record the host's KVM tracepoints,
entries, interrupt injections and task switches included, with the vCPU
threads named and each one's own tail kept. Everything, with the hashes
and copies of the binaries that the job built, goes to the reset-diag
artifact.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
The first round (CI run 39281579400) showed the i686 guest dying on the
runners with a SHUTDOWN exit while a double fault was being delivered,
on the vCPU that enters next after QEMU changes the flash's memory slot,
and passing with kvm_amd npt=0, under TCG and with a 64-bit kernel on
the same firmware. This round looks for the layer: the guest, the
host's KVM (SVM, NPT and its Hyper-V enlightenments) or the hypervisor
under it.

- Host traces also of KVM's MMU (root zaps, faults, MMIO SPTEs), of
  Hyper-V's flush hypercalls with their address space and return code,
  and, through a kprobe on svm_load_mmu_pgd (Linux v7.0
  arch/x86/kvm/svm/svm.c:4442), of each NPT root that KVM loads for a
  vCPU, which no tracepoint carries: a flush clears what a vCPU uses
  only if it names that root. For -smp 1 too, on the global trace
  clock, as events of different threads are compared.
  ci-diag-attribution.py reads each SHUTDOWN exit from the whole trace,
  matching the zap and flushes by time in whichever thread changed the
  memory map: a root loaded before the last zap points at KVM (L1); a
  successful flush of the dying vCPU's root between that zap and its
  fatal entry points at the hypervisor under KVM (L0); a missing, late,
  failed or wrong-root flush points at KVM (L1); with no root observed,
  without KVM's enlightened NPT flush, with the flush events not traced
  (their enable files are read back), or when a CPU's buffer may have
  lost events after the zap (ftrace's overwrite mode), it stays
  undetermined.
- Forensics: QEMU pauses on the reset instead of ending
  (-action reboot=shutdown,shutdown=pause), and ci-diag-qmp.py saves
  every vCPU's registers and guest RAM at a marker the guest prints
  just before the first write, and guest RAM again after the reset
  (KVM INITs the vCPU on the SHUTDOWN intercept, Linux v7.0
  arch/x86/kvm/svm/svm.c shutdown_interception, so registers survive
  only in the first dump). ci-diag-pagewalk.py walks the page tables
  from the saved CR3 in both dumps, with the permissions and reserved
  bits on the way, and compares (accessed and dirty bits masked) the
  IDT gates of #DF, the contributory faults, #PF and the timer, their
  GDT descriptors, the TSSs (the double fault's task under its own
  CR3), the handlers, the stack pages and the RIPs of the host
  trace's shutdown exits.
- Memory slot changes without firmware: the guest toggles the memory
  decoding of an ivshmem-plain device, so QEMU deletes and adds the
  slot of its BAR, for i686 (-smp 2 and 1) and x86_64; a boot whose BAR
  has no address or whose toggles delete its slot fewer than 200 times
  counts as failed, not passed.
- One write boot per guest kernel parameter (pti=on, idle=poll,
  nohz=off), per CPU feature left out (pge, smep and smap, pse, umip,
  fsgsbase, xsave: the CR4 bits that Linux sets and the firmware does
  not) and with smm=off, each recording the CR4 that the guest set.

A boot that does not produce what it should is a harness failure: a
missing artifact (QEMU's log, console, terminal, host trace and each
vCPU's tail, dumps, reports) gets a "MISSING <path>" line, a failed step (compression, a
report, a host trace without KVM events, a vacuous memory slot test, no
run to boot) a "HARNESS FAIL" line, and the script exits 1. The first round's answered variants are gone.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
The second round (CI run 39767911331) killed i686 guests, and no
x86_64 one, without firmware in two ways: QEMU deleting a memory slot
(an ivshmem-plain BAR whose memory decoding the guest turns off) killed
them within 9 of 200 deletions, and QMP's stop, info registers and cont
killed them at the cont. Both make KVM drop the vCPU's NPT root: a slot
deletion zaps every root (kvm_mmu_zap_all_fast), and the registers that
QEMU writes back after "info registers" carry a PAE vCPU's PDPTRs, so
KVM calls kvm_mmu_reset_context (Linux v7.0 arch/x86/kvm/x86.c:12449-
12477; QEMU v10.2.0 target/i386/kvm/kvm.c:5316-5321, 3616-3620). That a
PAE vCPU dies on its first entries on a fresh root is a hypothesis.
This round tests it, and the two things that pti=on, the one kernel
parameter under which the write test passed, changes.

Each cell boots 10 times, round-robin, with one vCPU and host traces:

- triggers: memslot (200 deletions of the BAR's slot), sync (5 pauses
  with a register read each), sync-plain (5 pauses without one);
- c1: Debian's 6.1 i686 kernel, whose PAE page directory pointer
  tables come from a 32-byte slab and whose CR3 changes on context
  switches;
- c2: c1 with pti=on: whole-page tables, CR3 changes at each kernel
  entry and exit too;
- c3: c1 with two processes ping-ponging over FIFOs: slab tables,
  frequent CR3 changes;
- c4: openSUSE's kernel-pae 7.2.11: whole-page tables (Linux commit
  780f97e30930, v6.16), CR3 changes on context switches;
- cells: c1-c4 with memslot, c1-c4 with sync, c1 with sync-plain, and
  x86_64 memslot and sync as controls.

The host traces gain kprobes on kvm_mmu_reset_context, which the
attribution now takes as an anchor like kvm_mmu_zap_all_fast, and on
shutdown_interception, for the CR3 in the VMCB at the shutdown (which
KVM calls undefined there: a lead only); the attribution assigns both
kprobes' events to a vCPU by their vcpu_id. QEMU logs its vCPU ioctls,
from which the summary counts KVM_SET_SREGS2 per running period.

One deviation from run.sh's QEMU line: the cells and triple faults run
with -machine vmport=off. The VMware backdoor port's handler reads the
registers (QEMU v10.2.0 hw/i386/vmport.c:93-107), so QEMU writes them
back to a guest that probes it: a memslot boot of openSUSE's kernel had
10 KVM_SET_SREGS2 with the port and 2 (QEMU's own, before the start)
without, and so had Fedora's x86_64 kernel; Debian's 6.1 had none after
the start even with the port.

c4's kernel comes from a rolling repository: the trust root is one HTTPS
fetch of the rpm, whose sha256 the diag pins, and the kernel runs only
inside the guest. When pinning, its RSA/SHA512 signature checked out
(rpmkeys -K) against the openSUSE Project Signing Key, fingerprint
AD485664E901B867051AB15F35A2F86E29B700A4: Fedora's
distribution-gpg-keys 1.115 (RPM-GPG-KEY-openSUSE-2022) has that
fingerprint, but its copy expired on 2026-06-19, so the check used the
key as the repository serves it (repodata/repomd.xml.key, renewed to
2030-05-27). If the URL stops serving the rpm, the fetch fails the
harness and the diag leaves out the c4 cells only.

Controls: a triple fault on purpose per architecture, the i686 write
boot (base-1), 10 write boots under pti=on, and tests/qemu/run.sh's
i686 scenarios under pti=on.

Decision rule, declared before the run. A cell "dies" if at least 8 of
its 10 boots die, "survives" if at most 1 dies and the others end their
test, and is "mixed" otherwise; a cell that the deadline cut short is
"incomplete" and decides nothing; a boot that neither dies nor ends its
test fails the harness. A cell is "invalid", and decides nothing, unless
each of its boots shows what its verdict rests on: its kernel's "Linux
version" line; PTI enabled with pti=on and not without; at least 1000
stressor round trips in the first second with the stressor; a death
after the trigger began (a deletion of the BAR's slot, a QMP stop); 0
KVM_SET_SREGS2 in every running period for plain pauses, and one after
a pause with a register read. The write pairs check their PTI line too,
and run.sh's boots their command lines (its guests print no PTI line).

- sync dies, sync-plain survives, x86_64 sync survives: the register
  write-back is the second trigger. sync-plain dies too: the stop alone
  kills, and the write-back is not the mechanism of that trigger.
- Under a trigger that kills c1: c2 and c4 survive while c3 dies, the
  whole-page tables protect; c2 and c3 survive while c4 dies, frequent
  CR3 changes protect; c2 survives while c3 and c4 die, only pti=on
  protects (both factors together, or something else that PTI
  changes); c2 dies, pti=on does not protect against that trigger. c4
  also differs from c1 in its kernel version, so it never decides
  alone.
- pti=on is a containment candidate only if the 10 write boots and
  run.sh's scenarios pass and c2 survives both triggers; even then it
  only contains the failure.

No boot starts after 70 minutes, so that the job uploads its results;
summary.txt lists the boots skipped. ci-diag-pagewalk.py goes, as no
boot dumps RAM any more.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
The third round (CI run 39936216534) points to why the i686 guests die:
the runners' KVM, Ubuntu's linux-azure, clears the guest CR3's low 12
bits whenever it loads an NPT root for a vCPU without CR4.PCIDE
(https://bugs.launchpad.net/bugs/2106673), and in 32-bit PAE paging
bits 5-11 address the page directory pointer table, so a guest whose
table does not start a page walks the wrong one. It left two questions
open:

- c3 (Debian's 6.1 with two processes ping-ponging over FIFOs: tables
  from a 32-byte slab, frequent CR3 changes) died 9 times under
  memslot, but one boot hung for its whole 90 s timeout after exactly 9
  deletions, with no reset, so the rule classed the cell "harness";
- tests/qemu/run.sh's i686 scenarios under pti=on never booted: the
  diag called run.sh outside make, so cargo took the runner's default
  toolchain, which lacks i686-unknown-linux-musl (exit 101), while make
  pins 1.93.1 (Makefile:48-52).

This round runs only those: c3 under memslot, 10 boots with host
traces; run.sh's i686 scenarios 3 times under pti=on, through make
test-e2e-qemu-i686 as CI runs them, with the pti=on wrapper on PATH;
and its aarch64 scenarios as they are, a control off the nested-KVM
path (TCG). The job's own make step before the diag is the i686 run
without pti=on. Every boot that run.sh starts must have a QEMU command
line, logged by the wrapper, that ends in pti=on, as run.sh's quiet
guests print no PTI line; a failed boot's output does not count as a
start.

A boot that its timeout ends after its test started its trigger
(memslot printed its BAR, just before its first toggle; sync's first
QMP stop), without a reset or the end of its test, is now "hung"
instead of "other": it counts towards the cell's boots, but is neither
a death nor a survival.

The kprobe on svm_load_mmu_pgd also records the guest CR3 that the
vCPU runs on (BTF-typed, as the shutdown probe's), so that each guest
page walk after a root load can be set against where that CR3's table
sits in its page. The third round's helpers that nothing calls any
more go: the write pairs, the deliberate triple fault, the openSUSE
kernel's package list and select_arch's x86_64 row.

Decision rule, declared before the run, as the third round's with hung
boots: c3-memslot "dies" if at least 8 of its 10 boots die, "survives"
if at most 1 dies and the others end their test, and is "mixed"
otherwise. If it dies, then under memslot, as under the register
write-back, the whole-page tables of c2 and c4 protect and frequent CR3
changes do not. pti=on is a containment candidate only if all 3 i686
runs pass with every boot provably under pti=on; even then it only
avoids the runners' defect.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
The runners' KVM, Ubuntu's linux-azure 7.0.0-1017-azure, carries the
SAUCE patch "KVM: SVM: Workaround overly strict CR3 check by Hyper-V"
(https://bugs.launchpad.net/bugs/2106673). Under Hyper-V, its
svm_load_mmu_pgd clears bits 0-11 of the guest CR3 that it gives the
hardware whenever CR4.PCIDE is clear. In 32-bit PAE paging bits 5-11
address the page directory pointer table, so a guest whose table does
not start a page walks the wrong one after an NPT root load. The third
round (CI run 39936216534) matched that: c1 (Debian's 6.1, sub-page
tables) died in 10 of 10 boots under memslot and under sync.

This round tests it as the cause on the failing path itself. It builds
kvm-amd.ko from the runners' exact source (linux-azure 7.0.0-1017.17
from the Ubuntu archive, pinned to the sha256 of the signed indexes)
twice: as shipped ("stock", the control) and without the mask in PAE
paging ("poc"):

-		if (!kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE))
+		if (!is_pae_paging(vcpu) && !kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE))

is_pae_paging() is !is_long_mode && is_pae && is_paging
(arch/x86/kvm/x86.h:301-304), and KVM's own load_pdptrs reads a PAE
vCPU's table at CR3 & GENMASK(11, 5). The mask stays for the case
behind the patch, an SMM entry from long mode with paging off and PCID
bits in CR3 (its RFC:
https://patchwork.kernel.org/project/kvm/patch/20240319163456.133942-1-vkuznets@redhat.com/),
which is_long_mode() alone would no longer cover.

With the make variables of Ubuntu's own build (kmake in
debian/rules.d/0-common-vars.mk), the stock module's code equals the
shipped one's: 0 of 856 symbols with code differ in their
instructions, resolved targets and relocations, once addresses,
encodings and section-relative relocation addends are left out (data
sections are not compared). The same comparison catches a shipped copy
with the mask's instruction NOPed, and one with a call retargeted. The
PoC differs in svm_load_mmu_pgd alone. ci-diag-kvm-amd.sh checks the
latter and reports the former.

The diag then records the runner's lockdown and module.sig_enforce,
and stops if either could refuse an unsigned module. Otherwise it
loads each build in place of kvm_amd, checking its srcversion and
parameters, and that /dev/kvm, which the unload removed, is open to
the runner user again once udev has applied the workflow's rule. It
boots c1 under memslot and sync DIAG_REPS times on each build, and
finally loads the shipped module again.

If the cause is right, stock dies and poc survives; poc dying, or
mixed, would falsify the mask as the cause. A failed VMRUN (KVM exit
9, the new verdict "refused") on poc, with stock not refused, would
point to Hyper-V rejecting CR3 bits 5-11 of a PAE guest. A cell boots
no more after its first refused boot, which decides it: each such boot
waits out its 90-s timeout, and ten per cell would take half the
run's time.

The cells trace KVM's events without the kprobes, in a new mode
(events), as the rebuilt module has no BTF for them. Each boot's
summary gives the page offsets of the first faulting guest page walk
after each root zap: under the mask a PAE vCPU's walk reads its PDPT
at the start of the table's page (PDPTE[3] at offset 0x018); without
it, where the guest put the table. run_suite goes, as no round uses
it.

Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
@xaionaro
xaionaro force-pushed the qemu-multiarch-reset-diag branch from 3c8c324 to bef5265 Compare October 24, 2026 10:36
@xaionaro xaionaro closed this Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant