Repository navigation
Conversation
On the ubuntu-26.04 runners (AMD EPYC 7763, 9V74 and 9V45, KVM nested under Azure), the i686 guest of the hii scenario triple-faults (KVM exit 8 on vCPU 1, run.sh's qemu.log) during the first test of hii_write, deterministically, while x86_64 and aarch64 pass; locally (Intel, KVM and TCG with AMD CPU models, pinned to CPU 1 too, Ubuntu's amd64 and amd64v3 QEMU builds) it passes. This runs the x86_64 and i686 QEMU tests, then boots their guests in several ways, mostly as run.sh does (a hii boot, then the write boot, on one fresh variable store): baselines, a fresh-store write, a guest that marks each step of a write on its terminal (a new variable and a Timeout update through efivarfs, then uefisettings with debug logs for its write path), unpinned and pinned with the unbound workqueues to CPU 0 or 1, -smp 1, panic=0, mitigations=off, Ubuntu's plain amd64 QEMU build, the i686 guest under qemu-system-x86_64, the x86_64 kernel and initrd on the i686 firmware and EFI drivers (EFI mixed mode), TCG and, on AMD, kvm_amd npt=0. Each logs QEMU's KVM exits from a verbose guest kernel; the base, CPU-1-pinned and npt=0 writes also record the host's KVM tracepoints, entries, interrupt injections and task switches included, with the vCPU threads named and each one's own tail kept. Everything, with the hashes and copies of the binaries that the job built, goes to the reset-diag artifact. Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
The first round (CI run 39281579400) showed the i686 guest dying on the runners with a SHUTDOWN exit while a double fault was being delivered, on the vCPU that enters next after QEMU changes the flash's memory slot, and passing with kvm_amd npt=0, under TCG and with a 64-bit kernel on the same firmware. This round looks for the layer: the guest, the host's KVM (SVM, NPT and its Hyper-V enlightenments) or the hypervisor under it. - Host traces also of KVM's MMU (root zaps, faults, MMIO SPTEs), of Hyper-V's flush hypercalls with their address space and return code, and, through a kprobe on svm_load_mmu_pgd (Linux v7.0 arch/x86/kvm/svm/svm.c:4442), of each NPT root that KVM loads for a vCPU, which no tracepoint carries: a flush clears what a vCPU uses only if it names that root. For -smp 1 too, on the global trace clock, as events of different threads are compared. ci-diag-attribution.py reads each SHUTDOWN exit from the whole trace, matching the zap and flushes by time in whichever thread changed the memory map: a root loaded before the last zap points at KVM (L1); a successful flush of the dying vCPU's root between that zap and its fatal entry points at the hypervisor under KVM (L0); a missing, late, failed or wrong-root flush points at KVM (L1); with no root observed, without KVM's enlightened NPT flush, with the flush events not traced (their enable files are read back), or when a CPU's buffer may have lost events after the zap (ftrace's overwrite mode), it stays undetermined. - Forensics: QEMU pauses on the reset instead of ending (-action reboot=shutdown,shutdown=pause), and ci-diag-qmp.py saves every vCPU's registers and guest RAM at a marker the guest prints just before the first write, and guest RAM again after the reset (KVM INITs the vCPU on the SHUTDOWN intercept, Linux v7.0 arch/x86/kvm/svm/svm.c shutdown_interception, so registers survive only in the first dump). ci-diag-pagewalk.py walks the page tables from the saved CR3 in both dumps, with the permissions and reserved bits on the way, and compares (accessed and dirty bits masked) the IDT gates of #DF, the contributory faults, #PF and the timer, their GDT descriptors, the TSSs (the double fault's task under its own CR3), the handlers, the stack pages and the RIPs of the host trace's shutdown exits. - Memory slot changes without firmware: the guest toggles the memory decoding of an ivshmem-plain device, so QEMU deletes and adds the slot of its BAR, for i686 (-smp 2 and 1) and x86_64; a boot whose BAR has no address or whose toggles delete its slot fewer than 200 times counts as failed, not passed. - One write boot per guest kernel parameter (pti=on, idle=poll, nohz=off), per CPU feature left out (pge, smep and smap, pse, umip, fsgsbase, xsave: the CR4 bits that Linux sets and the firmware does not) and with smm=off, each recording the CR4 that the guest set. A boot that does not produce what it should is a harness failure: a missing artifact (QEMU's log, console, terminal, host trace and each vCPU's tail, dumps, reports) gets a "MISSING <path>" line, a failed step (compression, a report, a host trace without KVM events, a vacuous memory slot test, no run to boot) a "HARNESS FAIL" line, and the script exits 1. The first round's answered variants are gone. Signed-off-by: Dmitrii Okunev <xaionaro@meta.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diagnostics only, not for merge. Runs the i686 (and x86_64) QEMU tests, then tests/qemu/ci-diag-reset.sh, which records host KVM traces around the i686 guest's triple fault on the AMD runners and uploads them as the reset-diag artifact. Built on #24 (b9603e6). Will be closed after the run.