Take machine offline and notify when dpkg is corrupted during prereq install

[DrewScoggins]

Generated by Copilot

Summary

Some prerequisite-install failures are not transient but indicate a broken host. The canonical case is the apt/dpkg database getting stuck in an interrupted state:

E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

Today this just fails the run repeatedly on the same bad machine. This PR detects that specific state during prerequisite installation and:

• Takes the machine out of Helix rotation by creating the offline sentinel file in $HELIX_CONFIG_ROOT (Helix stops sending work to a machine with that file).
• Notifies the team by writing a semaphore blob named after the machine to a new offline-machines container in pvscmdupload.

The semaphore blob is cleared only when the machine next uploads results successfully, so a machine has to actually be working to have its semaphore removed.

The notification write is best effort: it first uses the already-available azure libraries, then falls back to bootstrapping a throwaway venv with the dependencies, and if both fail the machine is still taken offline and a message is printed to the console.

Changes

• scripts/machine_health.py (new): dpkg-state detection, offline sentinel creation, semaphore write (with venv fallback), and semaphore clear.
• scripts/performance/constants.py: add OFFLINE_MACHINES_CONTAINER.
• scripts/run_performance_job.py: run the health check on prerequisite failure.
• scripts/benchmarks_ci.py: clear the semaphore after a successful results upload.

Detection

Detection mirrors exactly what triggers the failure: a read-only sudo apt-get check is probed for the dpkg was interrupted / dpkg --configure -a signatures, so only this non-transient condition takes a machine offline (flaky network failures do not).

Notes

• A companion change in dotnet-performance-infra (PerfMonitor) consumes the offline-machines container and alerts the team.
• The upload identity needs write/create permission on the new offline-machines container; PerfMonitor's identity needs read.

[Copilot]

Pull request overview

This PR adds a machine-health hook to detect a specific non-transient prereq install failure mode (corrupted/interrupted apt/dpkg state), then automatically removes the affected Helix machine from rotation and emits an external “offline machine” signal via Azure Storage until the machine proves healthy again by successfully uploading results.

Changes:

• Add scripts/machine_health.py to detect interrupted dpkg state, create Helix offline sentinel, and write/clear an “offline semaphore” blob.
• Add OFFLINE_MACHINES_CONTAINER constant for the new Azure Storage container.
• Invoke the health check on prereq-install failure and clear the semaphore after successful results upload.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
scripts/run_performance_job.py	Runs machine health check after prereq install failure (without affecting exit code).
scripts/performance/constants.py	Adds OFFLINE_MACHINES_CONTAINER constant for offline-machine semaphores.
scripts/machine_health.py	Implements dpkg corruption detection, Helix offline sentinel creation, and Azure blob semaphore write/clear.
scripts/benchmarks_ci.py	Clears the offline semaphore after a successful results upload.

Comments suppressed due to low confidence (1)

scripts/machine_health.py:132

• The fallback sentinel creation path uses sudo tee without -n, which can block on an interactive sudo password prompt (up to the 30s timeout) and delay/offline-marking in the exact failure scenario this script is meant to handle. Adding -n makes the operation non-interactive and fail fast.

    try:
        subprocess.run(
            ["sudo", "tee", offline_path],
            input=content,
            text=True,
            stdout=subprocess.DEVNULL,
            check=True,
            timeout=30,
        )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[LoopedBard3]

This would be good to fix.

[LoopedBard3]

We should fix the typos in the names, and to double check, deleting the helix offline file will be a manual thing we do?

[DrewScoggins]

We should fix the typos in the names, and to double check, deleting the helix offline file will be a manual thing we do?

Yes, deleting the file on the machine itself will be something we do manually. Deleting the file that is in the storage account is automatic.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

scripts/run_performance_job.py:334

• This prereq-install block has multiple typos ("prerequistes" / "prerequites") and the variable combined_prequisites is misspelled. These reduce log searchability and make this branch inconsistent with the Windows block’s combined_prerequisites.

            combined_prequisites = " && ".join(install_prerequisites)
            helix_pre_commands += [
                'echo "** Installing prerequistes **"',
                f"{combined_prequisites} || export PERF_PREREQS_INSTALL_FAILED=1",
                'test "x$PERF_PREREQS_INSTALL_FAILED" = "x1" && echo "** Error: Failed to install prerequites **"',

scripts/machine_health.py:125

• The sudo fallback for writing the Helix offline sentinel runs sudo tee without -n. If sudo ever requires a password, this will block until the timeout and still fail to create the sentinel. Using sudo -n matches the non-interactive pattern used elsewhere in this script and fails fast.

            ["sudo", "tee", offline_path],

[DrewScoggins]

Addressed the review feedback in the latest push:

• Typos: fixed the prerequistes/prerequites log strings and renamed combined_prequisites → combined_prerequisites in run_performance_job.py so the non-Windows branch matches the Windows branch and logs are searchable.
• Non-interactive sudo: both sudo probes in machine_health.py now use sudo -n (the apt-get check detection and the tee offline-sentinel fallback) so they fail fast instead of blocking on a password prompt.
• Malformed try/except in is_dpkg_broken(): the earlier "Use non-interactive sudo" autofix commit had removed the try: line while leaving the except clauses, which made the module fail to import (this is what the reviewer flagged). I restored the try: so the module parses and the detection works; verified with python -m py_compile.

Re: the offline sentinel — confirmed the local $HELIX_CONFIG_ROOT/offline file is removed manually when the machine is fixed; the storage-account semaphore blob is cleared automatically on the next successful results upload.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

scripts/machine_health.py:260

• The reason string and the subsequent semaphore-failure message claim the machine "has been taken out of Helix rotation" / "has still been taken offline", but create_offline_sentinel(reason) can return False (e.g., HELIX_CONFIG_ROOT unset, permission/sudo failure). This can produce misleading sentinel/blob content and console output when the offline sentinel was not actually created. Capture and check the return value and only claim the machine was taken offline when the sentinel creation succeeds; otherwise log an explicit error.

    reason = (
        f"apt/dpkg is in a corrupted (interrupted) state on {machine_name}. Prerequisite "
        f"installation cannot proceed. The machine has been taken out of Helix rotation. Recover it "
        f"by running 'sudo dpkg --configure -a' and removing $HELIX_CONFIG_ROOT/offline."
    )

scripts/machine_health.py:76

• This PR introduces a new critical operational safety path (dpkg corruption detection + taking the machine offline + Azure semaphore writes), but there are no unit tests covering the decision logic (e.g., signature detection in is_dpkg_broken, handling of apt-get missing, and the check_and_mark_offline flow). The repo already has pytest-based tests under scripts/tests/, so adding targeted tests with mocks for subprocess.run, os.getenv, and the Azure client helpers would help prevent regressions.

def is_dpkg_broken() -> bool:
    '''Return True if apt/dpkg is stuck in the interrupted state that requires 'dpkg --configure -a'.

    Detection mirrors exactly what triggers the failure: any apt-get invocation on such a machine
    prints the "dpkg was interrupted" error. ``apt-get check`` is a read-only diagnostic, so it is a