test: drive the live guest agent under concurrent load

test-suites/cases/01-guest-os/16-concurrency-and-robustness/metadata.json

@@ -0,0 +1,4 @@
+{
+  "id": "section-guest-os-concurrency-and-robustness",
+  "title": "Concurrency and robustness"
+}

test-suites/cases/01-guest-os/16-concurrency-and-robustness/tc-gos-concurrency-001/case.md

@@ -0,0 +1,81 @@
+<!-- SPDX-FileCopyrightText: © 2026 Phala Network <dstack@phala.network> -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+<a id="tc-gos-concurrency-001"></a>
+# TC-GOS-CONCURRENCY-001: Concurrent derivation determinism and per-call quote binding
+
+## Metadata
+
+- Priority: P0
+- Type: Functional, Security, Robustness, Regression
+- Minimum environment: SIMULATOR
+- Automation: Yes
+- Requirements: [req-gos-concurrency-001](../../../../catalog/feature-audit.md#req-gos-concurrency-001)
+- Risks: [risk-gos-concurrency-001](../../../../catalog/feature-audit.md#risk-gos-concurrency-001)
+- Source: `dstack/guest-agent/src/rpc_service.rs`, `dstack/guest-agent/src/rpc_service_v1.rs`, `dstack/dstack-attest/src/attestation.rs`
+
+## Prepared execution knowledge
+
+- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1.
+- The harness is [`shared/automation/guest-agent-robustness-case.py`](../../../../shared/automation/guest-agent-robustness-case.py).
+- Keys are compared by SHA-256 digest; only digests reach the artifacts.
+
+## Objective
+
+Verify that concurrent identical derivations return identical keys and that
+concurrent quotes each bind their own `report_data`. The per-method cases only
+send one request at a time.
+
+## Preconditions
+
+1. The lease-owned agent's internal (`DstackGuest`) and legacy (`Tappd`)
+   listeners are reachable.
+
+## Test Data
+
+- 24 concurrent calls per method (three times `workers = 8`), each on its own
+  connection.
+- A second derivation path and `DeriveKey` with `random_seed = true`, as
+  controls that the comparison can tell keys apart.
+- 24 distinct 64-byte `report_data` values.
+
+## Steps
+
+<a id="tc-gos-concurrency-001-step-01"></a>
+### Step 1: Confirm the listeners before any load
+
+Call `Version` on both listeners.
+
+**Expected results:**
+
+- Both listeners answer.
+
+<a id="tc-gos-concurrency-001-step-02"></a>
+### Step 2: Derive the same key from 24 connections at once
+
+Send 24 concurrent identical `DstackGuest.GetKey`, `Tappd.DeriveK256Key` and
+`Tappd.DeriveKey` (`random_seed = false`) requests, then the two controls.
+
+**Expected results:**
+
+- Every call returns HTTP 200.
+- Each method's 24 responses carry one key digest. `GetKey` also carries one
+  signature-chain digest. `DeriveKey` certificate chains are freshly issued and
+  are not compared.
+- A different path returns a different key, and 24 `random_seed = true`
+  derivations return 24 distinct keys.
+
+<a id="tc-gos-concurrency-001-step-03"></a>
+### Step 3: Quote 24 distinct report-data values at once
+
+Send 24 concurrent `DstackGuest.GetQuote` requests, each with its own
+`report_data`.
+
+**Expected results:**
+
+- Every call returns HTTP 200.
+- Each response echoes its own `report_data` and carries it at quote bytes
+  568..632 (`TDX_QUOTE_REPORT_DATA_RANGE`).
+
+## Postconditions
+
+Nothing is created outside the lease. Keep the per-call digest table.

test-suites/cases/01-guest-os/16-concurrency-and-robustness/tc-gos-concurrency-001/metadata.json

@@ -0,0 +1,36 @@
+{
+  "id": "tc-gos-concurrency-001",
+  "title": "Concurrent derivation determinism and per-call quote binding",
+  "priority": "P0",
+  "requirements": [
+    "req-gos-concurrency-001"
+  ],
+  "risks": [
+    "risk-gos-concurrency-001"
+  ],
+  "tags": [
+    "guest",
+    "concurrency-and-robustness"
+  ],
+  "fixture": {
+    "profile": "no-tee-dev",
+    "versions": {
+      "vmm": "candidate",
+      "guest": "candidate",
+      "kms": "candidate",
+      "gateway": "candidate",
+      "verifier": "candidate"
+    },
+    "destructive_scope": "lease-only",
+    "hardware_required": false,
+    "simulation_allowed": true
+  },
+  "actions_under_test": [
+    "DstackGuest.GetKey"
+  ],
+  "execution": {
+    "entrypoint": "shared/automation/guest-agent-robustness-case.py",
+    "args": [],
+    "timeout_seconds": 300
+  }
+}

test-suites/cases/01-guest-os/16-concurrency-and-robustness/tc-gos-concurrency-002/case.md

@@ -0,0 +1,82 @@
+<!-- SPDX-FileCopyrightText: © 2026 Phala Network <dstack@phala.network> -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+<a id="tc-gos-concurrency-002"></a>
+# TC-GOS-CONCURRENCY-002: Head-of-line blocking on the quote path
+
+## Metadata
+
+- Priority: P0
+- Type: Robustness, Performance, Availability, Security
+- Minimum environment: HARDWARE
+- Automation: Yes
+- Requirements: [req-gos-concurrency-002](../../../../catalog/feature-audit.md#req-gos-concurrency-002)
+- Risks: [risk-gos-concurrency-002](../../../../catalog/feature-audit.md#risk-gos-concurrency-002)
+- Source: `dstack/tdx-attest/src/linux.rs`, `dstack/guest-agent/src/rpc_service.rs`, `dstack/guest-agent/src/server.rs`
+
+## Prepared execution knowledge
+
+- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1.
+- The harness is [`shared/automation/guest-agent-robustness-case.py`](../../../../shared/automation/guest-agent-robustness-case.py).
+- Every call uses its own connection.
+- The simulator answers a quote in under a millisecond, so nothing blocks. The
+  harness reports **BLOCKED**, never PASS, when one quote costs less than 50 ms.
+
+## Objective
+
+Verify that a trivial method stays answerable while concurrent quotes saturate
+the agent, and that the agent is not restarted by its own watchdog.
+`tdx_attest::get_quote` holds a global mutex while it blocks on the host QGS.
+The watchdog heartbeat is a `Worker.Version` call to the agent itself, so a
+runtime parked on quotes gets the agent SIGABRTed (fixed in #1256).
+
+## Preconditions
+
+1. A lease-owned guest on physical Intel TDX hardware with a reachable host
+   QGS, and no other load on the agent.
+
+## Test Data
+
+- 20 unloaded `Version` samples and 3 single `GetQuote` calls, as baselines.
+- 24 threads issuing `GetQuote` with random `report_data` for 20 s, while
+  `Version` is sampled every 100 ms.
+- Bound: loaded `Version` p95 at most **1.0 s**.
+- The in-guest agent's (PID, start time) before and after, read over the
+  fixture's `ssh_argv` when it publishes one.
+
+## Steps
+
+<a id="tc-gos-concurrency-002-step-01"></a>
+### Step 1: Confirm the listener before the load
+
+Call `Version` on the `DstackGuest` listener.
+
+**Expected results:**
+
+- The listener answers.
+
+<a id="tc-gos-concurrency-002-step-02"></a>
+### Step 2: Saturate the quote path and poll a trivial method
+
+Measure the baselines, then hold 24 concurrent `GetQuote` calls for 20 s while
+sampling `Version`.
+
+**Expected results:**
+
+- One `GetQuote` costs at least 50 ms; otherwise the result is BLOCKED.
+- The agent's (PID, start time) is unchanged.
+- Every `GetQuote` and every `Version` probe returns HTTP 200.
+- Loaded `Version` p95 is at most 1.0 s.
+
+<a id="tc-gos-concurrency-002-step-03"></a>
+### Step 3: Confirm the listener after the load
+
+Call `Version` again.
+
+**Expected results:**
+
+- The listener answers.
+
+## Postconditions
+
+Nothing is created. Keep the baseline and loaded latency distributions and the
+measured quote cost.

test-suites/cases/01-guest-os/16-concurrency-and-robustness/tc-gos-concurrency-002/metadata.json

@@ -0,0 +1,36 @@
+{
+  "id": "tc-gos-concurrency-002",
+  "title": "Head-of-line blocking on the quote path",
+  "priority": "P0",
+  "requirements": [
+    "req-gos-concurrency-002"
+  ],
+  "risks": [
+    "risk-gos-concurrency-002"
+  ],
+  "tags": [
+    "guest",
+    "concurrency-and-robustness"
+  ],
+  "fixture": {
+    "profile": "guest-readonly",
+    "versions": {
+      "vmm": "candidate",
+      "guest": "candidate",
+      "kms": "candidate",
+      "gateway": "candidate",
+      "verifier": "candidate"
+    },
+    "destructive_scope": "lease-only",
+    "hardware_required": true,
+    "simulation_allowed": false
+  },
+  "actions_under_test": [
+    "DstackGuest.GetQuote"
+  ],
+  "execution": {
+    "entrypoint": "shared/automation/guest-agent-robustness-case.py",
+    "args": [],
+    "timeout_seconds": 300
+  }
+}

test-suites/cases/01-guest-os/16-concurrency-and-robustness/tc-gos-concurrency-003/case.md

@@ -0,0 +1,83 @@
+<!-- SPDX-FileCopyrightText: © 2026 Phala Network <dstack@phala.network> -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+<a id="tc-gos-concurrency-003"></a>
+# TC-GOS-CONCURRENCY-003: Agent survival and process identity after concurrent load
+
+## Metadata
+
+- Priority: P0
+- Type: Robustness, Availability, Regression
+- Minimum environment: SIMULATOR
+- Automation: Yes
+- Requirements: [req-gos-concurrency-003](../../../../catalog/feature-audit.md#req-gos-concurrency-003)
+- Risks: [risk-gos-concurrency-003](../../../../catalog/feature-audit.md#risk-gos-concurrency-003)
+- Source: `dstack/guest-agent/src/server.rs`, `dstack/guest-agent/src/rpc_service.rs`, `dstack/guest-agent/src/guest_api_service.rs`
+
+## Prepared execution knowledge
+
+- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1.
+- The harness is [`shared/automation/guest-agent-robustness-case.py`](../../../../shared/automation/guest-agent-robustness-case.py).
+- Process identity is (PID, start time from `/proc/<pid>/stat`). Release
+  builds use `panic = "abort"` and the agent is supervised, so a socket that
+  answers after the load may belong to a restarted process.
+
+## Objective
+
+Verify that the agent answers every concurrent request on all four listeners
+and is still the same process afterwards.
+
+## Preconditions
+
+1. The internal `DstackGuest`, legacy `Tappd`, external `Worker` and
+   `GuestApi` listeners are reachable.
+2. The case manifest publishes the agent's PID and `/proc/<pid>` is readable;
+   otherwise the case is BLOCKED.
+
+## Test Data
+
+Eleven methods, six times each, all in flight at once (66 calls):
+
+| listener | methods |
+|---|---|
+| `DstackGuest` | `GetKey`, `GetQuote`, `Info`, `Version`, `GetTlsKey` |
+| `Tappd` | `TdxQuote`, `Info` |
+| `Worker` | `Info`, `Version` |
+| `GuestApi` | `Info`, `SysInfo` |
+
+## Steps
+
+<a id="tc-gos-concurrency-003-step-01"></a>
+### Step 1: Confirm all four listeners before the load
+
+Call a trivial method on each listener.
+
+**Expected results:**
+
+- All four answer.
+
+<a id="tc-gos-concurrency-003-step-02"></a>
+### Step 2: Drive 66 concurrent calls across the four listeners
+
+Record the agent's identity and counters, issue the whole table concurrently,
+then send one `GetKey` and re-read the counters.
+
+**Expected results:**
+
+- Every call returns HTTP 200, and so does the `GetKey` probe.
+- The agent's (PID, start time) is unchanged.
+- The agent holds at most `workers` more open descriptors than before, when
+  `/proc/<pid>/fd` is readable.
+
+<a id="tc-gos-concurrency-003-step-03"></a>
+### Step 3: Confirm all four listeners after the load
+
+Call a trivial method on each listener again.
+
+**Expected results:**
+
+- All four answer.
+
+## Postconditions
+
+Nothing is created outside the lease. Keep the before/after counters and the
+identity pair.

test-suites/cases/01-guest-os/16-concurrency-and-robustness/tc-gos-concurrency-003/metadata.json

@@ -0,0 +1,36 @@
+{
+  "id": "tc-gos-concurrency-003",
+  "title": "Agent survival and process identity after concurrent load",
+  "priority": "P0",
+  "requirements": [
+    "req-gos-concurrency-003"
+  ],
+  "risks": [
+    "risk-gos-concurrency-003"
+  ],
+  "tags": [
+    "guest",
+    "concurrency-and-robustness"
+  ],
+  "fixture": {
+    "profile": "no-tee-dev",
+    "versions": {
+      "vmm": "candidate",
+      "guest": "candidate",
+      "kms": "candidate",
+      "gateway": "candidate",
+      "verifier": "candidate"
+    },
+    "destructive_scope": "lease-only",
+    "hardware_required": false,
+    "simulation_allowed": true
+  },
+  "actions_under_test": [
+    "DstackGuest.Version"
+  ],
+  "execution": {
+    "entrypoint": "shared/automation/guest-agent-robustness-case.py",
+    "args": [],
+    "timeout_seconds": 300
+  }
+}