[H3 CI] compare frozen hardware runs and retain serving evidence / 保留跨硬件服务证据

[edwingao28]

Description

Run the same frozen H3 serving workload on prepared H100/H200/B200 sites, record participating GPUs separately from allocated capacity, and retain latency, completion failures, validated media, and board power/energy with their measurement windows. The existing H200 defaults and manual CI authorization remain intact. This PR is stacked on #2894 and remains draft.

• All nine formal NVIDIA C1/C2/C4 cells completed 20/20 scheduled requests: H100 C1/C2, H100 C4, H200, and B200. The workload fixes 20 prompt/seed pairs, model revision, and eight-second video/audio specifications. These measurements do not establish sustained serving capacity or hardware winners. Retained startup logs report BF16 DiT parameters on all three NVIDIA sites; they do not establish every kernel or activation dtype. H100/H200 select FlashAttention while B200 selects dynamic cuDNN SDPA, so these are hardware-plus-runtime configurations, not isolated chip comparisons.
• CPU reprocessing verifies original CI identities and artifact checksums, then compares every C1 video frame and audio sample: H200/B200 and H200/H100. Each has 20 valid pairs and raw example-threshold results of 2 pass / 18 fail. Thresholds are explicitly uncalibrated; successful export is not a quality or release verdict. The 47.8 MB / 47.6 MB portable artifacts each retain 40 paired videos, source metadata, and original artifact links without duplicating whole source archives. All 54 files and 81 report/media links per artifact were verified; every paired metric is unchanged from the original exports.
• Opt-in request-correlated server timing records accepted, dispatched, forward-finished, and media-ready boundaries using the same monotonic clock. Historical results stay unavailable for these fields. The pinned runtime patch and sidecar identity are retained; real GPU timing and instrumentation overhead remain unverified.
• AMD recovery reuses the interrupted task-owned rootfs and verified model cache. The failed original extraction, bounded CPU permission repair, and login-node namespace denial remain retained. No valid MI355X generation has been established. The single-lease campaign is queued at dba9686a1 with no native runner, executed GPU steps, or allocation yet. Its budget begins after allocation: 120 minutes, a 15-minute readiness deadline, and 10 minutes reserved beyond the serving supervisor. It must pass compute-node entry and one full video/audio warmup before exactly 20 scheduled requests; failures must not be topped up. The workload is structurally checked against the formal H200 source before allocation. AMD records its TP1/Ulysses4/AITER configuration separately from NVIDIA TP2/Ulysses2.

Validation: the integrated local CPU suite passed 631 tests, with two Linux-only skips; actionlint and diff checks passed. The final single-lease change passed 80 focused local tests and current-head Linux CI passed 646 tests. Real AMD execution remains pending. The unchanged AgentX signal-lifecycle check failed once (210/211 passed); its second CPU attempt passed 211/211. The original failure is retained; no AgentX code was changed. The matching frontend #1100 is merged and deployed with participating-GPU efficiency, separate allocation accounting, measured watts, energy/valid clip, and recorded enforced-cap ratios. The fidelity/timing viewer #1101 is merged at 9c4d9cde and deployed. Final-head frontend CI passed 5,998 unit tests, 430 component tests and 1,906 integration executions across eight Chrome/Firefox shards. Both compact artifacts passed production EN/ZH desktop/mobile playback, audio, paired selection, report/raw downloads and original-source links. Collapsed reports now defer media loading; paired-page initial distinct MP4 requests fell from 40 to 2 in the controlled real-media check. Existing approved media storage is reused. Backend merge requirements remain unchanged.

Related Issue

Builds on #2894; no issue automatically closed.

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe)

Checklist

• I have tested my changes locally
• I have updated documentation if necessary
• For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of perf-changelog.yaml and have not edited historical entries
• Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /reuse-sweep-run on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.

中文说明

本 PR 基于 #2894，保持草稿。H100/H200/B200 复用同一冻结工作负载，九个正式 C1/C2/C4 测量点均完成 20/20 条计划请求，并保留参与 GPU、分配容量、延迟、失败记录、完整音视频和功率/能耗窗口。当前结果不代表持续服务容量或硬件优胜。原始启动日志显示三个 NVIDIA 配置的 DiT 参数均为 BF16，但不能推定所有内核和激活的精度；H100/H200 选择 FlashAttention，B200 选择 dynamic cuDNN SDPA，需按硬件与软件配置共同解读。

仅复用原始视频完成两组各 20 对逐帧/逐音频采样检查，校验 CI 身份与产物哈希。两组示例阈值均为 2 通过、18 未通过；阈值未校准，不能据此宣称画质回归或发布合格。便携报告保留全部配对视频和精确来源，避免重复打包完整运行目录。

新增可选服务端阶段计时，保留运行时补丁和旁路日志身份；历史字段维持不可用，真实 GPU 计时与插桩开销仍待验证。AMD 复用已提取 rootfs 和模型缓存，保留提取、CPU 修复超时及登录节点 namespace 拒绝证据；尚无通过验收的 MI355X 视频。当前唯一一轮 MI355X CI 仍等待原有 runner 调度，尚无 GPU 分配或已执行 GPU 步骤。分配后总时限为 120 分钟、启动截止 15 分钟、服务监督程序外预留 10 分钟。必须先通过计算节点检查与完整音视频预热，再执行恰好 20 个计划请求，不能补跑到 20 个成功。分配前核对正式 H200 工作负载内容，单独记录 AMD TP1/Ulysses4/AITER 与 NVIDIA TP2/Ulysses2 的拓扑和运行时差异。

当前整合版本本地 631 项 CPU 测试通过、两项 Linux 专属跳过，工作流语法通过；最终单次分配变更的 80 项本地测试及当前提交 Linux CI 的 646 项测试通过。AMD 实测待完成。未改动的 AgentX 信号清理测试首次出现一次失败（210/211 通过），保留原日志后仅复查该 CPU 任务，第二次 211/211 通过，未修改 AgentX 代码。前端 #1100 已合并部署参与卡效率、分配成本口径、功率、每条有效视频能耗和记录的实际功率上限占比。#1101 已以 9c4d9cde 合并并部署配对 fidelity、可选服务端计时和报告按需加载；最终 CI 通过 5,998 项单元测试、430 项组件测试和八组 Chrome/Firefox 共 1,906 次集成测试。两份真实产物的线上中英文、桌面/移动端播放、音频、配对选择、报告和原始下载均通过；复用已批准的媒体存储。后端合并授权和验收门槛不变。

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档