feat: EP registration + monitoring — subprocess isolation, structured failures, universal op-tracing dispatch, lazy CLI startup

[tezheng]

Summary

Six capability areas landed across 3 commits. Op-tracing is one; the others are load-bearing infrastructure the CLI relies on.

1. EP registration hardening (commands/sys.py)

isolated_ep_register runs each filesystem-backed EP registration in a fresh Python subprocess so Windows' process-wide, base-name-keyed loaded-modules table can't leak the first-loaded plugin_impl.dll's metadata into every subsequent call. ThreadPoolExecutor(max_workers=4) parallelizes the fleet — 14 s → ~9 s on a 7-EP box. Nested _worker() function shipped via inspect.getsource to python -c; subprocess.run(timeout=...) cleanly terminates hung children.

2. Structured EP failure attribution (session/ep_device.py)

WinMLEPRegistrationFailed upgraded from an opaque message wrapper to a structured exception carrying .code, .reason, .dll_path, .fallback_version, all parsed at construction. Reason table covers Win32 error codes 2, 5, 126, 127, 193, 1114; unknown codes get a generic fallback. fallback_version reads the DLL's PE VS_VERSIONINFO via pywin32 so [failed] rows still surface a truthful version even when ORT never loaded the binary.

3. EP registry API surface (session/ep_registry.py)

WinMLEP.arg0 (identifier for unregister_execution_provider_library), WinMLEP.to_dict() serializing plugin_version + per-device device_facts so Architecture/Driver rows survive the subprocess boundary, WinMLEPRegistry.unregister_ep() for the register → snapshot → unregister cadence.

4. Op-tracing monitor extension (session/monitor/openvino_monitor.py NEW + commands/perf.py)

Second op-tracing monitor implementation. Universal _resolve_ep_monitor dispatch: auto-selects by device (NPU/auto → try monitor A, fall back to B; GPU → B), explicit --ep honored, --op-tracing detail rejected when the resolved EP has no detail surface. Enabled via ORT env var + load_config provider option; per-inference CSV flush (not per-session-destroy); env var save/restore in __enter__/__exit__ so we never leak monitor state.

5. CLI cold-start perf (transformers_compat.py NEW, _transformers_compat.py deleted)

Prior state eagerly imported the transformers 5.x compat shim from winml.modelkit.__init__, dragging transformers (~10 s cold) into every invocation — including winml --help. That defeated the PEP 562 lazy dispatch the package already had. Fix: wrap the shim body in install(), register _OptimumImportHook at sys.meta_path[0] that fires install() the first time anything imports optimum.*. Lightweight commands never fire the hook; heavy commands fire it once, transparently. install() sets the guard flag at entry (prevents recursion when install() itself imports optimum) and rolls it back on any exception. winml --help cold path: 277 ms with transformers NOT in sys.modules.

6. Test coverage gap closed

tests/cli/ was outside the CI matrix and never caught the transformers regression that landed with the transformers-5 upgrade. Moved to tests/unit/cli/ per tests/CLAUDE.md layout rule, added cli group to .github/workflows/modelkit-ci.yml matrix, deleted the 460-line duplicate at repo root, added TestWallClock guardrail (winml --help < 8 s in a fresh subprocess).

Commits

• e6be7659 — feat(session): op-tracing monitor for second EP + universal dispatch
• 0e2b5f05 — perf(cli): auto-fire transformers 5.x compat shim via sys.meta_path hook
• 9a8993f4 — feat(session): isolate EP registration in fresh subprocesses for truthful per-DLL metadata

Pre-existing baseline failures (out of scope, reproduce on 15861f75)

• 3× _LAZY_IMPORTS[winml.modelkit.onnx] drift — onnx/__init__.py lacks the PEP 562 dispatch pattern other subpackages already use
• 3× TestSysCommand AttributeError: 'list' object has no attribute 'items' shape drift in sys command test fixtures

Both categories reproduce on 15861f75 before any commit on this branch and are explicitly declared out of scope in commit 0e2b5f05's message.

Test plan

• uv run pytest tests/unit/ tests/unit/cli/ --timeout=60 — expect 1149 passed, 6 pre-existing baseline failures, 8 legit skips
• uv run pytest tests/unit/session/monitor/test_openvino_monitor.py — 27 tests, all green
• uv run pytest tests/unit/commands/test_perf_optracing.py — 45 tests (7 new for op-tracing dispatch), all green
• uv run winml --help — sub-second, 'transformers' not in sys.modules
• $env:WINMLCLI_EP_PATH="...\x64\Release"; uv run winml sys — 5 truthful OpenVINO row versions surface + Architecture/Driver rendered for NPU/GPU + [failed] rows show compact Win32 reason + PE fallback version
• uv run winml perf -m microsoft/resnet-50 — exit 0, latency ~2.5 ms, throughput > 300 samples/sec
• uv run winml perf -m microsoft/resnet-50 --monitor — HW monitor renders, no monitor/EP errors
• uv run winml perf --help — --op-tracing help text mentions "Auto-selects the monitor for the chosen EP" (no vendor-specific dispatch language)

[timenick]

Let’s exclude the .md files from this pr.

[xieofxie]

please

• merge conflicts
• clean up commit-analysis and docs folder
• consider split pr into self-contained features like
  • update transformers to 5
  • speed up cold-start
  • EP related
  • OP tracing related

Thanks

[tezheng]

E2E verification checklist

Runnable verification protocol for this branch — see E2E-VERIFICATION.md at repo root (added in f084ca7f).

Verified on Intel/OpenVINO NPU host (author). Other platforms — please pick up the delta.

Minimum smoke (~5 min with warm cache)

uv run winml sys --format json
uv run winml perf -m microsoft/resnet-50 --device auto --ep auto --monitor
uv run winml perf -m ./convnext/model_opt_qdq.onnx --ep qnn --device npu --op-tracing basic
uv run winml perf -m t5-small --device cpu

Platforms wanted

Host	Priority checks not yet covered by author
Snapdragon (Qualcomm NPU)	perf + analyze with --ep qnn --device npu --op-tracing basic — currently ENV on author's box (Intel AI Boost NPU, not Qualcomm)
AMD (MIGraphX GPU)	sys --list-ep should surface MIGraphX rows; perf against --device gpu --ep migraphx
NVIDIA (TensorRT / TensorRTRTX GPU)	perf against --device gpu --ep nv_tensorrt_rtx
Fresh Windows install (WinML Catalog only)	Verify sys --list-ep renders Catalog source tag; perf with --ep openvino

How to classify failures

• REGRESSION: click parse error, ImportError, NameError, TypeError, or Python traceback originating from src/winml/modelkit/…. Block ship.
• ENVIRONMENTAL: hardware/EP unavailable on your host (No source for QNNExecutionProvider/npu exposed device class 'NPU', CompiledModel was not initialized, etc.). Doc has an ENV triage table. Not a ship blocker.
• DOWNSTREAM: known composite ONNX export issue (onnx_nodes: 0 → tuple index out of range on t5-small). Separate concern, unrelated to this commit's B3/B6 fixes.
• DOC-BUG: command runs but the doc's expected-output snippet doesn't match your platform's actual render. File a follow-up so the doc can be tuned.

Anti-pattern grep

Every regression signature has a grep pattern in the doc's grep table. If you find EP: ( in any output, T-01 tuple leak regressed. If you find TypeError: from_pretrained, B3 regressed. Full list in the doc.

Author's results (two runs, Intel host)

• Run 1 (initial doc): 42 PASS / 0 REG / 10 ENV
• Run 2 (revised doc): 39 PASS / 0 REG / 7 ENV / 1 DOWNSTREAM / 5 DOC-BUG (all format/casing polish; substantive claims verified)

Raw command outputs under commit-analysis/cc7a6650/e2e-results{,-v2}/ (untracked, session artifacts).

If you're picking up a platform

1. Run the minimum smoke first — that's the fastest signal.
2. If clean, run the section of the full matrix specific to your hardware.
3. Post results back here with PASS/REG/ENV counts.

[tezheng]

PR #1019 E2E Verification Plan

Branch: feat/ep-registration-and-monitoring (winml-cli, formerly ModelKit)
PR: #1019 — "feat: EP registration + monitoring — subprocess isolation, structured failures, universal op-tracing dispatch, lazy CLI startup"
Verification target commit: cc7a6650 ("feat(session): unified-source EP refactor + P0 fixes + simplification pass")

Context / why this doc exists

PR #1019's own "Test plan" checklist references a three-commit chain
(9a8993f4, 0e2b5f05, e6be7659) and a baseline of "reproduce on
15861f75". That history no longer exists — the branch was rewritten
upstream into a single squashed commit, cc7a6650, which the local repo
has already been synced to (git reset --hard, no force-push performed).
The PR description was not updated to match, so its stated expected
numbers (1149 passed, 6 pre-existing baseline failures, 8 legit skips)
are not guaranteed to still hold and must be re-derived, not assumed.

A prior manual check on cc7a6650 already found 6 failures in
tests/unit/commands/test_perf_optracing.py, all related to OpenVINO
op-tracing dispatch (RuntimeError: Op-tracing --ep openvino is not currently supported) — a different failure set than the two categories
the PR describes as pre-existing/out-of-scope. Task 1 below must
determine whether these 6 are old-known, newly-introduced, or something
else, before any task can be marked "matches expectation."

High-level goal

Independently re-verify every item in PR #1019's Test plan checklist
against the actual current branch tip (cc7a6650), using measured
results — not the PR body's (now possibly stale) claimed numbers — and
produce a clear per-item verdict: PASS (behavior matches what the
PR claims or a reasonable equivalent), FAIL (behavior contradicts
the claim, needs investigation/fix), or DEVIATION (behavior differs
from the claim but is explainable/documented, e.g. the OpenVINO
failures above).

High-level passing criteria

All of the following must be true to close this verification:

1. Every task below has a recorded PASS/FAIL/DEVIATION verdict with the
   actual command output (or a summary of it) as evidence — no verdict
   without evidence.
2. Any FAIL is either fixed (with a follow-up commit) or explicitly
   escalated to the user with a clear description of the discrepancy —
   never silently ignored or reclassified as DEVIATION to make the plan
   "pass."
3. The final report reconciles the PR body's stale claims (commit list,
   baseline failure count) against what's actually true on cc7a6650,
   so whoever reads the PR next isn't misled by outdated text.

Tasks

Each task is independent and can be picked up in any order, but Task 1
should run first since its finding affects how to interpret Task 2/3's
failure counts.

---

Task 1 — Full unit test suite baseline

Goal: establish the true current pass/fail/skip counts on cc7a6650,
and classify every failure as pre-existing/known vs. new.

Command:

uv run pytest tests/unit/ tests/unit/cli/ --timeout=60

Pass criteria:

• Command completes (no collection errors, no hangs).
• Every failing test is individually classified: matches one of the
  PR's two stated pre-existing categories (_LAZY_IMPORTS[winml.modelkit.onnx]
  drift, TestSysCommand shape drift), OR matches the 6
  already-found OpenVINO op-tracing dispatch failures, OR is genuinely
  new (→ FAIL, needs a bug report or fix).
• Skip count and reasons are sane (hardware-gated skips only, no
  accidental skips of things that should run).

Verdict recorded as: PASS if all failures fall into a known
category; FAIL if any failure is unclassified and looks like a real
regression; DEVIATION if the counts differ from the PR's claim but
every individual failure is still explainable.

---

Task 2 — OpenVINO monitor unit tests

Goal: confirm the second op-tracing monitor implementation
(OpenVINOMonitor) is fully green in isolation, independent of the
dispatch-layer failures found in Task 1/3.

Command:

uv run pytest tests/unit/session/monitor/test_openvino_monitor.py

Pass criteria: all tests pass (PR claims 27). If the count differs,
record the actual count — a differing count alone is not a FAIL as
long as every test present passes.

---

Task 3 — Perf/op-tracing dispatch unit tests

Goal: confirm commands/perf.py's _resolve_ep_monitor dispatch
logic (NPU/auto → try monitor A, fall back to B; GPU → B; explicit
--ep; --op-tracing detail rejection) works as designed.

Command:

uv run pytest tests/unit/commands/test_perf_optracing.py

Pass criteria: all tests pass (PR claims 45, 7 new for dispatch).
Known current state: 6 failures already found here, all
RuntimeError: Op-tracing --ep openvino is not currently supported /
related dispatch errors — these must be root-caused as part of this
task, not just re-confirmed. Determine: is OpenVINO op-tracing support
actually wired up on this branch, or does _resolve_ep_monitor have a
bug that never dispatches to it? The PR's own summary text explicitly
claims OpenVINO dispatch is implemented ("NPU/auto → try monitor A,
fall back to B") — if the tests say otherwise, that's a real
implementation/test mismatch to report, not paper over.

---

Task 4 — CLI cold-start performance

Goal: confirm winml --help no longer eagerly imports transformers
(PEP 562 lazy-dispatch regression fix).

Commands:

uv run winml --help

then verify transformers is absent from a fresh subprocess's
sys.modules (the PR's own TestWallClock guardrail test does this —
running it directly is an acceptable substitute for a manual check):

uv run pytest tests/unit/cli/test_import_time.py tests/unit/cli/test_main.py -k "WallClock or cold"

Pass criteria: winml --help returns sub-second; transformers is
not in sys.modules for that invocation.

---

Task 5 — winml sys with directory-sourced EP override (vendor matrix: QNN / OpenVINO / VitisAI)

Goal: confirm EP registration hardening (subprocess isolation,
structured failure attribution, PE fallback-version reads) produces
truthful output when pointed at a real EP DLL directory — for
whichever EP vendor(s) the test machine actually has hardware for.
The original version of this task only exercised OpenVINO (Intel), which
silently assumes an Intel test machine. Per
src/winml/modelkit/session/ep_device.py's EP_DEVICE_SPECS catalog,
the NPU device has three vendor-optimal EPs in preference order — QNN
(Qualcomm) → OpenVINO (Intel) → VitisAI (AMD) — so this task should be
run against whichever of the three matches the tester's actual hardware,
not assumed to be OpenVINO/Intel by default.

Vendor → hardware mapping (pick the row(s) matching the test
machine; run more than one row if the machine has multiple EP builds
available):

Vendor	EP name	Typical hardware	Directory to point WINMLCLI_EP_PATH at
Qualcomm	QNNExecutionProvider	Snapdragon/Copilot+ NPU (arm64/arm64ec)	A QNN EP build dir, e.g. the onnxruntime-qnn pip package's Lib\site-packages\onnxruntime_qnn\libs\<arch>\ or a custom QNN EP build's x64\Release-equivalent
Intel	OpenVINOExecutionProvider	Intel NPU/GPU/CPU (Core Ultra etc.)	A compiled OpenVINO EP build directory, <path>\x64\Release
AMD	VitisAIExecutionProvider	AMD Ryzen AI NPU	A compiled VitisAI EP build directory, <path>\x64\Release

Command (adjust path to an actual available EP build directory for
the vendor being tested — ask the user for the correct
WINMLCLI_EP_PATH target if none is on hand; do not fabricate a path):

$env:WINMLCLI_EP_PATH="<path-to-vendor-EP-build-dir>"; uv run winml sys

Pass criteria (apply per vendor tested): N truthful <vendor> row
versions surface (N = however many EP DLL variants exist in that
directory); Architecture/Driver columns render for NPU/GPU rows; any
[failed] rows show a compact Win32 reason string and a PE-read
fallback version rather than a blank/opaque failure — for whichever
vendor(s) were actually exercised.

Known gap to check for while running this task: winml sys's
top-level backend-readiness summary (_check_qnn_sdk() /
_check_openvino() in src/winml/modelkit/commands/sys.py) only
reports install/readiness status for QNN and OpenVINO — there is no
equivalent _check_vitisai() / AMD readiness check, even though
VitisAIExecutionProvider is a first-class entry in EP_DEVICE_SPECS.
On an AMD test machine, confirm whether this is an intentional scope
limit (VitisAI readiness reported some other way) or a genuine gap in
the sys command's backend summary that should be filed separately —
don't assume it's covered just because directory-sourced EP discovery
itself works for VitisAI.

Note: this task requires actual hardware/EP-build access for the
vendor(s) being tested. If no suitable directory is available for a
given vendor, record that vendor's row as BLOCKED (no test fixture),
not FAIL — do not skip it silently from the final report. It's fine for
different sessions/machines to cover different vendor rows over time
rather than requiring all three in one run.

---

Task 6 — winml perf baseline (no monitor)

Goal: confirm the EP registration/dispatch changes haven't
regressed baseline inference performance or exit codes.

Command (device/EP auto-detected by default; optionally force a
specific vendor with --ep qnn, --ep openvino, or --ep vitisai to
explicitly cover a particular combination rather than relying on
whatever the test machine auto-picks — useful when a machine has more
than one EP installed):

uv run winml perf -m microsoft/resnet-50

Pass criteria: exit code 0; throughput > 300 samples/sec. The
~2.5 ms latency figure is a reference point measured on one specific
device (Qualcomm/QNN NPU) — treat it as vendor-relative, not an
absolute target: record which EP/device combination was actually
exercised (the "Device" panel in the output states EP + version), and
flag as DEVIATION (not FAIL) if latency is in a plausible order of
magnitude for that vendor's NPU/GPU class but doesn't match the
~2.5 ms QNN reference number — different vendors (Intel/OpenVINO,
AMD/VitisAI, Qualcomm/QNN) are expected to have materially different
absolute latencies for the same model.

---

Task 7 — winml perf --monitor (hardware monitor)

Goal: confirm the HW monitor path still renders correctly and
doesn't throw monitor/EP errors after the registration changes.

Command:

uv run winml perf -m microsoft/resnet-50 --monitor

Pass criteria: HW monitor output renders (CPU/RAM/NPU utilization
etc.); no monitor- or EP-related errors or tracebacks in output.

---

Task 8 — winml perf --help wording check

Goal: confirm the --op-tracing help text was generalized away
from vendor-specific (QNN-only) language now that OpenVINO is a second
supported monitor.

Command:

uv run winml perf --help

Pass criteria: help text for --op-tracing mentions "Auto-selects
the monitor for the chosen EP" (or equivalent EP-agnostic phrasing) —
no leftover QNN-specific or vendor-specific wording implying only one
EP is supported.

---

Task 9 — Reconcile PR description with actual state

Goal: not a test — a documentation fix. After Tasks 1–8 are done,
summarize the discrepancies between the PR body's stated commit list /
baseline-failure claims and what's actually true on cc7a6650.

Pass criteria: a clear written summary (in the final report, not
necessarily a PR edit unless the user asks for one) of what changed
between what the PR says and what's actually on the branch now,
so a reviewer isn't misled.

Final report format

For each task: task number, verdict (PASS / FAIL / DEVIATION / BLOCKED),
one-line evidence summary, and a pointer to full command output if
retained. Close with the high-level goal's three passing criteria,
explicitly confirmed or not.

[timenick]

This only resolves ep_device when the caller explicitly supplies device, but from_pretrained() still documents and accepts calls with neither target set. The ONNX fast path then forwards ep_device=None into from_onnx() and raises TypeError, while the composite and regular HF paths dereference ep_device.device. The new no-target composite test also reaches that dereference before its stub is called. Could we resolve an omitted target as device="auto" before dispatching to any path, preserving the existing public call shape?

🤖 Generated with GitHub Copilot CLI

[timenick]

This fallback also catches failures for explicitly requested targets. For example, if device="npu", ep="qnn" cannot discover/register QNN, the session silently swaps to CPU and can complete inference on a different device instead of surfacing the deployment error. Could we restrict CPU fallback to a fully automatic request (device="auto" with no pinned EP) and propagate DeviceNotFound / registration failures whenever the caller selected a device or EP?

🤖 Generated with GitHub Copilot CLI

[timenick]

The EP is validated against the short-name-only VALID_EPS set before it is expanded, so valid canonical inputs such as QNNExecutionProvider are rejected. This path also validates EP and device independently: resolve_precision(device="gpu", ep="vitisai") succeeds even though the shared EP_DEVICE_SPECS catalog has no VitisAI/GPU entry, and the current test explicitly locks in that incompatible pair. Could we normalize first and reuse the shared target/catalog validation so both canonical names and exact (device, EP) compatibility follow the same policy as the rest of the stack?

🤖 Generated with GitHub Copilot CLI

[timenick]

Removing this else makes parent_config.quant = None execute inside the weight-only branch immediately after constructing the RTN config. Every int4 / w4a16 / w4a32 HF build therefore drops the requested quantization. Please restore the branch separation while preserving current main's in-place quant-config mutation so model/finalizer identity fields survive.

🤖 Generated with GitHub Copilot CLI

[timenick]

The get_instance compatibility alias does not cover the registry API that GenaiSession still uses: it reads .winml_available and calls .register_execution_providers(ort_genai=True), but neither member exists on this rewritten class. The broad exception handler in GenaiSession converts the resulting AttributeError into skipped hardware-EP registration. Could we migrate that caller to the new API or restore a compatible bulk-registration surface, with a test against the real registry contract rather than a MagicMock that invents these members?

🤖 Generated with GitHub Copilot CLI