Skip to content

feat: Add ONNX data to QNN profiler metrics#1102

Open
xieofxie wants to merge 5 commits into
mainfrom
hualxie/add_op_size
Open

feat: Add ONNX data to QNN profiler metrics#1102
xieofxie wants to merge 5 commits into
mainfrom
hualxie/add_op_size

Conversation

@xieofxie

Copy link
Copy Markdown
Contributor

Summary

  • Add env-gated ONNX metadata enrichment for QNN basic profiler metrics via WINMLCLI_OP_ADD_DATA
  • Include ONNX node type, attributes, and input tensor metadata keyed by input name
  • Add focused unit coverage for default and env-enabled output shapes

Verification

  • uv run ruff check --fix src\winml\modelkit\optracing\result.py src\winml\modelkit\optracing\qnn\profiler.py tests\unit\optracing\test_qnn_profiler.py
  • uv run pytest tests\unit\optracing\test_qnn_profiler.py

@xieofxie xieofxie requested a review from a team as a code owner July 13, 2026 06:33
Comment thread src/winml/modelkit/optracing/qnn/profiler.py Fixed
Comment thread src/winml/modelkit/optracing/qnn/profiler.py Fixed
Comment thread tests/unit/optracing/test_qnn_profiler.py Fixed
@xieofxie xieofxie changed the title Add ONNX data to QNN profiler metrics feat: Add ONNX data to QNN profiler metrics Jul 13, 2026
@xieofxie

Copy link
Copy Markdown
Contributor Author

Sample result

    {
      "name": "/resnet/encoder/stages.1/layers.0/layer/layer.1/convolution/Conv",
      "op_path": "/resnet/encoder/stages.1/layers.0/layer/layer.1/convolution/Conv",
      "op_id": 588,
      "duration_us": 45.286626051604955,
      "percent_of_total": 4.619123992288399,
      "onnx_op_type": "Conv",
      "onnx_attributes": {
        "dilations": [
          1,
          1
        ],
        "group": 1,
        "kernel_shape": [
          3,
          3
        ],
        "pads": [
          1,
          1,
          1,
          1
        ],
        "strides": [
          2,
          2
        ]
      },
      "onnx_inputs": {
        "X": {
          "name": "/resnet/encoder/stages.1/layers.0/layer/layer.0/activation/Relu_output_0_DequantizeLinear_Output",
          "data_type": "FLOAT",
          "dims": [
            1,
            128,
            56,
            56
          ]
        },
        "W": {
          "name": "onnx::Conv_533_DequantizeLinear_Output",
          "data_type": "FLOAT",
          "dims": [
            128,
            128,
            3,
            3
          ]
        },
        "B": {
          "name": "onnx::Conv_534",
          "data_type": "FLOAT",
          "dims": [
            128
          ]
        }
      }
    },

@DingmaomaoBJTU DingmaomaoBJTU left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the ONNX metadata enrichment for QNN basic profiler metrics. The schema-name input resolution, the env gating, and the on/off test coverage look solid. Left a few inline notes — the main one is that an enrichment error can currently void an otherwise-successful trace.


operators = _csv_operator_metrics(measured)
onnx_operator_data = (
_load_onnx_operator_data(self.onnx_path) if _is_op_add_data_enabled() else None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When WINMLCLI_OP_ADD_DATA is set, any exception raised inside _load_onnx_operator_data propagates out of _from_csv and discards profiling results that were already computed successfully. Several paths here can throw: load_onnx (missing/corrupt model, or a *_ctx.onnx context binary), TensorProto.DataType.Name() on an unknown enum value, and _serialize_attribute on unsupported attribute types.

Since this is opt-in enrichment, a metadata failure shouldn't be able to void an otherwise-valid trace. Suggest wrapping the call in try/except, logging a warning, and falling back to None.

Verified locally: with the env var set and an unreadable model path, _from_csv raises FileNotFoundError instead of returning the (valid) operator metrics.

return _sparse_tensor_attribute_metadata(value)
if attr_type == AttributeProto.SPARSE_TENSORS:
return [_sparse_tensor_attribute_metadata(tensor) for tensor in value]
return value

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fallthrough returns the raw protobuf value for any attribute type not handled above — notably TYPE_PROTO / TYPE_PROTOS (and UNDEFINED). Those are not JSON-serializable, so OpTraceResult.to_json() fails with TypeError: Object of type TypeProto is not JSON serializable.

Verified locally with a TYPE_PROTO attribute. Suggest handling those cases explicitly, or using a safe fallback such as str(value), so serialization can't break.

operator_data[node.name] = {
"onnx_op_type": node.op_type,
"onnx_attributes": {attr.name: _serialize_attribute(attr) for attr in node.attribute},
"onnx_inputs": _node_input_metadata(node, opset_versions, value_info, initializers),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This captures input tensor shapes/types but not the node's output tensor(s). Given the branch name (add_op_size), the output shape is often the more useful "size" signal for a profiler — and an op's inputs are usually another op's outputs anyway. Was omitting onnx_outputs intentional, or worth capturing alongside inputs? Non-blocking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants