feat: Add ONNX data to QNN profiler metrics#1102
Conversation
|
Sample result |
DingmaomaoBJTU
left a comment
There was a problem hiding this comment.
Reviewed the ONNX metadata enrichment for QNN basic profiler metrics. The schema-name input resolution, the env gating, and the on/off test coverage look solid. Left a few inline notes — the main one is that an enrichment error can currently void an otherwise-successful trace.
|
|
||
| operators = _csv_operator_metrics(measured) | ||
| onnx_operator_data = ( | ||
| _load_onnx_operator_data(self.onnx_path) if _is_op_add_data_enabled() else None |
There was a problem hiding this comment.
When WINMLCLI_OP_ADD_DATA is set, any exception raised inside _load_onnx_operator_data propagates out of _from_csv and discards profiling results that were already computed successfully. Several paths here can throw: load_onnx (missing/corrupt model, or a *_ctx.onnx context binary), TensorProto.DataType.Name() on an unknown enum value, and _serialize_attribute on unsupported attribute types.
Since this is opt-in enrichment, a metadata failure shouldn't be able to void an otherwise-valid trace. Suggest wrapping the call in try/except, logging a warning, and falling back to None.
Verified locally: with the env var set and an unreadable model path, _from_csv raises FileNotFoundError instead of returning the (valid) operator metrics.
| return _sparse_tensor_attribute_metadata(value) | ||
| if attr_type == AttributeProto.SPARSE_TENSORS: | ||
| return [_sparse_tensor_attribute_metadata(tensor) for tensor in value] | ||
| return value |
There was a problem hiding this comment.
This fallthrough returns the raw protobuf value for any attribute type not handled above — notably TYPE_PROTO / TYPE_PROTOS (and UNDEFINED). Those are not JSON-serializable, so OpTraceResult.to_json() fails with TypeError: Object of type TypeProto is not JSON serializable.
Verified locally with a TYPE_PROTO attribute. Suggest handling those cases explicitly, or using a safe fallback such as str(value), so serialization can't break.
| operator_data[node.name] = { | ||
| "onnx_op_type": node.op_type, | ||
| "onnx_attributes": {attr.name: _serialize_attribute(attr) for attr in node.attribute}, | ||
| "onnx_inputs": _node_input_metadata(node, opset_versions, value_info, initializers), |
There was a problem hiding this comment.
This captures input tensor shapes/types but not the node's output tensor(s). Given the branch name (add_op_size), the output shape is often the more useful "size" signal for a profiler — and an op's inputs are usually another op's outputs anyway. Was omitting onnx_outputs intentional, or worth capturing alongside inputs? Non-blocking.
Summary
Verification