fix: stop calling TensorRT 10.15 aliasing APIs unconditionally - #4468
Open
shoumikhin wants to merge 2 commits into
Open
fix: stop calling TensorRT 10.15 aliasing APIs unconditionally#4468shoumikhin wants to merge 2 commits into
shoumikhin wants to merge 2 commits into
Conversation
Building Torch-TensorRT against TensorRT 10.13 or older fails to compile:
core/runtime/TRTEngine.cpp:298: error: 'class nvinfer1::ICudaEngine'
has no member named 'getAliasedInputTensor'
That API was added in TensorRT 10.15, together with the IKVCacheUpdateLayer
that produces the aliasing it reports. The runtime calls it unconditionally,
while the package declares a bare tensorrt dependency with no minimum version,
so an older TensorRT produces a compile error rather than a clear message.
The call only reconciles the build-time aliased I/O map against what the engine
reports. On older TensorRT there is nothing to reconcile against, so the
build-time map stands on its own and the reconciliation is skipped.
Gate it with a direct version check at the point of use, matching how the
ScatterAdd plugin include is gated in core/plugins/register_plugins.cpp.
shoumikhin
force-pushed
the
trt-version-guard
branch
from
August 8, 2026 21:52
9c1e299 to
80f2e04
Compare
shoumikhin
force-pushed
the
trt-version-guard
branch
from
August 9, 2026 23:37
debf56a to
9bce699
Compare
Collaborator
|
@shoumikhin What Jetpack version are you using? |
Contributor
Author
|
JetPack 7.1 (L4T R38.4, CUDA 13.0, aarch64), whose apt channel ships TensorRT 10.13.3.9, where |
shoumikhin
force-pushed
the
trt-version-guard
branch
from
August 13, 2026 04:38
9bce699 to
73a906f
Compare
Three things Torch-TensorRT calls without checking were added in TensorRT 10.15:
ICudaEngine::getAliasedInputTensor, its Python counterpart
get_aliased_input_tensor, and the KV-cache update layer with its KVCacheMode enum.
Building against an older TensorRT fails outright:
core/runtime/TRTEngine.cpp: error: 'class nvinfer1::ICudaEngine'
has no member named 'getAliasedInputTensor'
and compiling any model fails, not only one with a cache:
File "py/torch_tensorrt/dynamo/conversion/_TRTInterpreter.py", line 519, in run
aliased_in = cuda_engine.get_aliased_input_tensor(out_name)
AttributeError: 'ICudaEngine' object has no attribute
'get_aliased_input_tensor'
That loop runs after every engine build over every output, so an ordinary model
with no aliasing at all hits it. The same call on the Python runtime fails the same
way at engine load.
Reconciliation compares the build-time alias map against what an engine reports
about itself. A build without the API cannot report anything, so there is nothing
to compare and the build-time map already stands alone. Skip only that loop and
keep everything after it, because user-declared aliases exist independently of the
API and their binding names still need recomputing.
Writing through an alias is different: it cannot be emulated. Where the code
promises a write-through it now refuses instead of quietly dropping it. Two paths
promise one:
* A cache passed in as a network input and written in place. Falling back to the
functional scatter computed the correct return value and never touched the
caller's tensor, which is worse than an error because a numerics check passes.
* A mutated module buffer. Lifting removes the writeback copy on the assumption
the engine writes through the aliased input.
Both now raise, naming the input or buffer and the version required. One shared
capability check backs both, so the two sites cannot drift apart.
This does not make Torch-TensorRT work on TensorRT older than 10.15. Other
10.15-only symbols are still referenced unconditionally, including trt.IAttention
in an annotation evaluated at import time, so importing the package on 10.13 fails
before any of this runs. This change removes these three from that list.
Testing:
ordinary model, aliasing API removed compiles, matches eager
module buffer write, KV layer removed raises, names the buffer
network input write, KV layer removed raises, names the input
unmodified TensorRT 11.1.0.106 unchanged, matches eager
tests/py/dynamo/runtime/test_aliased_io.py 13 passed
black --check on all five changed files clean
The three added tests fail without this change and pass with it. Confirmed on an
aarch64 device with system TensorRT 10.13.3.9 that all three APIs are absent there.
shoumikhin
force-pushed
the
trt-version-guard
branch
from
August 13, 2026 07:14
73a906f to
86ecb3d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Three things Torch-TensorRT calls without checking were added in TensorRT 10.15:
ICudaEngine::getAliasedInputTensor(C++)ICudaEngine.get_aliased_input_tensor(Python)KVCacheModeenumBuilding against an older TensorRT fails outright:
Once that is past, compiling any model fails, not only one with a cache:
That loop runs after every engine build, over every output, so an ordinary model with no
aliasing at all hits it. The same call on the Python runtime fails the same way at engine
load.
The fix
Reconciliation compares the build-time alias map against what an engine reports about
itself. A build without the API cannot report anything, so there is nothing to compare
and the build-time map already stands alone. Skip only that loop, and keep everything
after it, because user-declared aliases exist independently of the TensorRT API and their
binding names still need recomputing.
Writing through an alias is a different matter, because it cannot be emulated. Where the
code promises a write-through, it now refuses rather than quietly dropping it. Two paths
promise one:
functional scatter computed the correct return value and never touched the caller's
tensor. That is worse than an error, because a test that only checks numbers passes
while the cache silently stops updating.
engine writes through the aliased input.
Both now raise, naming the input or the buffer and the version required:
One shared capability check backs both sites, so they cannot drift apart.
Scope
This does not make Torch-TensorRT work on TensorRT older than 10.15. Other 10.15-only
symbols are still referenced unconditionally, including
trt.IAttentionin an annotationthat is evaluated at import time, so importing the package on 10.13 fails before any of
this runs. What this change does is take these three APIs off that list.
Testing
The three added tests fail without this change and pass with it. Also confirmed on an
aarch64 device with system TensorRT 10.13.3.9 that all three APIs are absent there, and
that the C++ version check evaluates false on it.