Skip to content

fix: stop calling TensorRT 10.15 aliasing APIs unconditionally - #4468

Open
shoumikhin wants to merge 2 commits into
pytorch:mainfrom
shoumikhin:trt-version-guard
Open

fix: stop calling TensorRT 10.15 aliasing APIs unconditionally#4468
shoumikhin wants to merge 2 commits into
pytorch:mainfrom
shoumikhin:trt-version-guard

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

The problem

Three things Torch-TensorRT calls without checking were added in TensorRT 10.15:

  • ICudaEngine::getAliasedInputTensor (C++)
  • ICudaEngine.get_aliased_input_tensor (Python)
  • the KV-cache update layer, with its KVCacheMode enum

Building against an older TensorRT fails outright:

core/runtime/TRTEngine.cpp: error: 'class nvinfer1::ICudaEngine'
    has no member named 'getAliasedInputTensor'

Once that is past, compiling any model fails, not only one with a cache:

File "py/torch_tensorrt/dynamo/conversion/_TRTInterpreter.py", line 519, in run
    aliased_in = cuda_engine.get_aliased_input_tensor(out_name)
AttributeError: 'ICudaEngine' object has no attribute 'get_aliased_input_tensor'

That loop runs after every engine build, over every output, so an ordinary model with no
aliasing at all hits it. The same call on the Python runtime fails the same way at engine
load.

The fix

Reconciliation compares the build-time alias map against what an engine reports about
itself. A build without the API cannot report anything, so there is nothing to compare
and the build-time map already stands alone. Skip only that loop, and keep everything
after it, because user-declared aliases exist independently of the TensorRT API and their
binding names still need recomputing.

Writing through an alias is a different matter, because it cannot be emulated. Where the
code promises a write-through, it now refuses rather than quietly dropping it. Two paths
promise one:

  • A cache passed in as a network input and written in place. Falling back to the
    functional scatter computed the correct return value and never touched the caller's
    tensor. That is worse than an error, because a test that only checks numbers passes
    while the cache silently stops updating.
  • A mutated module buffer. Lifting removes the writeback copy, on the assumption the
    engine writes through the aliased input.

Both now raise, naming the input or the buffer and the version required:

Writing in place to the network input 'cache' needs the TensorRT KV-cache update
layer, which was added in TensorRT 10.15. This build does not have it, and without
it the write would be computed and then discarded. Use TensorRT 10.15 or newer, or
return the updated tensor instead of writing into an input.

One shared capability check backs both sites, so they cannot drift apart.

Scope

This does not make Torch-TensorRT work on TensorRT older than 10.15. Other 10.15-only
symbols are still referenced unconditionally, including trt.IAttention in an annotation
that is evaluated at import time, so importing the package on 10.13 fails before any of
this runs. What this change does is take these three APIs off that list.

Testing

ordinary model, aliasing API removed        compiles, matches eager
module buffer write, KV layer removed       raises, names the buffer
network input write, KV layer removed       raises, names the input
unmodified TensorRT 11.1.0.106              unchanged, matches eager
tests/py/dynamo/runtime/test_aliased_io.py  13 passed
black --check on all five changed files     clean

The three added tests fail without this change and pass with it. Also confirmed on an
aarch64 device with system TensorRT 10.13.3.9 that all three APIs are absent there, and
that the C++ version check evaluates false on it.

@meta-cla meta-cla Bot added the cla signed label Aug 8, 2026
@github-actions github-actions Bot added component: core Issues re: The core compiler component: runtime labels Aug 8, 2026
@github-actions
github-actions Bot requested a review from narendasan August 8, 2026 21:45
Building Torch-TensorRT against TensorRT 10.13 or older fails to compile:

  core/runtime/TRTEngine.cpp:298: error: 'class nvinfer1::ICudaEngine'
      has no member named 'getAliasedInputTensor'

That API was added in TensorRT 10.15, together with the IKVCacheUpdateLayer
that produces the aliasing it reports. The runtime calls it unconditionally,
while the package declares a bare tensorrt dependency with no minimum version,
so an older TensorRT produces a compile error rather than a clear message.

The call only reconciles the build-time aliased I/O map against what the engine
reports. On older TensorRT there is nothing to reconcile against, so the
build-time map stands on its own and the reconciliation is skipped.

Gate it with a direct version check at the point of use, matching how the
ScatterAdd plugin include is gated in core/plugins/register_plugins.cpp.
@github-actions github-actions Bot added component: conversion Issues re: Conversion stage component: converters Issues re: Specific op converters component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Aug 8, 2026
@shoumikhin shoumikhin changed the title fix: build against TensorRT older than 10.15 fix: support TensorRT older than 10.15 Aug 8, 2026
@narendasan

Copy link
Copy Markdown
Collaborator

@shoumikhin What Jetpack version are you using?

@shoumikhin

Copy link
Copy Markdown
Contributor Author

JetPack 7.1 (L4T R38.4, CUDA 13.0, aarch64), whose apt channel ships TensorRT 10.13.3.9, where getAliasedInputTensor is absent from the installed headers and trt.KVCacheMode / add_kv_cache_update are missing from the Python module.

Three things Torch-TensorRT calls without checking were added in TensorRT 10.15:
ICudaEngine::getAliasedInputTensor, its Python counterpart
get_aliased_input_tensor, and the KV-cache update layer with its KVCacheMode enum.

Building against an older TensorRT fails outright:

    core/runtime/TRTEngine.cpp: error: 'class nvinfer1::ICudaEngine'
        has no member named 'getAliasedInputTensor'

and compiling any model fails, not only one with a cache:

    File "py/torch_tensorrt/dynamo/conversion/_TRTInterpreter.py", line 519, in run
        aliased_in = cuda_engine.get_aliased_input_tensor(out_name)
    AttributeError: 'ICudaEngine' object has no attribute
        'get_aliased_input_tensor'

That loop runs after every engine build over every output, so an ordinary model
with no aliasing at all hits it. The same call on the Python runtime fails the same
way at engine load.

Reconciliation compares the build-time alias map against what an engine reports
about itself. A build without the API cannot report anything, so there is nothing
to compare and the build-time map already stands alone. Skip only that loop and
keep everything after it, because user-declared aliases exist independently of the
API and their binding names still need recomputing.

Writing through an alias is different: it cannot be emulated. Where the code
promises a write-through it now refuses instead of quietly dropping it. Two paths
promise one:

  * A cache passed in as a network input and written in place. Falling back to the
    functional scatter computed the correct return value and never touched the
    caller's tensor, which is worse than an error because a numerics check passes.
  * A mutated module buffer. Lifting removes the writeback copy on the assumption
    the engine writes through the aliased input.

Both now raise, naming the input or buffer and the version required. One shared
capability check backs both, so the two sites cannot drift apart.

This does not make Torch-TensorRT work on TensorRT older than 10.15. Other
10.15-only symbols are still referenced unconditionally, including trt.IAttention
in an annotation evaluated at import time, so importing the package on 10.13 fails
before any of this runs. This change removes these three from that list.

Testing:

    ordinary model, aliasing API removed     compiles, matches eager
    module buffer write, KV layer removed    raises, names the buffer
    network input write, KV layer removed    raises, names the input
    unmodified TensorRT 11.1.0.106           unchanged, matches eager
    tests/py/dynamo/runtime/test_aliased_io.py     13 passed
    black --check on all five changed files       clean

The three added tests fail without this change and pass with it. Confirmed on an
aarch64 device with system TensorRT 10.13.3.9 that all three APIs are absent there.
@shoumikhin shoumikhin changed the title fix: support TensorRT older than 10.15 fix: stop calling TensorRT 10.15 aliasing APIs unconditionally Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: conversion Issues re: Conversion stage component: converters Issues re: Specific op converters component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: runtime component: tests Issues re: Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants