Skip to content

fix: path-string runtime cache freed itself out from under the engine - #4482

Draft
tp5uiuc wants to merge 2 commits into
pytorch:mainfrom
tp5uiuc:fix/python-runtime-cache-on-cpp-build
Draft

fix: path-string runtime cache freed itself out from under the engine#4482
tp5uiuc wants to merge 2 commits into
pytorch:mainfrom
tp5uiuc:fix/python-runtime-cache-on-cpp-build

Conversation

@tp5uiuc

@tp5uiuc tp5uiuc commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

What

A runtime_cache passed as a path string freed itself out from under the engine: the next createExecutionContext ran against a destroyed IRuntimeCache and segfaulted.

Why

TRTRuntimeConfig is documented as owning the engine-implicit RuntimeCache, but had no slot for it — _apply_settings built it as a local with autosave_on_del=True, so it was collected the moment the method returned, taking the live IRuntimeCache with it.

TorchTensorRTModule pre-wraps path strings into a module-owned handle, so engines reached through compile() never take this branch. Engines built straight from packed engine info do: they carry the default RuntimeSettings(runtime_cache=RUNTIME_CACHE_PATH) with no module around them to resolve it. As a side effect the disk cache was never populated on this path — autosave fired before the engine had generated a kernel.

How

  1. Guardensure_cache() returns None when the facade is torchbind-backed. No library path reaches that (every in-library TRTEngine construction site is gated on the same flag that picks the backing), but a hand-built engine on a build shipping libtorchtrt.so does, and set_runtime_cache rejects an explicitly-passed None despite advertising cache: IRuntimeCache = None. Skip the attach and warn instead of surfacing a pybind signature error; the two duplicated call sites fold into one helper.
  2. Lifetime — hold the implicit wrapper on the config, reused across reset() when the path is unchanged so a settings swap does not discard in-memory kernels. Mirrors what TorchTensorRTModule._resolve_runtime_cache already does for module-owned handles.

TestPythonRuntimeAliasedIO is skipped when the C++ runtime is available: it hand-builds a Python engine, which is a configuration the library never produces there. It stays live on python-only builds, matching the existing guard in test_000_runtime_cache.py and test_001_dynamic_shapes_kernel_strategy.py.

Testing

Build Result
C++ runtime, full tests/py/dynamo/runtime/ 169 passed, 51 skipped
python-only TRT-RTX, runtime-cache + aliased-IO 38 passed, 1 skipped
python-only TRT-RTX, before this change SIGSEGV in createExecutionContext

The crash is caught by test_bare_engine_executes_with_default_path_cache. On main, TestPythonRuntimeAliasedIO already fails on python-only TRT-RTX for the same root cause (Failed to create execution context), so this is pre-existing rather than introduced here.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR in so that relevant reviewers are notified

@meta-cla meta-cla Bot added the cla signed label Aug 13, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: api [Python] Issues re: Python API labels Aug 13, 2026
@github-actions
github-actions Bot requested a review from narendasan August 13, 2026 00:42
@tp5uiuc
tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch from c3c4cf7 to 03513c2 Compare August 13, 2026 01:36
@tp5uiuc tp5uiuc changed the title fix: python engine crashed attaching a runtime cache on builds with the C++ runtime fix: path-string runtime cache freed itself out from under the engine Aug 13, 2026
RuntimeCache.ensure_cache() returns None when the facade is torchbind-backed.
No library path reaches that: the runtime flavor is chosen from whether
libtorchtrt.so is loaded, and every in-library TRTEngine construction site is
gated on that same flag, so a python engine only exists when the facade is
python-backed. A hand-built TRTEngine on a build that ships the .so does reach
it, which is what TestPythonRuntimeAliasedIO constructs.

set_runtime_cache advertises `cache: IRuntimeCache = None` but its type caster
rejects an explicitly-passed None, so that None surfaced as a confusing pybind
signature error instead of a diagnosable one. Skip the attach and warn, and
fold the two duplicated ensure_cache -> set_runtime_cache call sites into one
helper while here.

Skip TestPythonRuntimeAliasedIO on TensorRT-RTX builds that also have the C++
runtime, rather than teach the cache plumbing to serve a configuration the
library never produces. The guard is deliberately narrow: on standard TensorRT
the runtime-config path is short-circuited in ensure_initialized, so a
hand-built engine is harmless there and the class stays live. It also stays
live on python-only builds, where the Python TRTEngine is the real runtime.
…_settings

TRTRuntimeConfig is documented as owning the engine-implicit RuntimeCache but
had no slot for it. When runtime_cache was a path string, _apply_settings
wrapped it in a RuntimeCache local carrying autosave_on_del=True, so the
wrapper was collected the moment the method returned -- taking the live
IRuntimeCache with it and leaving the engine's IRuntimeConfig holding freed
memory. The next createExecutionContext segfaults.

TorchTensorRTModule pre-wraps path strings into a module-owned handle, so
engines reached through compile() never take this branch. Engines built
straight from packed engine info do: they carry the default
RuntimeSettings(runtime_cache=RUNTIME_CACHE_PATH) with no module around them
to resolve it. Confirmed on a python-only TRT-RTX build, where such an engine
segfaults on first execute before this change and runs after it.

Hold the wrapper on the config, reusing it across reset() cycles when the path
is unchanged so a settings swap does not discard in-memory kernels -- mirroring
what TorchTensorRTModule._resolve_runtime_cache already does for module-owned
handles. As a side effect the disk cache now actually gets populated on this
path; previously autosave fired before the engine had generated a kernel.
@tp5uiuc
tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch from 03513c2 to 5522dcd Compare August 13, 2026 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: tests Issues re: Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant