fix: path-string runtime cache freed itself out from under the engine - #4482
Draft
tp5uiuc wants to merge 2 commits into
Draft
fix: path-string runtime cache freed itself out from under the engine#4482tp5uiuc wants to merge 2 commits into
tp5uiuc wants to merge 2 commits into
Conversation
tp5uiuc
force-pushed
the
fix/python-runtime-cache-on-cpp-build
branch
from
August 13, 2026 01:36
c3c4cf7 to
03513c2
Compare
RuntimeCache.ensure_cache() returns None when the facade is torchbind-backed. No library path reaches that: the runtime flavor is chosen from whether libtorchtrt.so is loaded, and every in-library TRTEngine construction site is gated on that same flag, so a python engine only exists when the facade is python-backed. A hand-built TRTEngine on a build that ships the .so does reach it, which is what TestPythonRuntimeAliasedIO constructs. set_runtime_cache advertises `cache: IRuntimeCache = None` but its type caster rejects an explicitly-passed None, so that None surfaced as a confusing pybind signature error instead of a diagnosable one. Skip the attach and warn, and fold the two duplicated ensure_cache -> set_runtime_cache call sites into one helper while here. Skip TestPythonRuntimeAliasedIO on TensorRT-RTX builds that also have the C++ runtime, rather than teach the cache plumbing to serve a configuration the library never produces. The guard is deliberately narrow: on standard TensorRT the runtime-config path is short-circuited in ensure_initialized, so a hand-built engine is harmless there and the class stays live. It also stays live on python-only builds, where the Python TRTEngine is the real runtime.
…_settings TRTRuntimeConfig is documented as owning the engine-implicit RuntimeCache but had no slot for it. When runtime_cache was a path string, _apply_settings wrapped it in a RuntimeCache local carrying autosave_on_del=True, so the wrapper was collected the moment the method returned -- taking the live IRuntimeCache with it and leaving the engine's IRuntimeConfig holding freed memory. The next createExecutionContext segfaults. TorchTensorRTModule pre-wraps path strings into a module-owned handle, so engines reached through compile() never take this branch. Engines built straight from packed engine info do: they carry the default RuntimeSettings(runtime_cache=RUNTIME_CACHE_PATH) with no module around them to resolve it. Confirmed on a python-only TRT-RTX build, where such an engine segfaults on first execute before this change and runs after it. Hold the wrapper on the config, reusing it across reset() cycles when the path is unchanged so a settings swap does not discard in-memory kernels -- mirroring what TorchTensorRTModule._resolve_runtime_cache already does for module-owned handles. As a side effect the disk cache now actually gets populated on this path; previously autosave fired before the engine had generated a kernel.
tp5uiuc
force-pushed
the
fix/python-runtime-cache-on-cpp-build
branch
from
August 13, 2026 01:48
03513c2 to
5522dcd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A
runtime_cachepassed as a path string freed itself out from under the engine: the nextcreateExecutionContextran against a destroyedIRuntimeCacheand segfaulted.Why
TRTRuntimeConfigis documented as owning the engine-implicitRuntimeCache, but had no slot for it —_apply_settingsbuilt it as a local withautosave_on_del=True, so it was collected the moment the method returned, taking the liveIRuntimeCachewith it.TorchTensorRTModulepre-wraps path strings into a module-owned handle, so engines reached throughcompile()never take this branch. Engines built straight from packed engine info do: they carry the defaultRuntimeSettings(runtime_cache=RUNTIME_CACHE_PATH)with no module around them to resolve it. As a side effect the disk cache was never populated on this path — autosave fired before the engine had generated a kernel.How
ensure_cache()returnsNonewhen the facade is torchbind-backed. No library path reaches that (every in-libraryTRTEngineconstruction site is gated on the same flag that picks the backing), but a hand-built engine on a build shippinglibtorchtrt.sodoes, andset_runtime_cacherejects an explicitly-passedNonedespite advertisingcache: IRuntimeCache = None. Skip the attach and warn instead of surfacing a pybind signature error; the two duplicated call sites fold into one helper.reset()when the path is unchanged so a settings swap does not discard in-memory kernels. Mirrors whatTorchTensorRTModule._resolve_runtime_cachealready does for module-owned handles.TestPythonRuntimeAliasedIOis skipped when the C++ runtime is available: it hand-builds a Python engine, which is a configuration the library never produces there. It stays live on python-only builds, matching the existing guard intest_000_runtime_cache.pyandtest_001_dynamic_shapes_kernel_strategy.py.Testing
tests/py/dynamo/runtime/createExecutionContextThe crash is caught by
test_bare_engine_executes_with_default_path_cache. Onmain,TestPythonRuntimeAliasedIOalready fails on python-only TRT-RTX for the same root cause (Failed to create execution context), so this is pre-existing rather than introduced here.Type of change
Checklist