Track the ExecuTorch pin and move it to new nightlies automatically - #4566
Open
shoumikhin wants to merge 15 commits into
Open
Track the ExecuTorch pin and move it to new nightlies automatically#4566shoumikhin wants to merge 15 commits into
shoumikhin wants to merge 15 commits into
Conversation
github-actions
Bot
added
component: tests
7 tasks
Contributor
Author
|
Follow-up that depends on this: #4567 (ships only the TensorRT delegate in the runtime wheel). That change needs the prebuilt ExecuTorch runtime this pin brings in, so it should land after this one. |
shoumikhin
force-pushed
the
executorch-pin-dev-line
branch
2 times, most recently
from
August 23, 2026 18:07
2f5c384 to
7666e2a
Compare
shoumikhin
marked this pull request as ready for review
August 23, 2026 19:10
github-actions
Bot
added
the
documentation
shoumikhin
force-pushed
the
executorch-pin-dev-line
branch
from
August 23, 2026 19:30
598939b to
1eb0a46
Compare
shoumikhin
marked this pull request as draft
August 23, 2026 19:31
shoumikhin
marked this pull request as ready for review
August 23, 2026 19:57
shoumikhin
force-pushed
the
executorch-pin-dev-line
branch
2 times, most recently
from
August 24, 2026 00:05
f05e654 to
a3a0360
Compare
lanluo-nvidia
added
the
ci: nightly
shoumikhin
force-pushed
the
executorch-pin-dev-line
branch
4 times, most recently
from
August 25, 2026 18:48
3f8e367 to
5d051f1
Compare
…nce runner gate
The ExecuTorch gate exports one program, x + 1, and TensorRT takes it whole. So
nothing in CI ever runs a program where TensorRT and ExecuTorch's own CUDA
backend each own part of the same graph. That coalesced case is the whole point
of combining the two backends, and it is not covered end to end today. There is
a composition test that checks both delegates land in the file, but it never
loads or runs the program.
This adds the missing run.
A new example, examples/torchtrt_executorch_example/export_coalesced.py, exports
cos(erfinv(tanh(x))). TensorRT has no converter for erfinv, so a CudaPartitioner
catch-all gives that operator to the CUDA backend while TensorRT keeps the rest.
The script fails if the saved .pte does not carry both a TensorRTBackend and a
CudaBackend delegate, so a partitioning change cannot quietly turn this into a
TensorRT-only run that still passes.
The script also writes <model>.expected next to the .pte, holding the output
shape and the eager reference value for an all-ones input. Both reference
runners fill inputs with 1.0 and this model is elementwise, so one number
describes the whole expected output. Reading it from a file, instead of
hard-coding a number in the shell script, keeps the expectation tied to the
model.
verify-executorch-reference-runner.sh now takes an optional third argument, the
coalesced .pte. When given, it runs both the CMake-built runner and the packaged
runner on it and compares every printed value against that reference. TensorRT,
AOTInductor and eager PyTorch use different kernels for the same math, so the
comparison uses a tolerance of 0.001 rather than matching printed digits.
The existing x + 1 assertions keep the same strength. They now go through the
same helper with a zero tolerance, because x + 1 on ones is exact in float32.
Usage:
python examples/torchtrt_executorch_example/export_coalesced.py \
--model_path=coalesced.pte
.github/scripts/verify-executorch-reference-runner.sh \
model.pte kv_cache_decode.pte coalesced.pte
Test plan
On a Linux x86_64 host with an NVIDIA A100 GPU:
- Ran export_coalesced.py. It reported delegates
['TensorRTBackend', 'CudaBackend', 'TensorRTBackend'] and wrote "[64,64]" and
"0.6722" into the .expected file.
- Ran the resulting .pte through the reference runner. It printed
"output[0] shape=[64,64]" and first 8 values of 0.6722, an exact match to the
eager result.
- Deleted the aoti_cuda_blob.ptd that the CUDA backend writes and ran again.
Same output, so this model needs no external weight file.
- Exercised the new shell assertion helper against captured runner output:
correct output passes; one wrong value fails; a wrong shape fails; a missing
values line fails; a value inside the tolerance passes and one outside it
fails.
- shellcheck, bash -n, black and isort are clean on the changed files.
Not yet observed in CI: the ExecuTorch runtime build job currently fails on main
when the packaged reference runner aborts on the existing x + 1 model, and the
test job is skipped while that is true. Both happen before this new code runs.
ExecuTorch 1.4.1 ships no linkable C++ runtime: its wheel contains zero shared libraries, its CMake package exports only a static _portable_lib, and no CUDA wheel exists for it on any channel. That is why the runtime wheel rebuilds ExecuTorch from source today, and it is the blocker for shipping only the TensorRT delegate. The prebuilt runtime landed on ExecuTorch main on 2026-08-20, six days after 1.4.1 was tagged, so no release carries it yet. Move the pin to the nightly line that does, keeping the release-line range on installable metadata so the same range prefers 1.5.0 over any dev build the day it ships, with no edit needed. The two pins now have to name one ExecuTorch rather than two that look close, because the delegate compiles headers from the source tree and links the runtime out of the wheel. Every wheel records its source commit, so add a test asserting the pinned commit is the pinned wheel's own git_version. Nothing else was enforcing that, and a mismatch is silent: both pins look plausible and the build succeeds. Deriving the range with a three-field split raised on the nightly form, so derive it from the release line the first two fields name. ExecuTorch's CUDA wheels are published only on the PyTorch nightly index, so every install site gains that channel. No site gains --pre: the pin names an exact dev build, which pip installs from an explicit version without it, and the existing --pre uses here are for torch. CI derives the channel from the row's own CU_VERSION, which keeps the runtime the delegate links to the same CUDA build as the rest of the job.
The CI installer globs torch_tensorrt*.whl, which also matches the ExecuTorch runtime wheel, whose install_requires names a dev build published only on the nightly channel. With no index on that line the whole pip invocation failed, and because line 1's set -e is commented out the failure was swallowed and the job died later with a confusing ImportError. The two range sites installed a range against the nightly channel, which gains a member every day, so they resolved to whatever was newest while the delegate is compiled from the commit the pin names. Both now request the pin exactly, which is the pairing the drift test exists to check; it was written to skip in exactly the state the ranges produced, so nothing reported it. setup.py keeps its range, because a published requirement has to stay resolvable for users off the same line. The two shapes now differ deliberately and test_derived_requirements_match_the_pin checks each for its own. Six printed install instructions gave a bare pip install of the executorch extra, which cannot resolve a dev pin from PyPI. They name the channel now. The discovery regex saw only == and >=, so a site added with any other PEP 440 operator was invisible to the drift check. It now recognises all of them.
The executorch suite is nightly-only, and the nightly matrix runs cu132 rows as
well as cu130 ones, so the fixed cu130 channel in tests/ci/runner.py would
install a CUDA 13.0 ExecuTorch into a CUDA 13.2 job. PR builds are pinned to
cu130 by filter-matrix.py, which is why watching PR CI could never show this.
Derived from CU_VERSION now, with cu130 as the local default, matching what the
two workflow files already did.
Three example docstrings still printed a bare `pip install -e ".[executorch]"`.
That resolved off PyPI before this pin moved to a dev build; it cannot now, so
they name the nightly index too. The runtime's ImportError advice and the
reference runner README already did.
The installer's hardcoded nightly channel gets the reason written down: a .dev
wheel exists on no other channel, so deriving it from ${CHANNEL} like the lines
above would break the install on exactly the test and release runs the index was
added for.
…alls from Raising the executorch floor to a dev build made `uv lock` fail outright. The lock's win32 required-environment in pyproject.toml resolved the extra from PyPI, whose executorch stops at 1.4.1 -- uv.lock records five win_amd64 wheels for it -- so nothing satisfied the new range and uv errors rather than falling back. Reproduced against a probe project: without a marker uv reports the win32 split unsatisfiable, with one it resolves. The requirement now carries `platform_system == 'Linux'`, the shape EXECUTORCH_RUNTIME_REQUIREMENT already uses, which also stops pip reporting no matching distribution for Windows users of the extra. The delegate is a Linux object and ExecuTorch publishes CUDA wheels for no other platform, so the marker states what was already true. docgen installed the extra with --pre against the nightly channel, so it resolved through the range and took whichever dev build was newest that morning while the delegate compiled from the pinned commit. It names the pin now, read out of dev_dep_versions.yml. The pip line that installs both wheels gets `|| exit 1`. linux-test.yml concatenates this installer ahead of the user script and line 1's `set -e` is commented out, so a failure there was discarded and the job died later with an unrelated-looking ImportError; measured with `false` in place of the pip call, exit was 0 and the user script still ran. Two tests were checking source text rather than behaviour. The CUDA-row test now calls _setup_commands with CU_VERSION set and unset and reads the URL, which catches keeping the os.environ.get line while hardcoding the channel -- the mutation the string match passed. The drift check now asserts the set of files that pin ExecuTorch, because a site changing to bare `executorch` stops matching the search entirely and left the old `assert found` satisfied. Also corrects two claims: 1.4.1 does ship _portable_lib.so, so the comment says its executorch/lib carries no standalone linkable runtime, and no install site gains --pre, since an exact .dev pin needs none.
…tup fails The reference-runner README and the runtime's ImportError advice both printed `pip install "torch-tensorrt[executorch]"` with no index, and the commit that introduced the pin claimed otherwise. That claim was checked against the wrong branch: the fix existed only on the stacked runtime-wheel change, so this branch kept shipping the bare command. It matters more here than a docs nit, because this branch is what raises the floor above PyPI's newest executorch, so the bare command now cannot resolve at all. All seven printed install instructions carry the channel. The drift checks were counting the wrong thing. The requirement test asserted a set of paths, but two files carry two sites each, so either could drop one and stay in the set: turning `executorch-build-linux.yml:88` or `:128` into bare `executorch` both survived. The commit test only asserted nonzero, so any single MODULE.bazel could switch to `branch = "nightly"` unnoticed. Both now assert a per-file site count through one helper, as a minimum rather than an exact number so it holds on the stacked branch too, which removes one README site. All five mutations are caught and each names the file. Counting also surfaced a fifth commit site the nonzero check could not see: the reference-runner README's EXECUTORCH_REF shell default, correctly pinned but unaccounted for. docgen's pin was invisible to both: it is built by a shell substitution, so `$(` is not a digit and the literal search never saw it, and deleting the line survived. The derived-requirement test now runs the command docgen embeds and compares what it prints. A failed setup step printed `::warning::` and fell through to pytest. Most of the executorch suite gates on pytest.importorskip, so a failed ExecuTorch install skipped those files, left the rest passing, and reported success with a populated junit xml -- green exactly when the suite could not test what it exists to test. Driving the real run_suite with a failing setup step reproduced it, and returning the code makes it red without invoking pytest. Pre-existing, but this branch makes it likely to fire, since a nightly pin is eventually pruned from the channel. The pin tests themselves ran on nightly only, so none of this drift machinery ran on a PR or a push to main -- when a pin actually goes stale. They need no GPU, no ExecuTorch and not even torch, so they move to their own l0 suite in every lane, and the nightly suite excludes them by keyword so nothing runs twice. Also: the reference-runner README no longer says the extra installs the runtime wheel, since that requirement is commented out in setup.py.
…U lane The drift checks read derived strings and never the values CI consumes, so three ways of silently shipping no ExecuTorch all stayed green. Dropping the requirement from the runner's setup command left the step succeeding with nothing installed, after which the suite skips on importorskip; emptying EXTRAS_REQUIRE["executorch"] broke every documented `pip install "torch-tensorrt[executorch]"`; and the runtime README was recorded as carrying one pin site when it carries two, so either could go bare while the other satisfied the count -- the exact hole the per-file counts were added to close. The checks now assert the argument list the runner builds, the extras entries by AST, and the true per-file counts. All five mutations fail now. run_suite had no test at all, so replacing its `return rc` with `continue` restored the silent-green behaviour the fail-closed change exists to prevent. It is driven directly now, asserting both the propagated exit code and that pytest never runs once setup has failed. The pin suite was landing on a GPU runner: Suite.runner defaults to the matrix validation runner, so a five-second text check became one CUDA-container job per python and CUDA row, behind a wheel build. It runs in the Python lint job instead, which is already ubuntu-latest and needs none of that. The claim that it needs "not even torch" was also wrong -- tests/py/dynamo/conftest.py imports torch at module scope, which is why the lint invocation passes --noconftest. The shell tier that runs the whole executorch directory now excludes the pin file too, so the dedup claim is true of both paths rather than just the manifest one. uv.lock still records the pre-bump range with no platform marker. uv-update.yml regenerates it on pushes to main touching setup.py, and only that workflow runs `uv sync --locked`, so this breaks nothing -- but the drift was invisible, since the lock writes a bare specifier the pin search cannot match. A strict=False xfail records it and turns into a real failure via XPASS once the lock is refreshed. Editing the lock by hand was the wrong fix: its resolved entry and hashes come from a resolver run against the nightly index. Also removes internal shorthand from the PR description, and corrects a line citation for the one deliberate range in executorch-build-linux.yml.
The lint step added for these checks could not execute. It invokes pytest, and the job installs .github/scripts/requirements.txt (PyGithub) plus the lint dependency group (black, clang-format); neither carries pytest, so the step exited 1 on "No module named pytest" before running a single assertion. pyyaml is needed too, because reading the pin file shells out to a yaml import. Both are installed now, and a test asserts the step exists and installs them, since deleting it is otherwise invisible: every assertion here still passes locally while nothing runs it on a pull request. Reproduced the failure in a stdlib-only venv and confirmed the fixed command passes with only those two. The step also gets if: always(), so an unrelated formatting failure earlier in the job no longer hides the pin check. Three properties the checks are supposed to protect had no coverage: Deleting both published extras from EXTRAS_REQUIRE left everything green. The loop iterated whatever keys existed, so removing them iterated nothing and was indistinguishable from them being correct. It now requires the two published keys to be present, and only those, which also stops an unrelated future extra from turning this red for naming no ExecuTorch. The workflow opt-out marker was ordinary prose, "verify the end user's workflow". Pasting that sentence above a requirement and widening it to a range passed. It is an explicit token now, and the upward scan walks through comment lines to find it, so a cosmetic line between the opt-out and the requirement neither reclassifies the site nor fails the build. Nothing asserted that printed install instructions name the nightly channel, which is why that regressed and was re-fixed three times in this change without anything noticing. One test covers all of them by reading whole blocks rather than single lines, since every instruction wraps and the index lands on a continuation. It catches the CI install of the locally built wheel too, which carries no extra and is the site that broke most often. Generated docs under docs/ are excluded: corrections belong in docsrc/, and the committed Sphinx output is stale there independently. Also: the executorch requirement now strips its local version label like the other four, so the wheel does not bind itself to one CUDA train; the lockfile xfail is strict, since a non-strict xfail reports XPASS and ignores it and so could never fail; the fail-closed comment says it covers every setup step rather than implying only executorch; an empty frozenset and the dead branch reading it are gone; and the sys.path mutations use monkeypatch so they do not leak between tests.
The check that the two pins name one ExecuTorch could not run anywhere. It skips unless the installed wheel is exactly the pinned version, so it means something only on the nightly GPU lane, and that lane deselected it. The deselection is written as "not test_executorch_pin" to skip the source-consistency checks in the same file, but -k matches the module name in the test id, so it dropped every test in the module including this one. Both deselection sites now keep it by name. Proved it on a host with the pinned wheel installed, whose recorded git_version is the pinned commit: the check passes at the correct pins, fails when the commit pin names a different tree, and fails when the commit pin is deleted outright. Before this it was deselected in all three states. Bumping the version alone still skips, correctly, because the installed wheel is then not the one the pin names and its provenance says nothing about whether the two pins agree. A test asserts both sites keep it, since re-tightening either one to a bare module name is a small and plausible edit that would silently restore the gap.
Every guard added in this change asserted that a string appeared somewhere in a file, so each certified the state it was written to prevent. The keyword guard grepped for the kept test's name. Changing "or" to "and" in both -k expressions left it green, and that expression collects nothing at all, which is worse than the bug the guard exists to catch. Reverting the expressions and leaving the name behind in a comment also left it green, and a comment explaining the keyword sits directly above it, which is where an editor would naturally write that name. It now runs pytest's own collection under each expression and requires exactly the pairing test to come back. The CI guard searched the workflow as one blob, so it could not tell which job it was reading. The same commit that fixed the lint failure also added pytest and pyyaml to cpp-linting, which has no pin check, so deleting them from the job that does run it stayed green and would have restored the original failure invisibly. Neutralising the command while leaving its filename in a shell comment, and setting a falsy step condition, were also green. It now parses the workflow, finds the job that actually invokes pytest on this file, and requires the installs in an earlier step of that same job. The unused installs are gone from cpp-linting. The requirement pattern captured an equality prefix and stopped, so "executorch==PIN,!=PIN", a specifier that excludes the version it appears to pin, compared equal to the pin. The same truncation rejected the legal PEP 508 spelling with spaces around the operator. Requirements are parsed now and compared as specifier sets, with a check that the pinned version actually satisfies them. The site scanner counted raw search hits, so gutting a pin to a bare "executorch" while putting the exact pin in a comment in the same file kept the per-file minimum satisfied. Comments no longer count, except in the bazel repositories, where the annotation beside the pinned commit is the only record of which wheel that commit belongs to. Also corrected two claims this change made: the executorch tier is reachable from a pull request through executorch-test-linux.yml as well as the nightly manifest, so it is not the only route, and the shell helper now says why one test is kept out of the deselection.
The uv.lock check was a strict xfail. uv.lock records ">=1.4.1,<1.5" while the pin derives ">=1.5.0.dev20260822,<1.6", so the assertion fails and the xfail is satisfied. Refresh the lock and the assertion passes, and a strict xfail reports that pass as a failure. The lint step runs this file with if: always() on every pull request, so one lock refresh would have made the lint job red on every subsequent pull request, for a file none of them touched, until someone edited this test. Measured: baseline 1 xfailed, and 1 failed once the specifier is bumped. My own docstring claimed the lock is machine-generated and not edited by hand. Two hand refreshes landed on 2026-08-23, inside ordinary version-bump changes, so that was wrong as well. It now accepts both resting states and only fails where something is actually wrong: a recorded range whose lower bound is above the pin, which means the lock names an ExecuTorch this repository does not pin. Behind the pin passes, the derived range passes, and ">=1.6,<1.7", an open-ended ">=1.7" and "==1.9.0" all fail. Comparing lower bounds rather than probing the specifier with sample versions: an upper-bound test missed the open-ended case, and a low sentinel version called the ordinary behind-the-pin state a failure.
test_derived_requirements_match_the_pin extracted the python3 -c one-liner from docgen.yml and ran it. Whatever that line said got executed on every pull request: rewriting it to write a file left the test green and the file written. Same class as the bash -c problem fixed in test_api.py last round, still live here. It now compares the command as text against the exact form that reads __executorch_version__ out of dev_dep_versions.yml. Four mutations caught, including a payload that writes a file and still prints the right version, with nothing executed. The CI reachability guard tested the raw string for "--collect-only", so it accepted "--co", pytest's own documented short form, which collects and asserts nothing. It also could not see an exit status being discarded. Now tokenised: --collect-only, --co, -h, --help, a "||" short-circuit and continue-on-error are all rejected, and all five are caught where four previously survived. The comment exemption for .md/.rst/.txt defeated exactly the threat its docstring names. Install commands live in prose files, so exempting them made a comment count as a pin there: the runtime README's install line gutted to a bare "executorch" passed as long as a decoy "# executorch==<pin>" sat beside it, and failed only with no comment present. The exemption is gone, and trailing comments no longer count either, since a decoy after a live requirement on the same line kept the per-file count satisfied. Five mutations caught, baseline green.
…it resolves The nightly-index guard matched only the named-distribution spelling, so the four sites that write "pip install .[executorch]" were unguarded: docgen.yml and the three export examples. The nightly index could be deleted from all four with the test green. Each of the four is now caught individually. Its second half was a bare substring test for the host, which proves a string sits nearby rather than that the instruction resolves. Rewriting every channel in the tree, 18 files, to a nonexistent cu999 left it green. The CUDA suffix is now checked against the set the project publishes for. Deliberately not compared against __cuda_version__: five sites legitimately say cu130 while the pin says 13.2, and I confirmed against the live index that cu130 and cu132 both carry 38 ExecuTorch wheels while cu999 carries none.
The printed install commands resolved no ExecuTorch. "torch-tensorrt[executorch]" with no version pin resolves the stable PyPI wheel, which carries no executorch extra, so the command exited 0 and installed nothing the feature needs. Add --pre to the six commands that name the extra and assert its presence in the guard that already reads them. Close four ways to neutralise the pin check while its guard stayed green: a ";" or "&" terminator after pytest, continue-on-error or a falsy if: on the owning job, and reducing the workflow trigger so it never runs on pull requests. The trigger check also handles PyYAML reading the unquoted "on" key as the boolean True. Close both ways to strip the pairing check while its guard stayed green: assert the workflow actually calls trt_tier_executorch, and validate suite lane names against the known set so a typo raises at import instead of silently dropping the suite from every matrix. Also: anchor the docgen pin check to a live line so a commented-out install no longer satisfies it; fix the lockfile range check crashing on a legal "==1.4.*" clause; correct the range comment to describe what the range admits; and note in the install advice that the feature is published for Linux only.
shoumikhin
force-pushed
the
executorch-pin-dev-line
branch
2 times, most recently
from
August 25, 2026 22:43
19b80c9 to
d8b7e43
Compare
shoumikhin
changed the title
shoumikhin
force-pushed
the
executorch-pin-dev-line
branch
from
August 25, 2026 23:08
d8b7e43 to
602cd36
Compare
The delegate is built against one ExecuTorch: __executorch_version__ selects the wheel it links against and __executorch_commit__ selects the tree it compiles from. Those two values repeat across the build workflows, the bazel modules, the docker and toolchain copies, and the docs, so they can drift apart or fall behind upstream with nothing to notice. Add a script and a daily workflow that move both pins to the newest ExecuTorch wheel on the nightly index. The source commit is read from the chosen wheel's own version.py, so the two pins always name one ExecuTorch rather than two that happen to be close. The update lands as a pull request, so the pin consistency checks and the delegate build and test lane decide whether the new wheel is usable before it reaches main. A day with no new nightly rewrites nothing and opens nothing. On a release branch the schedule is a no-op and the pin moves only by a manual run pointed at the stable line, so a cut release does not drift. Back the mechanism with consistency checks that run under the linter. Every requirement and comment that names ExecuTorch is asserted to match the pinned version, including the variable-index install once the variable's assignment is resolved and extensionless install files like justfile. The source commit is checked against the wheel's own provenance wherever that wheel is installed, and commits left in comments are not mistaken for pins. The wheel-content and CI-invocation checks measure effect, running the workflow's own step against a passing and a failing stub and requiring the exit status to follow, rather than enumerating bypass spellings. Install the built wheel in the runtime README rather than an unpublished package.
shoumikhin
force-pushed
the
executorch-pin-dev-line
branch
from
August 26, 2026 03:14
602cd36 to
93948e1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Torch-TensorRT has a small delegate that runs parts of a model with TensorRT inside ExecuTorch. To build that delegate, the project must point at one exact version of ExecuTorch. This is called the pin.
This change does two things:
executorch==1.4.1onto a recent ExecuTorch build.The problem
Problem 1: the old version cannot be used
The delegate is a shared library. It has to link against the ExecuTorch runtime, the C++ code that actually runs a model.
ExecuTorch 1.4.1 does not ship that runtime in a usable form:
_portable_lib.so, which is the Python extension, but no standalonelibexecutorch.so.So to link against 1.4.1, you had to rebuild all of ExecuTorch from source first. Recent ExecuTorch builds fixed this. They ship a real
libexecutorch.soand a CMake package you can link against. Moving the pin lets us use that.Problem 2: a pin goes stale fast
ExecuTorch ships a new build every night. A pin that someone bumps by hand falls behind, and then the delegate is tested against old code. Somebody has to remember to move it. That is the kind of chore that gets forgotten.
The solution
The pin is two values, not one
The delegate needs two things to agree:
__executorch_version__: which wheel to install and link against.__executorch_commit__: which source tree to compile the headers from.Both live in
dev_dep_versions.yml:If these two point at different ExecuTorch builds, the delegate can compile against one set of headers and link against a different runtime. That can break in quiet, confusing ways. So they must always name the same build.
Each ExecuTorch wheel records the commit it was built from. A test reads that value from the installed wheel and checks it equals
__executorch_commit__. Now the wheel and the source cannot drift apart. The test skips when ExecuTorch is not installed, so it does not block the pure-Python tests.A daily robot moves the pin
A script and a scheduled job move both values to the newest ExecuTorch nightly. The idea is simple:
git_versionfrom inside it. That is the same value the pairing test checks, so the two pin values always agree by construction.The script's own logic is tested in
test_update_executorch_pin.pyon the lint job. It checks the parts that fail in real life: pick the newest by date and not by longest string, refuse a wheel that has no recorded commit, and rewrite a real checkout that the pin guard then accepts.Checks that keep every pin site honest
The same version string is repeated in many files: workflows, Bazel modules, docker files, docs. A guard test runs on the lint job and holds all of them together:
A note on the install index
ExecuTorch's CUDA wheels live only on the PyTorch nightly index. So the install commands point there. A version like
1.5.0.dev...already allows pre-release builds, so those commands do not need--pre. Adding--prewould leak onto every other package in the same line. In CI the index is chosen from each row's own CUDA version:Type of change
Checklist
Testing
test_executorch_pin.py: 14 pass, 1 skips when ExecuTorch is not installed. On a Linux GPU host with the pinned wheel installed, the pairing test passes and correctly rejects the old 1.4.1 commit.test_update_executorch_pin.py: 12 pass. The script was run against the live nightly index. It moved the pin to the newest wheel, read that wheel's own source commit, rewrote every pin site, and left a tree the full guard accepts. Each part was also checked by breaking it on purpose: date ordering, the refusal of a wheel with no commit, and the version range bound all fail when their logic is removed. One test proves a bump updates a brand new pin site but never edits the script or its own test.Known gaps
uv.lockstill records the old ExecuTorch and is not regenerated here. The build does not read the lockfile, and a separate scheduled job refreshes it. The test that covers it accepts a lock behind the pin and only fails if the lock moves ahead of it.