Skip to content

Build and publish CUDA wheels - #21668

Open
shoumikhin wants to merge 75 commits into
gh/shoumikhin/95/headfrom
gh/shoumikhin/96/head
Open

Build and publish CUDA wheels#21668
shoumikhin wants to merge 75 commits into
gh/shoumikhin/95/headfrom
gh/shoumikhin/96/head

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

The problem

The wheel can carry the CUDA delegate, but no published wheel contains one: there is no CUDA
row in any workflow, so a GPU user has to build from source.

The change

Add the workflows that build and publish CUDA wheels for Linux x86_64 and aarch64, and a smoke test
that checks each wheel from the artifact itself. The aarch64 rows are validated on machines with no
GPU, so for those the smoke test cannot execute a model. It verifies the CUDA libraries are present,
that the declared runtime matches the wheel's CUDA version, that nothing resolves through the build
machine's toolkit, and that the shipped device code covers every GPU architecture the row claims.

executorch-1.5.0-cp312-cp312-manylinux_2_28_x86_64.whl    +cu130

A release publishes CUDA 12.6, 13.0 and 13.2, for Python 3.10 through 3.13. A pull request builds a
single row instead of all twelve, because a full matrix costs hours for little extra signal.

Which GPU architectures each row compiles for is chosen per row rather than detected on the builder.
Detecting it would produce a wheel carrying device code for whatever machine happened to build it,
which installs fine and then fails at the first GPU call.

The aarch64 CUDA 12.6 row also compiles for compute capability 8.7, which is an embedded module.
Every other row lists only the architectures the published PyTorch build for that train covers, and
by that rule 8.7 would be left out, because the generic aarch64 build of this train carries 8.0 and
9.0 only. It is included because this is the only row whose CUDA major version matches what that
module's software release ships, and because this wheel declares no PyTorch dependency: a user there
supplies the build that carries their architecture. Leaving 8.7 out does not protect them from a bad
pairing, it only removes the device code they need. Without it, a model reaching one of the shipped
optional operators, quantized matrix multiply, sort or random number generation, fails at the first
launch on that device.

Two guards keep a release honest:

  • if the shared matrix generator stops offering a combination this policy advertises, the step fails
    instead of quietly publishing fewer wheels. A missing job is otherwise a green check for a wheel
    that was never built.
  • if a row reaches the architecture list with no CUDA version, the build refuses rather than falling
    back to the builder's GPU. A TORCH_CUDA_ARCH_LIST that holds only named GPU families PyTorch
    accepts, such as "Hopper", is left to torch instead: those entries are dropped with a message and
    CMAKE_CUDA_ARCHITECTURES stays unset, because torch has already turned them into gencode flags by
    the time this runs and failing here would break a configuration that builds today. The three
    named forms CMake itself understands, "native", "all" and "all-major", are rejected before this
    logic runs:
    torch resolves the list with its own bundled CUDA architecture module, which does not know
    those names and stops the configure. That is upstream behaviour, not something this change
    introduces or can work around, so a caller has to name architectures explicitly.

Windows CUDA is deliberately absent. The separate shared libraries this wheel exists to ship are
Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not
link.

Test plan

  • built the full release matrix, twelve wheels, and confirmed each one's contents match the row it
    claims: the CUDA libraries present, the CUDA runtime declared, and device code for every GPU
    architecture the row advertises.
  • ran a GPU model end to end from a CI-built wheel on three NVIDIA GPUs covering three device
    architectures, with output identical to eager PyTorch on each (largest absolute difference 0),
    and inspected the wheel for a fourth device it cannot execute on.
  • ran the matrix filter over generated inputs, including incomplete and malformed ones, and
    confirmed it refuses rather than publishing a partial release: a missing CUDA version, a missing
    python, or a python present on rows this policy does not build are each reported by name.
  • confirmed a CPU row still produces a CPU wheel on a builder that happens to have a CUDA toolkit
    installed.
  • the newest architecture also ships in its portable form, so a GPU newer than any in the row can
    still run by having the driver compile it at load time. Checked with cuobjdump --list-ptx, since
    --list-elf prints identical output whether or not the portable form is present.
  • every library that carries GPU device code covers the whole row on its own.
  • the declared CUDA packages are compared against the expected set in BOTH directions. A one-way
    comparison accepted a wheel that omitted required packages, and a name-suffix comparison accepted
    cross-train names because for CUDA 13 the suffix is empty.
  • the python axis is an allowlist, matching the CUDA axis. Testing only the disabled list let any
    python not on it through: a 3.9 row was emitted successfully.
  • install_utils.py is in both CUDA workflows' path filters. It owns the supported CUDA train list
    and the toolkit detection, so a change there previously ran no CUDA wheel job.
  • the headers the wheel ships as its C++ SDK are in both filters too, for the same reason and
    matching the CPU workflows. extension/cuda is already covered by the CUDA-specific entry, but
    the rest of the SDK is not, and a CUDA wheel publishes all of it.
  • requesting the JetPack rows fails with its own reason instead of the generic empty-matrix message,
    since both of its lists are deliberately empty and no workflow asks for them.
  • torchao keeps its CUDA channel where that channel exists. Falling back to the plain nightly index
    was needed only on aarch64, where the CUDA channel publishes nothing, and doing it everywhere
    changed which torchao an x86_64 install resolves.
  • the CUDA smoke test now asserts the QnnBackend and OpenvinoBackend registrations that a CPU Linux
    row asserts. The CUDA build enables OpenVINO on every Linux architecture and downloads the QNN
    SDK on x86_64, so a CUDA wheel carries both backends; a previous premise that "a CUDA row is not
    built with them" was false, and dropping the checks meant those two backends were unverified on
    every CUDA wheel.

The x86_64 rows land on a GPU runner, so the smoke test there exports a model through the delegate,
runs it and compares against eager. The aarch64 rows have no accelerator, so that one check reports
a skip naming the reason and the row still verifies the shipped libraries, their declared
dependencies, their loader paths and the device code they carry.

@pytorch-bot

pytorch-bot Bot commented Aug 7, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21668

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure, 4 Unclassified Failures

As of commit 6c5de5d with merge base ed2d9ec (image):

NEW FAILURE - The following job has failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 7, 2026
shoumikhin added a commit that referenced this pull request Aug 7, 2026
## Why

The wheel now ships a CUDA delegate, but no release builds one. Every wheel workflow says:

    with-cuda: disabled

So a user who wants GPU support still clones the repository and builds from source, which is
the thing the shipped libraries were supposed to remove. Nothing publishes them because
nothing builds them.

## What this change does

Adds two workflows, one per architecture, that build CUDA wheels the same way the CPU ones
build the default wheel:

    build-wheels-cuda-linux.yml            x86_64
    build-wheels-cuda-aarch64-linux.yml    aarch64

Both call the same shared matrix generator the CPU rows already use, with CUDA turned on
instead of off, then narrow the result.

## Which rows get built, and why not all of them

The generator emits every CUDA version it knows about. Publishing all of them would ship
wheels for combinations nothing can verify, and a GPU wheel that installs and then cannot run
is worse than one that does not exist: the failure appears when a model runs, and it looks
like a model problem rather than a packaging one.

So a row is kept only when all three of these hold:

| | |
| --- | --- |
| a GPU exists that the row's device code covers | otherwise the wheel installs and dies at the first kernel launch |
| a PyTorch build is published for that CUDA version and architecture | otherwise the dependency cannot be satisfied |
| a machine is available to run a model before release | otherwise nothing checks it |

That leaves:

| architecture | CUDA | Python |
| --- | --- | --- |
| x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
| aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |

The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on,
rather than by what is convenient to verify here. A delegate built against one CUDA version
needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer
with nothing to pair with, which fails for whoever installs the pair rather than for the row
that omitted it. 13.2 is published for that reason even though no machine on hand can execute
it: the packaging properties are checked on every row, and running a model is a release-gate
step on hardware with the matching GPU.

Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated
reason in the example requirements, so a GPU row would inherit a known-broken build. The
free-threaded builds are excluded because the CUDA dependencies are not published for them.

A pull request builds one representative row rather than the whole matrix, because a full
matrix on every push costs hours for little added signal.

## Jetson devices

Jetson needs its own row: a JetPack container, one Python version, one CUDA version. It cannot
take a generic aarch64 wheel, because the generic builds carry no device code for its GPU
architecture and no portable fallback either.

That row is present in the filter but deliberately empty. Published PyTorch stopped shipping
device code for those GPUs after 2.8.0, so a Jetson row today would produce a wheel whose
PyTorch dependency cannot execute on the device. The lists are there to be filled in when that
changes.

## What to expect

Nothing changes for a CPU user. These are additional rows, and the existing workflows are
untouched.

| | before | after |
| --- | --- | --- |
| GPU support from an install | build from source | a published wheel |
| CUDA runtime | not shipped | declared as a dependency |

The build asks for the delegate explicitly rather than letting the build detect a toolkit. A
detected build is fine locally, but a release row states what it is producing, and a row that
silently produced a CPU wheel because the toolkit was missing would publish under a CUDA name.
The environment script fails early for the same reason: without it, packaging looks for CUDA
libraries that were never built and reports a confusing missing-file error minutes later.

Test plan:

A smoke test for the CUDA rows, checking what an artifact can be held to on a builder that has
no GPU:

- the CUDA libraries are actually in the wheel, so a row named for CUDA cannot ship without a
  delegate
- the CUDA runtime is declared, so a user has something to resolve it from
- each CUDA library reaches its runtime through a relative path, not through the builder's
  toolkit directory, which resolves on the builder and nowhere else

Then everything a CPU wheel is already held to: one owner per component, no build-tree search
paths, and a C++ application outside the wheel able to link what it ships.

Each CUDA check was run against a wheel that should fail it as well as one that should pass,
because a check that cannot fail is worse than no check. All three correctly reject a CPU
wheel and all three accept a CUDA wheel built from source, which ships both libraries, declares
the runtime, and carries a relative path to it.

One of them did not fail on a CPU wheel at first: it looped over libraries that were not there
and reported a pass, having inspected nothing. It now requires at least one to be present.

The filter was exercised against a matrix shaped like the generator's output: 18 rows narrowed
to 8, a pull request narrowed to 1, the aarch64 rows given the newer builder image, and an
empty result treated as a failure rather than passed through, since a workflow with no build
job reads as a green check for a build that never happened.

ghstack-source-id: 10f5862
ghstack-comment-id: 5220374521
Pull-Request: #21668
shoumikhin added a commit that referenced this pull request Aug 7, 2026
## The problem

The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says:

```
with-cuda: disabled
```

So a user who wants GPU support still clones the repository and builds from source, which is the
thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds
them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on.

## The change

Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel,
calling the same shared matrix generator with CUDA turned on, then narrowing the result.

```
.github/workflows/build-wheels-cuda-linux.yml            x86_64
.github/workflows/build-wheels-cuda-aarch64-linux.yml    aarch64
```

| architecture | CUDA | Python |
| --- | --- | --- |
| x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
| aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |

The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather
than by what is convenient to verify. A delegate built against one CUDA version needs an
ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing
to pair with, which fails for whoever installs the pair rather than for the row that omitted it.

Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason,
so a GPU row would inherit a known-broken build. A pull request builds one representative row
rather than the whole matrix.

Jetson devices need their own row, with a different container and a single Python and CUDA
version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty:
published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a
wheel whose PyTorch dependency cannot execute on the device.

## Before and after

```
BEFORE                                  AFTER

pip install executorch                  pip install executorch
  CPU only, always                        CPU by default
                                          GPU wheels published per CUDA version

GPU support means cloning the           a published wheel carries the delegate
repository and building

a delegate built for CUDA 13.2          every consumer CUDA version has a
has no ExecuTorch wheel to pair with    matching ExecuTorch wheel
```

## Test plan

A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU:

- the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate
- the CUDA runtime is declared, so a user has something to resolve it from
- each CUDA library reaches its runtime through a relative path, not the builder's toolkit
  directory, which resolves on the builder and nowhere else

Then everything a CPU wheel is already held to: one owner per component, no build-tree search
paths, and a C++ application outside the wheel able to link what it ships.

Each check was run against a wheel that should fail it as well as one that should pass, because a
check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel
built from source. One did not fail at first: it looped over libraries that were not there and
reported a pass having inspected nothing, so it now requires at least one to be present.

The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to
24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty
result treated as a failure rather than passed through, since a workflow with no build job reads as
a green check for a build that never happened.

ghstack-source-id: bdf89eb
ghstack-comment-id: 5220374521
Pull-Request: #21668
shoumikhin added a commit that referenced this pull request Aug 7, 2026
## The problem

The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says:

```
with-cuda: disabled
```

So a user who wants GPU support still clones the repository and builds from source, which is the
thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds
them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on.

## The change

Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel,
calling the same shared matrix generator with CUDA turned on, then narrowing the result.

```
.github/workflows/build-wheels-cuda-linux.yml            x86_64
.github/workflows/build-wheels-cuda-aarch64-linux.yml    aarch64
```

| architecture | CUDA | Python |
| --- | --- | --- |
| x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
| aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |

The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather
than by what is convenient to verify. A delegate built against one CUDA version needs an
ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing
to pair with, which fails for whoever installs the pair rather than for the row that omitted it.

Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason,
so a GPU row would inherit a known-broken build. A pull request builds one representative row
rather than the whole matrix.

Jetson devices need their own row, with a different container and a single Python and CUDA
version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty:
published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a
wheel whose PyTorch dependency cannot execute on the device.

## Before and after

```
BEFORE                                  AFTER

pip install executorch                  pip install executorch
  CPU only, always                        CPU by default
                                          GPU wheels published per CUDA version

GPU support means cloning the           a published wheel carries the delegate
repository and building

a delegate built for CUDA 13.2          every consumer CUDA version has a
has no ExecuTorch wheel to pair with    matching ExecuTorch wheel
```

## Test plan

A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU:

- the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate
- the CUDA runtime is declared, so a user has something to resolve it from
- each CUDA library reaches its runtime through a relative path, not the builder's toolkit
  directory, which resolves on the builder and nowhere else

Then everything a CPU wheel is already held to: one owner per component, no build-tree search
paths, and a C++ application outside the wheel able to link what it ships.

Each check was run against a wheel that should fail it as well as one that should pass, because a
check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel
built from source. One did not fail at first: it looped over libraries that were not there and
reported a pass having inspected nothing, so it now requires at least one to be present.

The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is
treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed
run rather than a clear error. The checks now assert the accepted spelling.

The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to
24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty
result treated as a failure rather than passed through, since a workflow with no build job reads as
a green check for a build that never happened.

ghstack-source-id: df65c37
ghstack-comment-id: 5220374521
Pull-Request: #21668
shoumikhin added a commit that referenced this pull request Aug 7, 2026
## The problem

The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says:

```
with-cuda: disabled
```

So a user who wants GPU support still clones the repository and builds from source, which is the
thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds
them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on.

## The change

Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel,
calling the same shared matrix generator with CUDA turned on, then narrowing the result.

```
.github/workflows/build-wheels-cuda-linux.yml            x86_64
.github/workflows/build-wheels-cuda-aarch64-linux.yml    aarch64
```

| architecture | CUDA | Python |
| --- | --- | --- |
| x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
| aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |

The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather
than by what is convenient to verify. A delegate built against one CUDA version needs an
ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing
to pair with, which fails for whoever installs the pair rather than for the row that omitted it.

Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason,
so a GPU row would inherit a known-broken build. A pull request builds one representative row
rather than the whole matrix.

Jetson devices need their own row, with a different container and a single Python and CUDA
version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty:
published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a
wheel whose PyTorch dependency cannot execute on the device.

## Before and after

```
BEFORE                                  AFTER

pip install executorch                  pip install executorch
  CPU only, always                        CPU by default
                                          GPU wheels published per CUDA version

GPU support means cloning the           a published wheel carries the delegate
repository and building

a delegate built for CUDA 13.2          every consumer CUDA version has a
has no ExecuTorch wheel to pair with    matching ExecuTorch wheel
```

## Test plan

A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU:

- the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate
- the CUDA runtime is declared, so a user has something to resolve it from
- each CUDA library reaches its runtime through a relative path, not the builder's toolkit
  directory, which resolves on the builder and nowhere else

Then everything a CPU wheel is already held to: one owner per component, no build-tree search
paths, and a C++ application outside the wheel able to link what it ships.

Each check was run against a wheel that should fail it as well as one that should pass, because a
check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel
built from source. One did not fail at first: it looped over libraries that were not there and
reported a pass having inspected nothing, so it now requires at least one to be present.

The rows keep whichever builder image the release provides. Substituting a plain manylinux image
broke the shared build setup, which expects conda to be present in that image.

The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is
treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed
run rather than a clear error. The checks now assert the accepted spelling.

The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to
24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty
result treated as a failure rather than passed through, since a workflow with no build job reads as
a green check for a build that never happened.

ghstack-source-id: 0f17009
ghstack-comment-id: 5220374521
Pull-Request: #21668
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main
with:
package-type: wheel
os: linux-aarch64

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have CI machines to test arm + cuda?

[ghstack-poisoned]
model_tests=[
test_base.ModelTest(
model=Model.Mv3,
backend=Backend.XnnpackQuantizationDelegation,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not AOTI-CUDA?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we also make sure the AoT side of things also install the right version of torch when someone does 'pip install executorch' which gets these cuda libs, they better work with AoT torch cuda wheel generated AOTI blobs which uses cubin and other cuda runtime APIs.

@digantdesai digantdesai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stamping to unblock, left some comments.

[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/binaries/all Release PRs with this label will build wheels for all python versions ciflow/binaries ciflow/cuda ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants