Ship linkable libraries in the macOS wheel

[shoumikhin]

The macOS wheel shipped one fused Python extension and no linkable libraries, so a
C++ application got nothing from it: the headers and the CMake package were there, but
every component a consumer asked for resolved to nothing. On Linux the same wheel
ships the runtime, the kernels, the delegates, the thread pool and the profiler as
separate libraries, and a C++ application links them directly.

The three mechanisms that made this Linux only now have Mach-O equivalents:

runtime search path   $ORIGIN            loader_path
keeping a registration
only library linked   --no-as-needed     -force_load, already present
library identity      ELF soname         install name relative to rpath

Windows is still refused, because there the runtime carries no export annotations for
a DLL, which is a missing capability rather than a different spelling of one.

The packaging entries and the package config asked for a .so by name, so they would
have looked for a file the build never emits. Both now derive the suffix from the
platform, and the config also accounts for Mach-O putting the version before the
suffix, libfoo.1.dylib, where ELF puts it after, libfoo.so.1.

The C++ guide said the prebuilt libraries were a Linux thing, which this change makes untrue, so it
now names both platforms.

The two wheel test suites now run on macOS as well, which is what makes the split
verified rather than claimed: the Python extension links these libraries itself, so it
passes whether or not the package config names them or the shipped headers are
complete. Porting them needed a suffix helper, the Mach-O spelling of the symbol
queries, since nm -D asks for a dynamic symbol table that Mach-O does not have, and
otool in place of readelf.

Two of those checks asked for readelf by name and returned early when it was absent,
which on macOS is always, so the run reported success having examined nothing, and one
of them printed a check mark on the way out. They are the two that state the property
this change exists to create: that the extension records a dependency on each shipped
library rather than containing it, and that each library tells a consumer to record the
name it actually ships under. Both now ask the platform's own question instead, otool -L
for recorded dependencies and otool -D for the recorded identity, comparing the last
path component because Mach-O spells both as paths where ELF uses bare names.

Two more checks used to keep a documented skip there, on the grounds that ldd resolves
dependencies transitively while otool -L only lists recorded names. That compared the
wrong tools. The question is whether the loader can find what a library asks for, and
the way to ask it on macOS is to load the library: a dlopen that binds every symbol
answers what ldd -r answers on Linux. Both now do that, each load in its own process,
because dyld satisfies a request by install name from what is already loaded and would
otherwise let one library stand in for another's dependency, or hand back the original
file in place of the relocated copy under test. Between them they are the only checks
that prove a macOS consumer can load what this change ships, so the commit had none.

Two smaller readers were reporting on less than they inspected. The architecture check
skipped every file lipo could not parse, so a wheel it could parse none of would have
passed having compared nothing, and the symbol reader matched a definition by prefix,
which counts a longer symbol beginning with the same text as a second definer.

Test Plan: exercised both platform branches of every changed helper.

The CMake helpers, driven with real CMake and the Apple branch forced, so the macOS
answers are checked rather than assumed:

shipped runtime path   Linux $ORIGIN                  macOS loader_path
two entry path         Linux $ORIGIN/../../lib:...    macOS loader_path/../../lib;...
library identity       Linux version only             macOS install name rpath

The Python helpers, loaded under each platform:

suffix                 .so                            .dylib
defined symbols        nm -DC                         nm -gU -C
undefined symbols      nm -DC --undefined-only        nm -gu -C
library file name      libexecutorch.so               libexecutorch.dylib

The symbol queries and the load command reader were then run against a real Mach-O
library on macOS: 137 defined and 133 undefined symbols listed, and the load commands
read, including the dependency and search path entries and the torch dependency the
libtorch check looks for.

Linux is unaffected, verified by building a wheel from this change and confirming it
still ships the same six libraries.

The two load checks are the exception to that first line: they need a macOS wheel to
load, so the macOS wheel row is what proves them rather than a local run. The symbol
reader was rerun against a versioned symbol table to confirm the anchoring accepts a
complete name carrying a version suffix and rejects a truncation of it.

[shoumikhin]

Stack from ghstack (oldest at bottom):

• Name the Python extension after what it contains #21816
• -> Ship linkable libraries in the macOS wheel #21771
• Ship the OpenVINO delegate as its own library #21770
• Build and publish CUDA wheels #21668
• Ship the CUDA delegate in the wheel #21645
• Ship the quantized kernels as their own library #21642
• Ship a C++ SDK in the wheel #21639
• Split the runtime out of the Python bindings extension #21610

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21771

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 231 Pending, 2 Unrelated Failures, 3 Unclassified Failures

As of commit 3db4f2e with merge base d827cf7 ():

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build Linux CUDA Wheels / pytorch/executorch / build-manywheel-py3_13-cuda12_6 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build macOS Wheels / pytorch/executorch / build-wheel-py3_10-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_10-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• Build Linux CUDA Wheels / pytorch/executorch / upload / upload-manywheel-py3_13-cuda12_6 (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-demo-ios / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.