feat: support the Python stable ABI (abi3 and abi3t)

[henryiii]

Only build one wheel for 3.12+ and 3.15t+ per platform and support all future versions of Python.

Implemented with Fable, added abi3t, got numpy working, got chrono working, iterated with linked Claude sessions on several downstream projects (finding both fixes and performance optimizations to keep this from being to costly), simplified with Fable, and reviewed with Astra, which found five issues. This was originally following #6104, with a commit for each "PR" in that roadmap, but there was quite a bit of iteration, so I opened this as one PR. I'm thinking it might be preferred to review this way these days anyway.

This supports abi3t on 3.15+ and abi3 on 3.12+. Original performance was pretty bad, but optimized a bit and some parts are faster than the unstable ABI (though those optimizations can of course be applied there too later). It depends very much on what you are doing (some things are slower with stable ABI, like attribute access with no call), but for most projects, this should be performant enough. awkward showed a 1-3% drop for one microbenchmark, otherwise unmeasurable. After fixes, iminuit shows no measurable slowdown (one microbenchmark is 17% faster due to an optimization to method-descriptor calls we could also apply to non-stable ABI builds).

Questions:

• Should we auto-detect and use scikit-build-core's cmake variables?

🤖 AI text below 🤖

Description

Adds support for the CPython stable ABI (Py_LIMITED_API >= 0x030C0000, abi3) and the free-threaded stable ABI (abi3t, CPython 3.15+, PEP 803). One *.abi3.so per platform then runs on every CPython from the targeted version on.

Draft: opened so that others can build their projects against it. Tried so far on boost-histogram, iminuit, and awkward. PyTorch can't use it, due to custom code calling C-API functions, but checked to make sure it still works with the mode turned off.

How to try it:

• CMake: pybind11_add_module(example STABLE_ABI example.cpp) or -DPYBIND11_STABLE_ABI=ON (CMake 3.26+, FindPython mode). PYBIND11_STABLE_ABI_VERSION selects the targeted version (default 3.12; 3.15 on free-threaded Python).
• setuptools: Pybind11Extension(..., py_limited_api=True).
• Any other build: define Py_LIMITED_API=0x030C0000 and name the module *.abi3.so.

What changes:

• Type objects are created from PyType_Spec with PyType_FromMetaclass(). This path is also available to regular builds on 3.12+ as the PYBIND11_TYPE_CREATION_VIA_SPEC opt-in. (Edit: now opt-out)
• Field reads of CPython structs go through public functions (PyType_GetSlot(), PyObject_GetTypeData(), checked tuple/list accessors, __mro__/__dict__ lookups).
• Methods are bound through a pybind11 instancemethod type, which is a vectorcall method descriptor, so obj.method(...) does not create a bound method.
• chrono.h uses the datetime types by attribute; numpy.h under abi3t needs NumPy 2.5+.
• Not available under the stable ABI: embed.h, subinterpreter.h, py::custom_type_setup, GIL-held assertions; py::metaclass() is restricted. See docs/advanced/stable_abi.rst for the full table.
• Stable-ABI and regular modules in one process do not share internals (the internals ID gets a _stable / _stable_ft tag); pybind11_conduit_v1 bridges them.

CI builds and runs the test suite for abi3 on 3.12 to 3.14 and abi3t on 3.15t, and runs the 3.12 abi3 binaries on 3.13 and 3.14 and the 3.15t abi3t binaries on GIL-enabled 3.15.

Suggested changelog entry:

• Support the CPython stable ABI (abi3, Py_LIMITED_API >= 0x030C0000) and the free-threaded stable ABI (abi3t, CPython 3.15+). Enable with STABLE_ABI in pybind11_add_module(), py_limited_api=True in Pybind11Extension, or by defining Py_LIMITED_API. Type creation from PyType_Spec is also available to regular builds as the PYBIND11_TYPE_CREATION_VIA_SPEC opt-in.
• Type objects are now created with PyType_FromMetaclass() by default on CPython 3.12+. Define PYBIND11_TYPE_CREATION_VIA_SPEC=0 (CMake -DPYBIND11_TYPE_CREATION_VIA_SPEC=OFF) to restore the previous path.

---

📚 Documentation preview 📚: https://pybind11--6198.org.readthedocs.build/

[henryiii]

If this is better in smaller parts, there are some good candidates for that, including two bug fixes and a nice performance speedup:

🤖 AI text below 🤖

Yes. Beyond the method-descriptor patch, there are two real bug fixes and one large feature that do not depend on Py_LIMITED_API, plus a pile of prep refactors.

Bug fixes (strongest candidates)

• type_is_managed_by_our_internals() cross-module fix (ad10b98, include/pybind11/detail/cpp_conduit.h). Master compares tp_new against this module's pybind11_object_new, which fails for types whose base came from another module. The branch switches to the registry lookup PyPy already used. No test was added on the branch, so it needs one before it ships alone.
• pybind11_traverse/pybind11_clear under PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET on 3.13+ (part of 4d65cb2). Master unconditionally calls PyObject_VisitManagedDict/PyObject_ClearManagedDict on 3.13+, even when that macro puts the dict at tp_dictoffset and the type has no Py_TPFLAGS_MANAGED_DICT. The dict is then never visited or cleared by the GC, at best. I did not run this on master, so verify with a 3.13 build and that macro before filing.

Feature, independent of the stable ABI

• Spec-based type creation / PyType_FromMetaclass default on 3.12+ (fbc9d96, 8f47ff2, c05d08d, a5f341a, 04459a5, e669ac1, 4d65cb2). PYBIND11_TYPE_CREATION_VIA_SPEC is a separate switch from Py_LIMITED_API and already has its own CI job. It is big, but it is the natural first PR: the stable ABI PR then shrinks to "the spec path, plus limited-API gating".

Method descriptor / instancemethod (the one you named)

• 73cd296 and 721fabc give pybind11's own instancemethod type vectorcall and Py_TPFLAGS_METHOD_DESCRIPTOR. On the branch that type exists only under Py_LIMITED_API; master uses PyInstanceMethod_Type, which lacks the flag. A standalone PR has to make the own type unconditional. If you do that, 973e9ef comes along for free: the metaclass tp_getattro can be dropped everywhere, which removes the slow MRO walk on class attribute reads, and the interned-name type_lookup() speedup applies to regular builds too.

Prep refactors, low value alone but they shrink the main diff

• Traceback walk by attribute in error_already_set::what() (683f376).
• PyThreadState_GetInterpreter() instead of tstate->interp, with the PyPy fallback (5c50c3d, part of e0bc493).
• get_bases()/get_mro()/get_tp_name()/tp_name_is_one_of() helpers (d1af605, 1865f0b) and the PyTuple_GET_ITEM-style macro replacements (cdbc9b4).
• value_and_holder::type_is_exact() replacing the repeated Py_TYPE(v_h.inst) == v_h.type->type in init.h and pybind11.h.
• The cross-module test modules factored into an exec function (test-only).

Not separable

• The chrono per-TU fix (88fb22a) fixes helpers the branch itself introduced; master inlines the macros, so it does not apply.
• SABIModule export (573d161), eval.h, numpy.h, complex.h, the TLS/critical-section changes, and the conftest compile-time version logic are stable-ABI-only.

[rwgk]

@henryiii this is the first pass reviewing with

codex GPT-6.1-Sol ultra:

Rechecked against 75be228. All six findings remain after the rebase:

1. [P1] Pin registry candidates before acquiring the internals lock.
   get_object_handle() retains registry iterators and raw instances across values_and_holders(), which acquires the internals lock through all_type_info(). Under abi3t, contention suspends the outer registry critical section, allowing another thread to erase entries or rehash the shard. py::get_override() can then resume with invalid iterators or freed wrappers. Please collect strong candidates under the registry lock and inspect them afterward, as find_registered_python_instance() already does, retaining the selected wrapper through override lookup.

2. [P2] Keep dictionary-offset helpers available for non-limited builds.
   Defining PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET in a normal CPython 3.12+ build now makes including pybind11.h fail to compile. The new spec path calls type_dictoffset and type_basicsize, but both helpers exist only under Py_LIMITED_API. I reproduced this with CPython 3.14.4 headers; the same include compiles without that macro. Please provide non-limited equivalents or retain direct field access for this configuration.

3. [P2] Check base metaclasses before selecting the spec path.
   The fallback condition checks only the requested/default metaclass. Binding Base with py::metaclass(Meta), where Python defines class Meta(type): pass, selects the legacy path. Subsequently binding py::class_<Derived, Base>(m, "Derived") selects the spec path with the default pybind11_type. PyType_FromMetaclass() then raises a metaclass conflict; the previous PyType_Ready path accepts this binding. Please consider the base metaclasses when deciding whether to fall back.

4. [P2] Recognize Windows free-threaded Python suffixes.
   The fallback detection matches cpython-…t, whereas Windows Python 3.15t uses cp315t-win_amd64 / .cp315t-win_amd64.pyd. Since Python_FREE_THREADED was added in CMake 4.4, CMake 3.30–4.3 misdetects this interpreter as GIL-enabled. Default STABLE_ABI consequently targets 3.12 and fails compilation because free-threaded stable ABI requires 3.15.

5. [P2] Preserve FindPython paths for Windows abi3t consumers.
   With add_subdirectory(pybind11) and Python discovery inside that subdirectory, Python_INCLUDE_DIRS and Python_SABI_LIBRARY_DIRS remain scoped there. A caller using GIL-enabled Windows Python 3.15+, PYBIND11_ABI3T=ON, and pybind11_add_module(... STABLE_ABI ...) reaches the manual setup with empty paths. find_library(... NO_DEFAULT_PATH REQUIRED) fails even when python3t.lib is installed. Please obtain these paths from the global imported target or preserve them for callers.

6. [P2] Include the second chrono translation unit when selecting chrono tests.
   -DPYBIND11_TEST_OVERRIDE=test_chrono excludes the independently listed test_chrono_second_tu.cpp, while test_chrono.py unconditionally imports its submodule. The extension builds, but pytest fails collection with ImportError. Please include the helper translation unit whenever chrono is selected.

Finding 2 was reproduced through compilation at this revision. The others are based on code inspection; no test suites were run.