feat(async): greenlet Submitter and PluginManager.run_async

.pre-commit-config.yaml

@@ -46,4 +46,4 @@ repos:
     -   id: mypy
         files: ^(src/|testing/)
         args: []
-        additional_dependencies: [pytest]
+        additional_dependencies: [pytest, types-greenlet]

changelog/704.removal.rst

@@ -0,0 +1,8 @@
+Hook options are now :class:`pluggy.HookspecConfiguration` /
+:class:`pluggy.HookimplConfiguration` objects (markers attach these instead of
+dicts). ``PluginManager.parse_hookimpl_opts`` /
+``parse_hookspec_opts`` remain as a deprecated pytest/support concession that
+returns legacy dicts and are only invoked during registration when a subclass
+overrides them and no modern configuration attribute was found.
+``HookspecOpts`` / ``HookimplOpts`` TypedDicts remain importable for
+pytest/typing compatibility.

changelog/706.trivial.rst

@@ -0,0 +1,3 @@
+``HookSpec`` now stores its :class:`pluggy.HookspecConfiguration` as
+``config``; the old ``opts`` attribute remains as a deprecated alias
+property.

changelog/707.feature.rst

@@ -0,0 +1,9 @@
+Hook implementations are now represented by dedicated types:
+:class:`pluggy.NormalImpl` for normal implementations and
+:class:`pluggy.WrapperImpl` for (old- and new-style) wrappers, both
+subclasses of :class:`pluggy.HookImpl`.
+``HookimplConfiguration.create_hookimpl()`` selects the appropriate
+subclass, and ``WrapperImpl.setup_and_get_completion_hook()`` exposes
+wrapper setup/teardown as a ``CompletionHook`` callback.
+``HookImpl`` now stores its configuration as ``hookimpl_config``; the old
+``opts`` attribute remains as a deprecated alias property.

changelog/708.feature.rst

@@ -0,0 +1,12 @@
+:class:`pluggy.HookCaller` is now a runtime-checkable
+:class:`~typing.Protocol` implemented by the new concrete callers
+:class:`pluggy.NormalHookCaller` (split normal/wrapper implementation
+lists), :class:`pluggy.HistoricHookCaller` (call memorization and replay,
+no wrappers) and :class:`pluggy.SubsetHookCaller`.
+``isinstance(caller, HookCaller)`` keeps working for all of them.
+
+The hook execution engine now runs a dual-sequence multicall: wrappers own
+their setup/teardown through ``CompletionHook`` callbacks which run LIFO
+after the normal implementations, removing all wrapper flag branching from
+the hot loop. Hook call monitoring callbacks still receive a single
+combined implementation list.

changelog/709.feature.rst

@@ -0,0 +1,6 @@
+New :class:`pluggy.ProjectSpec` hub bundles a project's
+:class:`~pluggy.HookspecMarker`, :class:`~pluggy.HookimplMarker` and
+:meth:`~pluggy.ProjectSpec.create_plugin_manager` under one project name.
+:class:`~pluggy.HookspecMarker`, :class:`~pluggy.HookimplMarker` and
+:class:`~pluggy.PluginManager` now accept either a project name string or
+a ``ProjectSpec`` instance; plain strings keep working unchanged.

changelog/710.feature.rst

@@ -0,0 +1,9 @@
+New async bridge for awaitable hook results:
+``await pm.run_async(lambda: pm.hook.my_hook())`` runs the hook call in a
+greenlet context where awaitable results from hook implementations are
+awaited on the current event loop ("await-me-maybe"). Outside
+``run_async``, awaitable results are passed through unchanged, as before.
+Requires the optional ``greenlet`` dependency, installable via
+``pluggy[async]``. The bridge is a persistent per-manager ``Submitter``;
+async wrapper generators are not auto-awaited (a manual
+``async_generator_to_sync`` helper is provided).

docs/api_reference.rst

@@ -8,6 +8,9 @@ API Reference
 .. autoclass:: pluggy.PluginManager
     :members:
 
+.. autoclass:: pluggy.ProjectSpec
+    :members:
+
 .. autoclass:: pluggy.PluginValidationError
     :show-inheritance:
     :members:
@@ -29,6 +32,17 @@ API Reference
     :members:
     :special-members: __call__
 
+.. autoclass:: pluggy.NormalHookCaller()
+    :members:
+    :special-members: __call__
+
+.. autoclass:: pluggy.HistoricHookCaller()
+    :members:
+    :special-members: __call__
+
+.. autoclass:: pluggy.SubsetHookCaller()
+    :members:
+
 .. autoclass:: pluggy.HookCallError()
     :show-inheritance:
     :members:
@@ -40,14 +54,20 @@ API Reference
 .. autoclass:: pluggy.HookImpl()
     :members:
 
-.. autoclass:: pluggy.HookspecOpts()
+.. autoclass:: pluggy.NormalImpl()
     :show-inheritance:
     :members:
 
-.. autoclass:: pluggy.HookimplOpts()
+.. autoclass:: pluggy.WrapperImpl()
     :show-inheritance:
     :members:
 
+.. autoclass:: pluggy.HookspecConfiguration()
+    :members:
+
+.. autoclass:: pluggy.HookimplConfiguration()
+    :members:
+
 
 Warnings
 --------

docs/index.rst

@@ -800,10 +800,13 @@ and particular plugins in it:
 
 Parsing mark options
 ^^^^^^^^^^^^^^^^^^^^
-You can retrieve the *options* applied to a particular
-*hookspec* or *hookimpl* as per :ref:`marking_hooks` using the
+Markers attach :class:`~pluggy.HookspecConfiguration` /
+:class:`~pluggy.HookimplConfiguration` objects to functions. The
 :py:meth:`~pluggy.PluginManager.parse_hookspec_opts()` and
-:py:meth:`~pluggy.PluginManager.parse_hookimpl_opts()` respectively.
+:py:meth:`~pluggy.PluginManager.parse_hookimpl_opts()` methods remain as a
+**deprecated** pytest/support concession that returns legacy dict-shaped
+options; registration only calls them when a subclass overrides them and no
+modern configuration attribute was found.
 
 
 .. _calling:

pyproject.toml

@@ -35,9 +35,13 @@ readme = {file = "README.rst", content-type = "text/x-rst"}
 requires-python = ">=3.10"
 
 dynamic = ["version"]
+
+[project.optional-dependencies]
+async = ["greenlet"]
+
 [dependency-groups]
 dev = ["pre-commit", "tox"]
-testing = ["pytest", "pytest-benchmark", "coverage"]
+testing = ["pytest", "pytest-benchmark", "coverage", "greenlet"]
 
 
 [tool.setuptools]

src/pluggy/__init__.py

@@ -1,28 +1,44 @@
 __all__ = [
+    "HistoricHookCaller",
     "HookCallError",
     "HookCaller",
     "HookImpl",
     "HookRelay",
+    "HookimplConfiguration",
     "HookimplMarker",
     "HookimplOpts",
+    "HookspecConfiguration",
     "HookspecMarker",
     "HookspecOpts",
+    "NormalHookCaller",
+    "NormalImpl",
     "PluggyTeardownRaisedWarning",
     "PluggyWarning",
     "PluginManager",
     "PluginValidationError",
+    "ProjectSpec",
     "Result",
+    "SubsetHookCaller",
+    "WrapperImpl",
     "__version__",
 ]
+from ._config import HookimplConfiguration
+from ._config import HookspecConfiguration
+from ._hooks import HistoricHookCaller
 from ._hooks import HookCaller
 from ._hooks import HookImpl
 from ._hooks import HookimplMarker
-from ._hooks import HookimplOpts
 from ._hooks import HookRelay
 from ._hooks import HookspecMarker
-from ._hooks import HookspecOpts
+from ._hooks import NormalHookCaller
+from ._hooks import NormalImpl
+from ._hooks import SubsetHookCaller
+from ._hooks import WrapperImpl
 from ._manager import PluginManager
 from ._manager import PluginValidationError
+from ._project import ProjectSpec
+from ._pytest_compat import HookimplOpts
+from ._pytest_compat import HookspecOpts
 from ._result import HookCallError
 from ._result import Result
 from ._warnings import PluggyTeardownRaisedWarning

src/pluggy/_async.py

@@ -0,0 +1,173 @@
+"""
+Async support for pluggy using greenlets.
+
+This module provides async functionality for pluggy, allowing hook
+implementations to return awaitable objects that are automatically awaited
+when running in an async context (see :meth:`PluginManager.run_async
+<pluggy.PluginManager.run_async>`).
+"""
+
+from __future__ import annotations
+
+from collections.abc import AsyncGenerator
+from collections.abc import Awaitable
+from collections.abc import Callable
+from collections.abc import Generator
+from typing import Any
+from typing import cast
+from typing import Final
+from typing import TYPE_CHECKING
+from typing import TypeVar
+
+
+_T = TypeVar("_T")
+_Y = TypeVar("_Y")
+_S = TypeVar("_S")
+
+if TYPE_CHECKING:
+    import greenlet
+
+
+#: Sentinel distinguishing "worker never completed" from a legitimate
+#: ``None`` result.
+_UNSET: Final = object()
+
+
+class Submitter:
+    """Bridge between synchronous hook execution and an async event loop.
+
+    While :meth:`run` is active, awaitables passed to :meth:`maybe_submit` or
+    :meth:`require_await` are switched to the awaiting parent greenlet, which
+    awaits them and switches the result back. When inactive,
+    :meth:`maybe_submit` passes awaitables through unchanged
+    ("await-me-maybe").
+    """
+
+    _active_submitter: greenlet.greenlet | None
+
+    def __init__(self) -> None:
+        self._active_submitter = None
+
+    def __repr__(self) -> str:
+        return f"<Submitter active={self._active_submitter is not None}>"
+
+    def maybe_submit(self, coro: Awaitable[_T]) -> _T | Awaitable[_T]:
+        """Await an awaitable if active, else return it unchanged.
+
+        This enables backward compatibility for datasette
+        and https://simonwillison.net/2020/Sep/2/await-me-maybe/
+        """
+        active = self._active_submitter
+        if active is not None:
+            # We're in a greenlet context; switch to the parent with the
+            # awaitable. The parent awaits it and switches back the result.
+            res: _T = active.switch(coro)
+            return res
+        else:
+            return coro
+
+    def require_await(self, coro: Awaitable[_T]) -> _T:
+        """Await an awaitable, raising an error if not in async context.
+
+        Ownership of ``coro`` is taken either way: when inactive it is closed
+        before raising, so callers never leak a coroutine that would later
+        warn about never having been awaited.
+        """
+        active = self._active_submitter
+        if active is not None:
+            res: _T = active.switch(coro)
+            return res
+        else:
+            # Not going to await it, so dispose of it rather than let it be
+            # collected unawaited (a RuntimeWarning raised inside __del__).
+            close = getattr(coro, "close", None)
+            if close is not None:
+                close()
+            raise RuntimeError("require_await called outside of async context")
+
+    async def run(self, sync_func: Callable[[], _T]) -> _T:
+        """Run a synchronous function in a worker greenlet with async support.
+
+        Awaitables submitted by hook implementations during the call are
+        awaited on this coroutine's event loop.
+        """
+        try:
+            import greenlet
+        except ImportError:
+            raise RuntimeError("greenlet is required for async support") from None
+
+        if self._active_submitter is not None:
+            raise RuntimeError("Submitter is already active")
+
+        main_greenlet = greenlet.getcurrent()
+        result: object = _UNSET
+        exception: BaseException | None = None
+
+        def greenlet_func() -> None:
+            nonlocal result, exception
+            try:
+                result = sync_func()
+            except BaseException as e:
+                exception = e
+
+        worker_greenlet = greenlet.greenlet(greenlet_func)
+        # Let maybe_submit/require_await switch back to this greenlet.
+        self._active_submitter = main_greenlet
+        try:
+            # Run the worker; every switch back carries an awaitable to
+            # process, until the worker finishes (switching back None).
+            awaitable = worker_greenlet.switch()
+            while awaitable is not None:
+                try:
+                    awaited_result = await awaitable
+                except BaseException as e:
+                    # Raise at the submission site inside the worker.
+                    awaitable = worker_greenlet.throw(e)
+                else:
+                    awaitable = worker_greenlet.switch(awaited_result)
+        finally:
+            self._active_submitter = None
+
+        if exception is not None:
+            raise exception
+        assert result is not _UNSET, "worker greenlet did not complete"
+        return cast(_T, result)
+
+
+def async_generator_to_sync(
+    async_gen: AsyncGenerator[_Y, _S], submitter: Submitter
+) -> Generator[_Y, _S, None]:
+    """Convert an async generator to a sync generator using a `Submitter`.
+
+    This helper allows wrapper implementations to use async generators while
+    maintaining compatibility with the sync generator interface expected by
+    the hook system. The submitter must be active (i.e. the generator must be
+    consumed under :meth:`Submitter.run`).
+    """
+    try:
+        # Start the async generator.
+        value = submitter.require_await(async_gen.__anext__())
+
+        while True:
+            try:
+                sent_value = yield value
+                try:
+                    value = submitter.require_await(async_gen.asend(sent_value))
+                except StopAsyncIteration:
+                    return
+
+            except GeneratorExit:
+                # Generator is being closed; close the async generator too.
+                submitter.require_await(cast(Awaitable[Any], async_gen.aclose()))
+                raise
+
+            except BaseException as exc:
+                # Exception was thrown into the generator; forward it into
+                # the async generator.
+                try:
+                    value = submitter.require_await(async_gen.athrow(exc))
+                except StopAsyncIteration:
+                    return
+
+    except StopAsyncIteration:
+        return