Skip to content

[improve][build] Upgrade async-http-client to 3.0.13 and netty-reactive-streams to 2.0.19 - #26366

Open
lhotari wants to merge 2 commits into
apache:masterfrom
lhotari:lh-improve-ahc-netty-reactive
Open

[improve][build] Upgrade async-http-client to 3.0.13 and netty-reactive-streams to 2.0.19#26366
lhotari wants to merge 2 commits into
apache:masterfrom
lhotari:lh-improve-ahc-netty-reactive

Conversation

@lhotari

@lhotari lhotari commented Aug 18, 2026

Copy link
Copy Markdown
Member

Motivation

async-http-client and netty-reactive-streams are the HTTP stack behind the admin client.
They were originally part of #26358 (networking dependency updates) and are split out here,
because that PR reproducibly failed four integration and system test jobs and these two are the
most likely trigger.

Modifications

gradle/libs.versions.toml:

library from to
async-http-client 3.0.10 3.0.13
netty-reactive-streams 2.0.6 2.0.19

Plus the corresponding jar names in the server and shell distribution LICENSE.bin.txt files.

Why these are split out — what CI showed

On #26358 the following jobs failed, twice, identically (master and the twelve sibling
dependency PRs were green):

  • CI - Integration - Cli
  • CI - System - Function
  • CI - System - Pulsar Connectors - Process
  • CI - System - Pulsar Connectors - Thread

All four share one root cause. pulsar-admin / pulsar-client print a JDK restricted-method
warning to stderr, and the tests assert the CLI produces no stderr output:

WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::loadLibrary has been called by io.netty.util.internal.NativeLibraryUtil
         in an unnamed module (file:/pulsar/lib/io.netty-netty-common-4.2.17.Final.jar)

What was ruled out while diagnosing this:

  • Not the distribution contents. Diffing the built apache-pulsar-*-bin.tar.gz jar list
    against the base commit shows only the intended version bumps. No new Netty jars appear, and
    netty-transport-native-epoll / -io_uring are present in both.
  • Not a Netty upgrade. Netty stays at 4.2.17.Final.
  • Not AHC switching transports by default. ahc-default.properties has
    useNativeTransport=false and useOnlyEpollNativeTransport=false in both 3.0.10 and 3.0.13.
  • Not flaky. It reproduced on two independent runs, and the same jobs pass everywhere else.

So the trigger is a change in when a Netty native library is loaded, not in which Netty is
used. AHC 3.0.13 does add DNS/IP-cooldown behaviour (failedIpCooldownEnabled,
loadBalance) that 3.0.10 lacks, which is a plausible path to an availability probe such as
IoUring.isAvailable() loading the native library, but this PR does not claim that as proven —
it exists so CI can confirm or refute it in isolation.

Related: bin/pulsar passes --enable-native-access=ALL-UNNAMED while
bin/pulsar-admin-common.sh does not, which is why the warning is only visible from the CLI.
That gap is a pre-existing issue and is fixed separately in #26365.

Verifying this change

  • Make sure that the change passes the CI checks.

This change is already covered by existing tests — specifically the CLI, Function and
Connectors integration/system suites named above, which are exactly what this PR is meant to
exercise.

Verified locally with ./gradlew sanityCheck and ./gradlew checkBinaryLicense.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

…ve-streams to 2.0.19

- async-http-client       3.0.10 -> 3.0.13
- netty-reactive-streams  2.0.6  -> 2.0.19

These two form the HTTP stack behind the admin client. They were originally part of the
networking dependency update (apache#26358) and are split out here because that PR
reproducibly failed four integration and system test jobs, and these are the most likely
trigger.

The failure mode is that pulsar-admin / pulsar-client print a JDK restricted-method
warning to stderr:

  WARNING: java.lang.System::loadLibrary has been called by
           io.netty.util.internal.NativeLibraryUtil in an unnamed module
           (file:/pulsar/lib/io.netty-netty-common-4.2.17.Final.jar)

Tests that assert the CLI produces no stderr output then fail. Netty itself is not changed
by this PR, and the server distribution's jar list is identical apart from these two
version bumps, so the trigger is a change in when a Netty native library gets loaded
rather than which Netty is used. Note that bin/pulsar passes
--enable-native-access=ALL-UNNAMED but bin/pulsar-admin-common.sh does not, which is why
the warning is only visible from the CLI; that gap is addressed separately.

Splitting this out lets CI confirm whether these two libraries are in fact the trigger,
while the rest of the networking updates proceed independently.

Assisted-by: Claude Code (Opus 5)
@lhotari lhotari added the area/dependency Pull requests that update a dependency file label Aug 18, 2026
@lhotari lhotari added this to the 5.0.0-M2 milestone Aug 18, 2026
@lhotari

lhotari commented Aug 18, 2026

Copy link
Copy Markdown
Member Author

Update: the mechanism is now confirmed from the async-http-client sources, so this is no longer a hypothesis.

ChannelManager's constructor, on the path where AHC creates its own event loop group:

3.0.10

if (config.isUseNativeTransport()) {
    transportFactory = getNativeTransportFactory(config);
} else {
    transportFactory = NioTransportFactory.INSTANCE;
}

3.0.13

if (config.isUseNativeTransport()) {
    transportFactory = getNativeTransportFactory(config);
} else {
    transportFactory = autoSelectTransportFactory();
}

autoSelectTransportFactory() is new, and its own comment describes the change in default behaviour:

// Default when useNativeTransport is unset: native transport if its lib is on the classpath (silently),
// else NIO. Use -Dio.netty.transport.noNative=true to force NIO.

It calls EpollTransportFactory.isAvailable(), and that availability probe is what loads the Netty
native library. On Java 24+ that produces the java.lang.System::loadLibrary restricted-method
warning on stderr, which fails the tests asserting the CLI writes nothing to stderr.

So with useNativeTransport=false (still the default in both versions, unchanged in
ahc-default.properties), 3.0.10 went straight to NIO and never touched a native library, whereas
3.0.13 probes for and prefers a native transport.

Why only the CLI is affected: AsyncHttpConnector calls confBuilder.setEventLoopGroup(...) when
shared resources provide one. With an externally supplied group AHC takes the other branch and
simply maps the group's type, never invoking the selection logic. The plain CLI admin client has no
shared event loop group, so it goes through autoSelectTransportFactory().

This is a behaviour change rather than a defect: preferring native transport is a reasonable default,
and AHC deliberately prefers epoll over io_uring there because "io_uring needs RLIMIT_MEMLOCK (often
constrained in CI/containers)". The correct fix is to permit the native access rather than to suppress
the transport, which is what #26365 (CLI scripts) and #26367 (test JVMs) do. Once those land, this
upgrade should be unblocked.

For completeness, two AHC settings that might look like alternatives but are not:

  • useNativeTransport=true does not avoid the native load; it makes it more likely. Note also that
    3.0.10 throws IllegalArgumentException when no native transport is available, while 3.0.13
    degrades to NIO with a warning, so graceful fallback only exists from 3.0.13.
  • useOnlyEpollNativeTransport is only consulted when useNativeTransport=true. With the default
    false, io_uring is preferred over epoll; setting it true restricts selection to epoll.

…y-reactive

# Conflicts:
#	distribution/shell/src/assemble/LICENSE.bin.txt
#	gradle/libs.versions.toml

@void-ptr974 void-ptr974 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/dependency Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants