Skip to content

[fix][broker] Fix early completion and false timeouts in SplitManager and UnloadManager - #26363

Open
void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:fix/split-manager-timeout-cleanup-race
Open

[fix][broker] Fix early completion and false timeouts in SplitManager and UnloadManager#26363
void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:fix/split-manager-timeout-cleanup-race

Conversation

@void-ptr974

@void-ptr974 void-ptr974 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Motivation

SplitManager and UnloadManager keep one in-flight future per bundle so concurrent callers can wait for the same split or unload operation.

The event and close paths completed that future before removing it from the in-flight map. The timeout path attached cleanup with whenComplete, but discarded the dependent stage and returned the original future. As a result, callers could observe a completed wait while the old entry was still visible. An immediate retry for the same bundle could then reuse the completed future instead of waiting for the new operation. A stale timeout callback could also remove a replacement entry by key.

Because the state event has already been published, this race does not normally stop the underlying split or unload operation. It affects completion tracking: a retry for the same bundle can return the previous request's result before the new operation completes. In a narrower race, stale cleanup can remove the new request's tracking entry, causing the caller or scheduler to wait until the configured timeout (60 seconds by default) and report a failure even if the underlying operation succeeds.

Modifications

  • Remove the matching in-flight entry before completing its future for state events and manager shutdown.
  • Return a dependent timeout stage that completes only after cleanup has run.
  • Guard timeout cleanup with remove(bundle, future) so a stale callback cannot remove a newer request.
  • Replace reflection-based test inspection with package-private, test-visible request-count accessors.
  • Add deterministic completion-boundary, immediate-retry, concurrent-waiter, and close-versus-completion tests for both managers.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added and extended unit tests and was verified locally with:

  • ./gradlew :pulsar-broker:test --tests org.apache.pulsar.broker.loadbalance.extensions.manager.SplitManagerTest --tests org.apache.pulsar.broker.loadbalance.extensions.manager.UnloadManagerTest
  • ./gradlew :pulsar-broker:checkstyleMain :pulsar-broker:checkstyleTest
  • ./gradlew quickCheck
  • git diff --check

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

@void-ptr974 void-ptr974 changed the title [fix][broker] Ensure split and unload waits complete after in-flight cleanup [fix][broker] Fix premature completion and false timeouts for split and unload retries Aug 18, 2026
@void-ptr974 void-ptr974 changed the title [fix][broker] Fix premature completion and false timeouts for split and unload retries [fix][broker] Fix early completion and false timeouts in SplitManager and UnloadManager Aug 18, 2026
@void-ptr974
void-ptr974 marked this pull request as ready for review August 18, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant