[feature](be) Parallelize garbage sweep across data directories - #65699
Draft
wenzhenghu wants to merge 25 commits into
Draft
[feature](be) Parallelize garbage sweep across data directories#65699wenzhenghu wants to merge 25 commits into
wenzhenghu wants to merge 25 commits into
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#68 Problem Summary: Migrate the branch-3.1 shutdown tablet sweep configurability to master. This change adds dynamic round budget and interval configs, refactors shutdown tablet cleanup into round-based sweeping with fallback defaults for invalid runtime values, exposes BE-side backlog and timing metrics, and ports focused BE unit tests to the master tablet manager layout. ### Release note Support configuring shutdown tablet sweep round budget and interval on master, and expose BE-side sweep observability metrics. ### Check List (For Author) - Test: Attempted ./run-be-ut.sh --run with TabletMgrTest filters, but the build was blocked by an existing contrib/openblas environment error; also ran ./build-support/clang-format.sh and git diff --check - Behavior changed: Yes. Shutdown tablet cleanup pacing is dynamically configurable and additional sweep metrics are exposed. - Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29627 ms |
Contributor
TPC-DS: Total hot run time: 178611 ms |
Contributor
ClickBench: Total hot run time: 25.08 s |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29666 ms |
Contributor
TPC-DS: Total hot run time: 178110 ms |
Contributor
ClickBench: Total hot run time: 25.02 s |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29094 ms |
Contributor
TPC-DS: Total hot run time: 177720 ms |
Contributor
ClickBench: Total hot run time: 25.02 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29488 ms |
Contributor
TPC-DS: Total hot run time: 176813 ms |
Contributor
ClickBench: Total hot run time: 24.84 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29889 ms |
Contributor
TPC-DS: Total hot run time: 176864 ms |
Contributor
ClickBench: Total hot run time: 25.07 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Missing remote storage resources can remain absent across garbage sweep epochs. Treating that condition as a generic job failure caused the failed-job counter to grow every epoch even though the GC marker was intentionally retained for retry. Keep missing-resource handling as a warning with backlog retention, while preserving failure propagation for malformed metadata, metadata operations, and remote I/O. Add coverage for the error boundary and for shutdown-tablet dispatch after a worker-owned DataDir becomes unused.
### Release note
Missing remote storage resources no longer mark the DataDir remote GC job as failed; their GC markers remain in the backlog for retry.
### Check List (For Author)
- Test:
- Static checks: `build-support/clang-format.sh`, `build-support/check-format.sh`, and `git diff --check` passed
- Unit Test: Not completed locally; `run-be-ut.sh` failed during CMake configuration because the local OpenBLAS probe could not resolve `*_DEFAULT_UNROLL_*` macros. The user will run the targeted BE unit tests manually.
- Behavior changed: Yes. Missing storage resources are warning-only with backlog retention; malformed PB, metadata, and remote I/O errors still fail the job.
- Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29659 ms |
Contributor
TPC-DS: Total hot run time: 176941 ms |
Contributor
ClickBench: Total hot run time: 24.88 s |
### What problem does this PR solve? Issue Number: None Related PR: apache#65699 Problem Summary: StorageEngineTest.TestBrokenDisk defined a mutable string config inside the test body. The config registry retained the address of that stack object after the test returned, so a later broken-disk health check accessed freed stack storage and triggered an AddressSanitizer stack-use-after-return failure. Use the process-lifetime config::broken_storage_path registration and restore its original value after the test. ### Release note None ### Check List (For Author) - Test: - Static checks: `build-support/clang-format.sh`, `build-support/check-format.sh`, and `git diff --check` passed - Unit Test: Pending manual execution by the user - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29433 ms |
Contributor
TPC-DS: Total hot run time: 176770 ms |
Contributor
ClickBench: Total hot run time: 24.98 s |
### What problem does this PR solve? Issue Number: None Related PR: apache#65699 Problem Summary: The BE unit-test config suite clears the shared config registry. TestBrokenDisk previously restored the broken_storage_path entry through a stack-backed DEFINE_mString, which allowed persistence assertions to run but caused an AddressSanitizer stack-use-after-return. After removing that unsafe definition, the field was no longer registered and be_custom.conf was not generated. Re-register the field with process-lifetime config::broken_storage_path storage so both the registry and object lifetime remain valid. ### Release note None ### Check List (For Author) - Test: - Static checks: `build-support/clang-format.sh`, `build-support/check-format.sh`, and `git diff --check` passed - Unit Test: Pending CI and manual execution - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29515 ms |
Contributor
TPC-DS: Total hot run time: 177849 ms |
Contributor
ClickBench: Total hot run time: 24.91 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
wenzhenghu
marked this pull request as ready for review
July 30, 2026 10:11
wenzhenghu
marked this pull request as draft
July 30, 2026 12:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
The BE garbage sweep coordinator previously executed path-scoped cleanup serially
across all DataDirs. A slow or heavily loaded DataDir could therefore extend the
entire sweep epoch and delay cleanup on other independent disks. Shutdown tablet
cleanup also used fixed pacing and provided limited round-level observability.
This change adds an optional per-DataDir garbage sweep worker model while aligning
the configurable shutdown tablet sweep implementation with master. Each DataDir
owns a bounded worker queue, and the coordinator dispatches typed jobs in phase
order, waits for every phase to complete, and aggregates failures before advancing.
The feature is controlled by
enable_data_dir_sweep_workerand is disabled bydefault. When disabled, the same typed jobs and phase barriers execute
synchronously. This preserves synchronous scheduling, but the error-propagation
changes described below apply in both modes and are not identical to the
pre-change behavior.
Implementation Notes
independent DataDirs to execute the same phase concurrently.
the same physical device.
aggregate per-disk results, and requeue every unresolved tablet after executor
or submission failures.
shutdown_tablet_sweep_round_budget:[1, 10000], default200;shutdown_tablet_sweep_interval_ms:[0, 10000], default1000;backlog, failure, and trash-capacity metrics.
submit failures, worker concurrency and FIFO ordering, synchronous fallback,
shutdown lifecycle, remote GC retry behavior, and trash capacity refresh.
Behavior Changes
no longer skips the shutdown-tablet, global-metadata, remote-GC, or capacity
phases.
failed.
protobuf markers fail the job. A missing storage resource is warning-only: its
marker remains in the backlog for retry and does not increment the generic
failed-job counter.
remote-GC job completes before any capacity-refresh job starts.
Release note
Add an optional
enable_data_dir_sweep_workersetting to parallelize garbagesweeping across independent DataDirs. The setting is disabled by default.
Shutdown tablet sweep pacing is also dynamically configurable with additional
observability.
Check List (For Author)
Test:
build-support/clang-format.shpassedbuild-support/check-format.shpassedgit diff --checkpassed./build.sh --bepassed on wzh90 withGLIBC_COMPATIBILITY=OFFDataDirSweepWorkerTest: 9 tests passed on wzh90StorageEngineTest: 4 tests passed on wzh90TabletMgrTest: 16 tests passed on wzh90incrementing failed-job counters.
becomes broken/unused.
run-be-ut.shattempt stopped during CMake configuration becausethe OpenBLAS probe could not resolve
*_DEFAULT_UNROLL_*macros; the newtests did not execute locally.
/dev/sdcand/dev/sdbdevices.496 ms and 434 ms; the epoch took 952 ms.
525 ms; the epoch took 538 ms, a 43.5% reduction.
path -> shutdown -> remote GC -> capacity refresh phase barriers.
resolved-entry budget of 2 and a 200 ms interval, the backlog decreased
from 16 to 0; 16 tablets were resolved and none failed.
entries and 1,000 ms at the use site.
trash directories each. The active epoch completed before BE exit without
a forced kill, and all 40,000 directories were removed.
worker-unavailable or failed-job counters.
connectivity and BE health.
covered by the BE unit tests rather than live fault injection.
Behavior changed: Yes. Enabling
enable_data_dir_sweep_workerparallelizespath-scoped jobs across DataDirs while preserving phase ordering and
per-DataDir serialization. Disabling it retains synchronous scheduling only;
the error-propagation changes listed above still apply.
Does this need documentation: No