[improvement](be) Configure shutdown tablet sweep#65699
Open
wenzhenghu wants to merge 9 commits into
Open
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#68 Problem Summary: Migrate the branch-3.1 shutdown tablet sweep configurability to master. This change adds dynamic round budget and interval configs, refactors shutdown tablet cleanup into round-based sweeping with fallback defaults for invalid runtime values, exposes BE-side backlog and timing metrics, and ports focused BE unit tests to the master tablet manager layout. ### Release note Support configuring shutdown tablet sweep round budget and interval on master, and expose BE-side sweep observability metrics. ### Check List (For Author) - Test: Attempted ./run-be-ut.sh --run with TabletMgrTest filters, but the build was blocked by an existing contrib/openblas environment error; also ran ./build-support/clang-format.sh and git diff --check - Behavior changed: Yes. Shutdown tablet cleanup pacing is dynamically configurable and additional sweep metrics are exposed. - Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29627 ms |
Contributor
TPC-DS: Total hot run time: 178611 ms |
Contributor
ClickBench: Total hot run time: 25.08 s |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29666 ms |
Contributor
TPC-DS: Total hot run time: 178110 ms |
Contributor
ClickBench: Total hot run time: 25.02 s |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29094 ms |
Contributor
TPC-DS: Total hot run time: 177720 ms |
Contributor
ClickBench: Total hot run time: 25.02 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29488 ms |
Contributor
TPC-DS: Total hot run time: 176813 ms |
Contributor
ClickBench: Total hot run time: 24.84 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29606 ms |
Contributor
TPC-DS: Total hot run time: 177062 ms |
Contributor
ClickBench: Total hot run time: 25.41 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
This change ports the configurable BE shutdown tablet sweep from branch-3.1 to master. It adds dynamic round budget and interval configs, preserves the original shutdown-tablet cleanup semantics, and exposes round-level observability. Follow-up review fixes tighten the dynamic config bounds to [1, 10000] and [0, 10000], clarify reached-end round termination semantics, extend BE unit coverage for boundary and retry-order cases, rename the success counters and metrics from moved to resolved, and fix the invalid-config fallback test so it validates the use-site fallback path directly.
Implementation Notes
Old flow:
flowchart TD A["start_trash_sweep"] --> B["init last_it at shutdown tablet queue begin"] B --> C["delete one batch"] C --> D["set limit to 200"] D --> E["fetch batch tablets"] E --> F{"process fetched tablets"} F -->|success| G["decrease limit"] F -->|failure| H["append to failed tablets"] G --> I{"limit is exhausted"} H --> I I -->|yes| J["return true"] I -->|no| K{"batch is empty"} K -->|yes| L["return false"] K -->|no| E J --> M{"continue outer while loop"} M -->|yes| N["sleep one second"] N --> C M -->|no| O["splice failed tablets back to queue tail"]New flow:
flowchart TD A["start_trash_sweep"] --> B["sweep shutdown tablets"] B --> C["init last_it at shutdown tablet queue begin"] C --> D["delete one round"] D --> E["get effective round budget"] E --> F["fetch shutdown tablets with chunk limits"] F --> G{"process fetched tablets"} G -->|resolved| H["consume budget and update resolved count"] G -->|failed| I["append to failed tablets"] H --> J{"reached queue tail"} I --> J J -->|yes| K["return round result"] J -->|no| L{"budget is exhausted"} L -->|yes| M["mark need_continue"] L -->|no| F M --> K K --> N{"round needs another pass"} N -->|yes| O["wait next round with effective interval"] O --> D N -->|no| P["splice failed tablets back to queue tail"]Implementation details:
last_itin queue order;use_count() > 1are skipped in the current pass;limit = 200andsleep(1)become runtime-configurable round budget and interval;max_to_fetchlimits how many removable entries are returned per fetch;scan_chunklimits how many queue entries are scanned while holding the shutdown-tablet lock;RoundResultrecords resolved count, failed count, elapsed time, and whether another round is needed.Release note
Support configuring BE shutdown tablet sweep pacing and observability on master.
Check List (For Author)
Test
master migration plus review-driven fixes.
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)