[fix](cloud) Use committed TSO and per-table transaction waits for bounded incremental reads - #67820
Open
luwei16 wants to merge 5 commits into
Open
[fix](cloud) Use committed TSO and per-table transaction waits for bounded incremental reads#67820luwei16 wants to merge 5 commits into
luwei16 wants to merge 5 commits into
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: Strongly consistent cloud incremental reads drain earlier transactions even when those transactions fall outside the requested historical window. A delayed write can therefore block an otherwise readable window. Track commit TSO allocations and unfinished transactions under the allocator lock, and publish a readable prefix together with the reserved TSO window only after their journal write succeeds. Accepted bounded windows skip transaction watermark/conflict polling and retain the MetaService visible-version refresh. Allocate commit TSO after bitmap preparation, callbacks and metadata validation. Preserve the earliest registration across submission retries and release it only after real VISIBLE/ABORTED, including lazy-commit reconciliation. On master recovery, retain the durable prefix until a fixed, instance-wide transaction bound passes a strict MetaService check after the configured recovery delay. The fixed wait retains the agreed old-master fencing limitation. ### Release note In cloud mode, strongly consistent incremental queries with an explicit end on every incremental relation use committed TSO. Unready windows return MySQL error 5100 (ERR_INCR_WINDOW_NOT_READY), or Flight UNAVAILABLE with business metadata; clients should retry the same window. information_schema.tso_status exposes COMMITTED_TSO and COMMITTED_TSO_PHYSICAL_TIME. The TSO persistence window defaults to one second. Upgrade MetaService before enabling prefix recovery on new FEs. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 79 distinct focused FE tests, 9 BE scanner tests and 3 MetaService tests passed. - test_committed_tso and test_binlog_changes_syntax passed; new output generated by the standard regression runner. - Real master/follower MySQL and Flight statement/prepared error contracts passed. - Three-FE failover with an allocated, PREPARED transaction preserved the old prefix and historical reads; confirmed abort released recovery and advanced the prefix. - FE build and Checkstyle, ASAN BE/Cloud builds, clang-format 16 and build hygiene passed. clang-tidy reported no changed-line diagnostics with a matching-toolchain wrapper and an analysis-only overlay for an existing unmatched suppression comment; five unchanged scanner diagnostics remain. - No throughput or latency benchmark was run. - Behavior changed: Yes (bounded cloud read admission, visible-prefix system columns, later commit TSO allocation and one-second persistence) - Does this need documentation: Yes (included docs/committed-tso.md)
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: The committed-TSO window check made getFlightInfoStatement pass through every FlightRuntimeException. Other Flight failures therefore lost the original INTERNAL wrapper, message prefix and cause chain, and unrelated status codes could reach clients unchanged. Only pass through an exception carrying the ERR_INCR_WINDOW_NOT_READY business code; retain the original wrapping for all other exceptions. ### Release note Preserve the existing Arrow Flight SQL error wrapping for failures other than ERR_INCR_WINDOW_NOT_READY. Window-not-ready errors still expose the retryable status and committed TSO details. ### Check List (For Author) - Test: Unit Test (all 7 DorisFlightSqlProducerTest tests passed via run-fe-ut.sh; the new cases reproduce the previous wrapping failures); FE Checkstyle passed with 0 violations - Behavior changed: Yes (restore the original INTERNAL wrapper for other Flight errors) - Does this need documentation: No (restore existing error handling)
…l reads ### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: A slow commit on one table holds the global committed TSO and rejects otherwise complete incremental windows on unrelated tables. Keep the durable-prefix fast path, reject ends after the current TSO immediately, and let intermediate windows wait for a fixed snapshot of registered transactions involving their tables. Capture the snapshot under the allocator lock and release that lock during the wait; real terminal notifications and reconciliation wake readers without another journal flush. Preserve the recovery guard and distinguish future/recovering windows from visibility wait timeouts through follower RPC, MySQL and Arrow Flight SQL. ### Release note Bounded strongly consistent cloud incremental reads can proceed above the durable committed TSO when their relevant transactions are finished. Visibility wait timeouts return error 5101 (ERR_INCR_VISIBLE_WAIT_TIMEOUT); future or recovering windows retain error 5100 (ERR_INCR_WINDOW_NOT_READY). Both include the current and committed TSO and retry details. ### Check List (For Author) - Test: 65 distinct focused FE unit tests; full FE build and Checkstyle; test_committed_tso SQL regression generated and verified; live MySQL and Flight statement/prepared checks on one master and two followers, including transaction visibility wakeup. - Behavior changed: Yes (table-scoped waiting above the durable prefix and distinct visibility-timeout error). - Does this need documentation: Yes (docs/committed-tso.md updated).
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Remove docs/committed-tso.md from the change as requested. The implementation and tests are unchanged. ### Release note None ### Check List (For Author) - Test: No need to test (documentation deletion only); git diff --check passed. - Behavior changed: No - Does this need documentation: No
luwei16
requested review from
924060929,
gavinchou,
liaoxin01,
morrySnow and
mymeiyi
as code owners
September 10, 2026 14:22
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
luwei16
changed the title
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
### What problem does this PR solve? Issue Number: None Related PR: apache#67820 Problem Summary: The committed TSO change adds two columns to information_schema.tso_status, but SchemaTableTest still expects four columns and fails FE unit CI. Expect all six columns and verify the names and positions of the two new columns while retaining the original column checks. ### Release note None ### Check List (For Author) - Test: Unit Test; reproduced the original SchemaTableTest failure, then passed all 7 SchemaTableTest and TsoStatusMetadataGeneratorTest cases with run-fe-ut.sh. FE Checkstyle and git diff --check passed. - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run cloudut |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #67181, #67594
Problem Summary:
Bounded cloud
@incrreads currently drain transactions by a transaction-ID watermark. This can wait for transactions whose eventual commit TSO falls outside the requested window. For example, a read ending at 22:21:24 can time out waiting for a transaction started at 22:21:25, even though that transaction cannot contribute rows to the window. A global committed-TSO check alone would also let a slow commit on one table block otherwise complete windows on unrelated tables.This PR tracks transactions when they receive a commit TSO and uses that information to establish the read boundary:
VISIBLEorABORTEDconfirmation; incomplete lazy commits and unknown RPC results continue to block advancement. A background worker reconciles old registrations with MetaService.tso_service_window_duration_mschanges from 5000 to 1000 ms.For strongly consistent cloud reads with an explicit
endTimestampon every incremental relation, compare the requested end with the TSO physical times:5100 / ERR_INCR_WINDOW_NOT_READY.5101 / ERR_INCR_VISIBLE_WAIT_TIMEOUTifchange_visible_timeout_msexpires.The wait releases the allocator lock and wakes on terminal transaction notifications. An unrelated table can proceed while another table holds the global prefix, and a successful wait does not require another journal flush. Cloud partition visible versions are still refreshed from MetaService. Classic, eventual-consistency, unbounded and mixed bounded/unbounded reads retain their existing paths.
information_schema.tso_statusaddsCOMMITTED_TSOandCOMMITTED_TSO_PHYSICAL_TIME. The latter is epoch milliseconds and identifies an end that requires no transaction wait across all tables; larger ends can still be readable after the table-specific check. Error details include the requested end, current and committed TSO/physical times, reason, timeout and retry delay. Follower RPC preserves both business codes. Arrow Flight SQL returnsUNAVAILABLEwith the correspondingdoris-error-codeanddoris-error-name; other errors keep their existing wrapping. Clients should retry the same window without advancing offsets on either error.Recovery and compatibility: A new master retains the persisted prefix, waits
tso_service_window_duration_ms + 1000ms, captures a fixed exclusive transaction-ID bound, and requires an instance-wide strict MetaService check to finish all earlier transactions, including expired transactions that are still running. Reads above the durable prefix return 5100 while recovery is incomplete. Upgrade MetaService first: FE requires explicit acknowledgement of the strict check. Older persisted records remain readable with an unknown committed prefix, and new FE RPC fields are optional; older followers classify 5101 as the original retryable 5100 until upgraded.The fixed recovery delay is a temporary assumption and cannot fence an old master that continues allocating TSOs. This PR does not implement Connector retry logic. The one-second combined journal record increases write frequency relative to the former five-second reservation window; no throughput benchmark is claimed. Queries with different ends on multiple incremental relations conservatively use their maximum end for the table set.
Release note
Improve bounded, strongly consistent cloud incremental reads by skipping transaction waits for windows covered by the durable committed TSO and waiting only for relevant registered transactions for newer windows. Expose committed TSO in
information_schema.tso_status, distinguish future/recovering windows (5100) from visibility wait timeouts (5101), and preserve retry details through MySQL and Arrow Flight SQL. The default TSO persistence interval is one second. Upgrade MetaService before FE to support strict recovery checks.Check List (For Author)
Test
test_committed_tsoandtest_binlog_changes_syntaxpassed. The updatedtest_committed_tsooutput was generated with the standard runner and verified by a second run.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)