[fix](ivm) Stop the incremental delta from reading partitions the MV dropped - #67814
Open
yujun777 wants to merge 1 commit into
Open
[fix](ivm) Stop the incremental delta from reading partitions the MV dropped#67814yujun777 wants to merge 1 commit into
yujun777 wants to merge 1 commit into
Conversation
…dropped partition_sync_limit keeps only a recent slice of each base table's partitions, so the MV can be missing a partition the base table still has. The incremental delta still read that base partition through its stream and through the join-opposite snapshot, so a change to a non-partitioned dimension produced delta rows for dates the MV has no partition for. The insert failed with "no partition for this tuple", and since re-syncing cannot bring an expired partition back, the task burned its retries and failed; the write being atomic, the partitions the MV does keep were not repaired either. Key changes: - add MTMVPartitionUtil.generateRelatedBasePartitionIds, which returns the base partitions the MV's partition definition keeps, per base table partitioned by the MV's partition column - IvmIncrRefreshManager passes that set to IvmRewriteContext.incremental, and IvmDeltaRewriter applies it as an upper bound on the partitions each base table may be read from, intersected with ivm_partition_window_limit when that is set too - the bound is the partition set the MV is aligned to rather than the partitions it already has, so a base partition whose MV partition partition sync has yet to add stays readable and the refresh still recovers it by syncing - a base table absent from the set keeps its full read, and a set that covers every partition of its table leaves the plan untouched, so an MV that mirrors all of its base partitions is unaffected - add MTMVPropertyUtil.hasPartitionSyncLimit, the gate that keeps the whole restriction off MVs without the property - remove the nonConcurrent group from seven mtmv_p0/ivm suites that use no debug points and no global variables or config, so they run in the parallel pool again Unit Test: - test_ivm_partition_sync_limit: new, the MV keeps only a recent slice of the base partitions and a late-arriving dimension row must repair the partition it keeps without failing on the one it dropped - test_ivm_partition_sync_limit_with_window: new, partition_sync_limit and ivm_partition_window_limit together, the readable base partitions are their intersection - testIncrementalScopeRestrictsDeltaToScopePartitions, testIncrementalScopeCoveringAllPartitionsLeavesScanUnchanged, testIncrementalScopeWithoutPartitionsProducesEmptyRelation - testHasPartitionSyncLimit, testGenerateRelatedBasePartitionIdsWithoutSyncLimit, testGenerateRelatedBasePartitionIdsOnSelfManageMv, testGenerateRelatedBasePartitionIdsWithoutMvPartitionInfo - test_ivm_partition_sync_limit, test_ivm_partition_sync_limit_with_window, MTMVPartitionUtilTest, IvmDeltaRewriterTest, IvmIncrRefreshManagerTest
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 17013 ms |
Contributor
TPC-DS: Total hot run time: 83115 ms |
Contributor
ClickBench: Total hot run time: 14.83 s |
Contributor
Author
|
run vault_p0 |
Contributor
Author
|
run feut |
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
partition_sync_limitkeeps only a recent slice of each base table's partitions, so an MV can be missing a partition its base table still has. The incremental delta read that base partition anyway, through the delta scan and through the join-opposite snapshot. A change to a non-partitioned dimension therefore produced delta rows for dates the MV has no partition for, and the insert failed withno partition for this tuple.Re-syncing cannot bring an expired partition back, so the retry loop in
executeIvmAttemptcould never recover: the task failed after exhausting its attempts, and because the write is atomic, the partitions the MV does keep were left unrepaired as well. A strictREFRESH ... INCREMENTAL, and a scheduled refresh of an MV declared withoutFALLBACK, fail outright.What changed
The incremental delta now reads only the base partitions the MV's partition definition keeps.
MTMVPartitionUtil.generateRelatedBasePartitionIdsreturns that partition set per base table partitioned by the MV's partition column, and an empty value whenpartition_sync_limitis not set, which is the only property that can leave the MV without a base partition.IvmIncrRefreshManagerpasses it toIvmRewriteContext.incremental, andIvmDeltaRewriterapplies it as an upper bound on the partitions each base table may be read from, intersected withivm_partition_window_limitwhen that is set too.The COMPLETE paths are unchanged.
The PR also removes the
nonConcurrentgroup from sevenmtmv_p0/ivmsuites that use neither debug points nor global variables or config, so they run in the parallel pool again.Test
regression-test/suites/mtmv_p0/ivm/test_ivm_partition_sync_limit.groovy(new): the MV keeps only a recent slice of the base partitions, and a late-arriving dimension row must repair the partition it keeps without failing on the one it dropped. Partitions are added by hand with literal dates and no dynamic-partition scheduler, and the expectation does not depend on the run date.regression-test/suites/mtmv_p0/ivm/test_ivm_partition_sync_limit_with_window.groovy(new):partition_sync_limitandivm_partition_window_limittogether, the readable base partitions are their intersection.testIncrementalScopeRestrictsDeltaToScopePartitions,testIncrementalScopeCoveringAllPartitionsLeavesScanUnchanged,testIncrementalScopeWithoutPartitionsProducesEmptyRelationtestHasPartitionSyncLimit,testGenerateRelatedBasePartitionIdsWithoutSyncLimit,testGenerateRelatedBasePartitionIdsOnSelfManageMv,testGenerateRelatedBasePartitionIdsWithoutMvPartitionInfotest_ivm_partition_sync_limit,test_ivm_partition_sync_limit_with_window,MTMVPartitionUtilTest,IvmDeltaRewriterTest,IvmIncrRefreshManagerTestTrace issue: #65418