[fix](ivm) Refresh the surviving partitions after an IVM baseline rebuild - #67802
Merged
Conversation
…uild Dropping a base-table partition marks the IVM baseline as broken, and partition sync then removes the very MV partitions that barrier named. handlePendingIvmBaselineRebuild derived its rebuild set from the intersection of the barrier with the current MV partitions, so that set was always empty: the task reported SUCCESS with refresh mode NOT_REFRESH and cleared the barrier without refreshing anything, leaving the delta accumulated on the surviving partitions unapplied while the MV claimed to be up to date. Treat the pending baseline rebuild as a pre-step instead of a terminal branch, so the attempt list still runs and the surviving partitions catch up in the same task. Key changes: - MTMVTask.handlePendingIvmBaselineRebuild rebuilds only the barrier partitions that still exist, releases the barrier and then lets the normal attempts run instead of returning early - the attempt list is rewritten in place: a lone COMPLETE attempt needs no pre-rebuild, and a complete baseline rebuild rewrites the list to COMPLETE rather than executing it inline - add MTMV.releaseIvmBaselineRebuild, which compare-and-clears on schemaChangeVersion and journals the new state immediately like the other ivmInfo mutations, so the IVM attempt that follows is not rejected by validateIvmRefreshStart - MTMVRelationManager.markIvmBaselineRebuild takes an explicit all-partitions-changed flag instead of inferring it from an empty partition map Unit Test: - test_ivm_partition_drop_live_delta: new, partitions added and dropped by hand with literal dates, asserts the MV matches the base table right after INCREMENTAL FALLBACK - test_ivm_partition_baseline_rebuild_dup_keys: new, covers the TRUNCATE path where the affected MV partition survives, on a duplicate-key MV so a double-applied delta would show up as extra rows - test_ivm_partition_baseline_rebuild, test_ivm_partition_sync_retry, test_ivm_partition_unique_key, test_ivm_partition_window_limit, test_ivm_partition_window_remove, test_ivm_one_row_relation_partitioned, test_ivm_strict_failure_partition_atomicity
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 17000 ms |
Contributor
TPC-DS: Total hot run time: 82896 ms |
Contributor
ClickBench: Total hot run time: 14.89 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
morrySnow
approved these changes
Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Dropping a base-table partition marks the IVM baseline as broken, because those rows disappear through metadata rather than through row binlog entries. Partition sync then removes the very MV partitions that barrier named, so
handlePendingIvmBaselineRebuildintersected the barrier with the current MV partitions and always got an empty set. The refresh reported SUCCESS with refresh modeNOT_REFRESHand cleared the barrier without refreshing anything, leaving the delta that had accumulated on the surviving partitions unapplied: the task claimed the MV was up to date while the MV was missing rows.Reproduction: drop a base partition, insert a row into a surviving partition, then run
REFRESH MATERIALIZED VIEW ... INCREMENTAL FALLBACK. The task succeeds, the expired MV partition is dropped, and the new row never reaches the MV.What changed
MTMVTask.handlePendingIvmBaselineRebuildis a pre-step now instead of a terminal branch. It rebuilds only the barrier partitions that still exist, releases the barrier, and then lets the normal attempt list run, so the surviving partitions catch up in the same task. Partitions the barrier named that partition sync already dropped need no rebuild: the partition and its IVM offsets are both gone, which is what satisfies those entries.COMPLETEattempt rebuilds everything anyway and needs no pre-rebuild; a complete baseline rebuild rewrites the list toCOMPLETEinstead of executing it inline.MTMV.releaseIvmBaselineRebuildreleases the barrier under aschemaChangeVersioncompare-and-clear and journals the new state immediately, like the otherivmInfomutations. Without the release the IVM attempt that follows would be rejected byvalidateIvmRefreshStart; without the compare-and-clear a concurrent base-table change would lose the barrier entry it had just recorded.MTMVRelationManager.markIvmBaselineRebuildtakes an explicit all-partitions-changed flag instead of inferring it from an empty partition map.The COMPLETE paths are unchanged. They end the task, so the existing
addTaskResultcleanup releases the barrier once the refresh succeeds.A strict
REFRESH ... INCREMENTAL, and a scheduled refresh of an MV declared withoutFALLBACK, still fail while the barrier is pending, because that check runs before partition sync. Relaxing it is left to a follow-up PR.Test
regression-test/suites/mtmv_p0/ivm/test_ivm_partition_drop_live_delta.groovy(new): partitions added and dropped by hand with literal dates and no dynamic-partition scheduler, so the case is deterministic. Asserts the MV matches the base table right afterINCREMENTAL FALLBACK. The scenario was reproduced on the pre-fix build, where the fallback task reported SUCCESS while the MV was missing the row written to the surviving partition.regression-test/suites/mtmv_p0/ivm/test_ivm_partition_baseline_rebuild_dup_keys.groovy(new): covers theTRUNCATEpath, where the affected MV partition survives and the pre-step really has something to rebuild. Runs on a duplicate-key MV with repeated identical rows, so a delta applied twice would show up as extra rows instead of hiding behind a unique key.test_ivm_partition_baseline_rebuild,test_ivm_partition_sync_retry,test_ivm_partition_unique_key,test_ivm_partition_window_limit,test_ivm_partition_window_remove,test_ivm_one_row_relation_partitioned,test_ivm_strict_failure_partition_atomicityTrace issue: #65418