[core][rest] Add branch merge support for append-only tables#7882
Open
JunRuiLee wants to merge 2 commits into
Open
[core][rest] Add branch merge support for append-only tables#7882JunRuiLee wants to merge 2 commits into
JunRuiLee wants to merge 2 commits into
Conversation
Add mergeBranch(sourceBranch, targetBranch) to Table and Catalog interfaces, with default/delegate implementations. Add branch-merge.enabled @immutable table option to CoreOptions. When enabled, the table is restricted to pure-append commits only, guaranteeing that branch merge is always safe. Add BranchMergeHandler interface and extend BranchManager with mergeBranch() and merge validation utilities.
Implement FileSystemBranchManager.mergeBranch() with: - Symmetric branch-merge.enabled validation on both branches - Append-only table validation (no primary keys) - Row-tracking consistency check between branches - Schema compatibility validation (fields and schema history) - Active-file diff to find branch-only files - DefaultBranchMergeHandler for committing merged files Enforce pure-append invariant when branch-merge.enabled=true: - SchemaValidation rejects primary keys, deletion vectors, and data evolution at table creation time - FileStoreCommitImpl rejects compaction and INSERT OVERWRITE at commit time Add Catalog and REST API support (CatalogBranchManager, RESTCatalog, POST /branches/merge endpoint). Add comprehensive tests covering positive merge, row-tracking, snapshot expiration, schema conflicts, and negative cases. Add branch merge documentation and REST OpenAPI spec.
JunRuiLee
force-pushed
the
branch_merge_final2_part1
branch
from
616f113 to
1a55a59
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR is the first part of #7863, focusing on the core APIs, implementation, and REST support for branch merge.
It adds
mergeBranch, which incrementally adds data files that exist only in the source branch to the target branch, without replacing either branch. Correctness is guarded by the immutable table option'branch-merge.enabled' = 'true', which enforces the pure-append invariant required by file-level branch merge: compaction andINSERT OVERWRITEare rejected, and deletion vectors are not supported.Tests
Added core tests:
AppendOnlySimpleTableTest#testMergeBranchAppendOnlySimpleTableTest#testMergeBranchMultipleTimesAppendOnlySimpleTableTest#testMergeBranchFailsOnStaleDuplicateCommitAppendOnlySimpleTableTest#testMergeBranchBidirectionalAppendOnlySimpleTableTest#testMergeBranchEmptyDiffAppendOnlySimpleTableTest#testMergeBranchSchemaConflictAppendOnlySimpleTableTest#testMergeBranchSchemaHistoryConflictAppendOnlySimpleTableTest#testMergeBranchRowTrackingTableAppendOnlySimpleTableTest#testMergeBranchRowTrackingMultipleTimesAppendOnlySimpleTableTest#testMergeBranchRowTrackingAfterTargetWritesAppendOnlySimpleTableTest#testMergeBranchRowTrackingBetweenNonMainBranchesAppendOnlySimpleTableTest#testMergeBranchRowTrackingMismatchAppendOnlySimpleTableTest#testMergeBranchRowTrackingStaleMergeAppendOnlySimpleTableTest#testMergeBranchSameBranchAppendOnlySimpleTableTest#testMergeBranchSamePartitionAppendOnlySimpleTableTest#testMergeBranchNonExistentBranchAppendOnlySimpleTableTest#testMergeBranchMultiBucketAppendOnlySimpleTableTest#testMergeBranchNonExistentTargetBranchAppendOnlySimpleTableTest#testMergeBranchBetweenNonMainBranchesAppendOnlySimpleTableTest#testMergeBranchWithoutBranchMergeEnabledAppendOnlySimpleTableTest#testMergeBranchAfterSnapshotExpirationAppendOnlySimpleTableTest#testBranchMergeEnabledRejectsOverwritePrimaryKeySimpleTableTest#testMergeBranchPrimaryKeyTableAdded REST tests:
RESTCatalogTest#testMergeBranchRESTCatalogTest#testBranches