Skip to content

HIVE-29641: Upgrade Calcite to 1.42.0 #6523

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zabetak merged 42 commits into from
Aug 24, 2026
Merged

HIVE-29641: Upgrade Calcite to 1.42.0 #6523

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
46cb44f
Bump Calcite version to 1.38, minor code adjustments:
rubenada Jun 3, 2026
71f80e0
Use non-deprecated version of RexBuilder#makeAbstractCast in HiveSqlS…
rubenada Jun 3, 2026
dce6775
HiveHepExtractRelNodeRule: do the conversion using a shuttle, instead…
rubenada Jun 3, 2026
8068e48
Update Calcite default charset system property (use UTF16 instead of …
rubenada Jun 4, 2026
43ebfd1
In MaterializationView rules, we cannot use HiveHepExtractRelNodeRule…
rubenada Jun 4, 2026
af5c91f
Fix HiveRelFieldTrimmer#rewriteGBConstantKeys: the condition on its j…
rubenada Jun 4, 2026
2084de3
Add asm dependency to parent pom, trying to fix the javadoc error see…
rubenada Jun 5, 2026
eb9fe81
Adjust test plans due to
rubenada Jun 8, 2026
b78ca04
Bump Calcite to 1.42 + required code adjustments:
rubenada Jun 8, 2026
f5bbac2
Adjust test plans due to change in representation of floating point v…
rubenada Jun 8, 2026
5b26a90
In CalcitePlanner genFilterRelNode: simplify filter condition before …
rubenada Jun 10, 2026
dd02a16
Move fix from previous commit as earlier as possible (incorrect binar…
rubenada Jun 10, 2026
b047ec3
Avoid incorrect comparison ($snapshotIdInputRef <= NULL) in HiveAugme…
rubenada Jun 11, 2026
adf3d9f
minor: adjust test plan, use ORDER BY ... <ordinal> instead or repeat…
rubenada Jun 12, 2026
aab25a3
Identify NOT BETWEEN expression in SearchTransformer to avoid plan ch…
rubenada Jun 15, 2026
8e60574
Previous change (identify NOT BETWEEN expression in SearchTransformer…
rubenada Jun 15, 2026
90be70b
Test adjustments: rowcount changes in TestExplainCBOFormattedCliDrive…
rubenada Jun 17, 2026
57d3e2d
Bump maven.shade.plugin.version to 3.6.2 to resolve druid-handler jav…
rubenada Jun 18, 2026
172520b
Override HiveTypeSystemImpl#deriveDecimalDivideType because CALCITE-6…
rubenada Jun 18, 2026
2f043ff
Add ASM dependency 9.9.1 for druid handler shade plugin (still trying…
rubenada Jun 18, 2026
7d1df93
Adjust test output stats_histogram.q.out
rubenada Jun 19, 2026
3d3c0c7
Rever changes on Jenkinsfile + Fix druid-handler shade ASM error by r…
rubenada Jun 19, 2026
7098526
Adjust test files: condition reordering, IS [NOT] NULL simplification…
rubenada Jun 19, 2026
c01d8e9
Due to "[CALCITE-6044] RelMetadataQuery should regard single-row rela…
rubenada Jun 23, 2026
323a62f
Test adjustments (input8.q.out, lineage2.q.out, lineage3.q.out, ppd_c…
rubenada Jun 25, 2026
bdfe1ae
Adjust test file cross_prod_1.q.out: Equivalent plan; expression "A.v…
rubenada Jun 25, 2026
31b3ec8
Adjust test file vector_aggregate_9.q.out: Decima cast is not removed.
rubenada Jun 29, 2026
29a1e0d
Test adjustment vector_coalesce.q.out: equivalent plan, with some IS_…
rubenada Jun 29, 2026
7b4527e
Test adjustment vector_interval_2.q.out: simplified predicates
rubenada Jun 29, 2026
2b5231e
Test adjustment join45.q.out, join47.q.out, mapjoin47.q, smb_mapjoin4…
rubenada Jun 29, 2026
947fddb
Test adjustment join46.q.out, mapjoin46.q.out, vectorized_join46.q.ou…
rubenada Jun 30, 2026
8b4fbf5
Test adjustment iceberg_bucket_map_join_1.q.out:
rubenada Jul 1, 2026
82997f6
Update tests files:
rubenada Jul 3, 2026
cc4115a
Add org.jooq:joou-java-6 to ql/pom.xml to resolve init-metastore CI e…
rubenada Jul 6, 2026
7d876e2
Add Calcite bug reference
rubenada Jul 14, 2026
fea0fca
Review:
rubenada Jul 17, 2026
ed6c635
Fix TestHivePointLookupOptimizerRule
rubenada Jul 17, 2026
d470bb6
minor
rubenada Aug 14, 2026
c26be48
Alter slightly the DISTINCT-version queries to restore back the origi…
rubenada Aug 14, 2026
8f01d60
Update comment
rubenada Aug 14, 2026
e2a2021
Test
rubenada Aug 14, 2026
53833fe
Revert test
rubenada Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions hbase-handler/src/test/results/positive/hbase_queries.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ STAGE PLANS:
alias: hbase_table_1
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: UDFToDouble(key) is not null (type: boolean)
predicate: key is not null (type: boolean)
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: UDFToDouble(key) (type: double)
Expand Down Expand Up @@ -604,7 +604,7 @@ STAGE PLANS:
alias: hbase_table_1
Statistics: Num rows: 1 Data size: 188 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: UDFToDouble(key) is not null (type: boolean)
predicate: key is not null (type: boolean)
Statistics: Num rows: 1 Data size: 188 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: int), value (type: string), UDFToDouble(key) (type: double)
Expand Down
20 changes: 12 additions & 8 deletions iceberg/iceberg-handler/src/test/queries/positive/iceberg_bucket_map_join_1.q
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,12 +58,13 @@ select * from target_table inner join
(select date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;

-- Filter on string_col (low-NDV) so the DISTINCT-side row estimate keeps MapJoin-from-Reducer as the cost-model winner.
explain
select * from target_table inner join
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where string_col = 'pipeline') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;
select * from target_table inner join
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where string_col = 'pipeline') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;

-- 2. MapJoin
Expand All @@ -78,12 +79,13 @@ select * from target_table inner join
(select date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;

-- Filter on string_col (low-NDV) so the DISTINCT-side row estimate keeps MapJoin-from-Reducer as the cost-model winner.
explain
select * from target_table inner join
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where string_col = 'pipeline') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;
select * from target_table inner join
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where string_col = 'pipeline') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;

-- 3. VectorBucketMapJoin
Expand All @@ -98,12 +100,13 @@ select * from target_table inner join
(select date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;

-- Filter on string_col (low-NDV) so the DISTINCT-side row estimate keeps MapJoin-from-Reducer as the cost-model winner.
explain
select * from target_table inner join
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where string_col = 'pipeline') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;
select * from target_table inner join
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where string_col = 'pipeline') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;

-- 4. VectorMapJoin
Expand All @@ -118,10 +121,11 @@ select * from target_table inner join
(select date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;

-- Filter on string_col (low-NDV) so the DISTINCT-side row estimate keeps MapJoin-from-Reducer as the cost-model winner.
explain
select * from target_table inner join
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where string_col = 'pipeline') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;
select * from target_table inner join
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where coalesce(decimal_col,'') = '50000000000000000005905545593') s
(select distinct date_col, 'pipeline' string_col, decimal_col from source_table where string_col = 'pipeline') s
on s.date_col = target_table.date_col AND s.string_col = target_table.string_col AND s.decimal_col = target_table.decimal_col;
103 changes: 56 additions & 47 deletions iceberg/iceberg-handler/src/test/results/positive/dynamic_partition_pruning.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1458,37 +1458,40 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcpart_double_hour_n0
filterExpr: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
filterExpr: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)
Statistics: Num rows: 2 Data size: 188 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
predicate: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)

@zabetak zabetak Jul 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new plan is equivalent but it seems that the simplification didn't lead to a significant improvement. The simplification here seems to have introduced a new Filter[col0 is not null] higher up so at the end of the day the number of comparisons remains the same.

@zabetak zabetak Aug 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this be restored with CALCITE-7722?

@rubenada rubenada Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. This is not an occurrence of point 3 (which will be mitigated with CALCITE-7722), but rather point 1.

@zabetak zabetak Aug 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if point 1 and point 3 are fully distinct. I suspect that both are due to different simplifications.

Statistics: Num rows: 1 Data size: 94 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: CAST( UDFToInteger((hr / 2.0D)) AS STRING) (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Filter Operator
predicate: _col0 is not null (type: boolean)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
keys: _col0 (type: string)
minReductionHashAggr: 0.4
mode: hash
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Dynamic Partitioning Event Operator
Target column: hr (string)
Target Input: srcpart_iceberg
Partition key expr: hr
Group By Operator
keys: _col0 (type: string)
minReductionHashAggr: 0.4
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Target Vertex: Map 1
Dynamic Partitioning Event Operator
Target column: hr (string)
Target Input: srcpart_iceberg
Partition key expr: hr
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Target Vertex: Map 1
Execution mode: vectorized
Reducer 2
Reduce Operator Tree:
Expand Down Expand Up @@ -1587,21 +1590,24 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcpart_double_hour_n0
filterExpr: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
filterExpr: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)
Statistics: Num rows: 2 Data size: 188 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
predicate: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)
Statistics: Num rows: 1 Data size: 94 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: CAST( UDFToInteger((hr / 2.0D)) AS STRING) (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Filter Operator
predicate: _col0 is not null (type: boolean)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: vectorized
Reducer 2
Reduce Operator Tree:
Expand Down Expand Up @@ -3480,37 +3486,40 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcpart_double_hour_n0
filterExpr: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
filterExpr: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)
Statistics: Num rows: 2 Data size: 188 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
predicate: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)
Statistics: Num rows: 1 Data size: 94 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: CAST( UDFToInteger((hr / 2.0D)) AS STRING) (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Filter Operator
predicate: _col0 is not null (type: boolean)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
keys: _col0 (type: string)
minReductionHashAggr: 0.4
mode: hash
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Dynamic Partitioning Event Operator
Target column: hr (string)
Target Input: srcpart_iceberg
Partition key expr: hr
Group By Operator
keys: _col0 (type: string)
minReductionHashAggr: 0.4
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Target Vertex: Map 1
Dynamic Partitioning Event Operator
Target column: hr (string)
Target Input: srcpart_iceberg
Partition key expr: hr
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Target Vertex: Map 1
Execution mode: vectorized
Reducer 2
Execution mode: vectorized
Expand Down
Loading
Loading