[fix](nereids) Respect aggregate null-row semantics in CASE rewrite by morrySnow · Pull Request #67818 · apache/doris

morrySnow · 2026-09-10T13:47:36Z

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem

agg(CASE WHEN predicate THEN value END) cannot always be replaced by
agg(value) over FILTER predicate. Aggregates such as array_agg preserve a
NULL element for an unmatched row, so filtering that row changes the result.

Root cause

The CASE rewrite did not require a semantic guarantee that the aggregate
ignores a complete input row whenever any aggregate argument is SQL NULL. The
same guarantee is also consumed when inferring not-null predicates below a
global aggregate, but it was not represented as an explicit class-wide
contract.

Reproduction

For an input containing one matching row and one non-matching row,
array_size(array_agg(CASE WHEN predicate THEN value END)) is 2: one value
and one NULL element. The old rewrite filtered the non-matching row and returned
1.

Fix

Gate CASE simplification on an explicit null-row semantic marker.
Define the marker as invariance under adding or removing any row in which at
least one aggregate argument is NULL. Consequently, an input consisting only
of such rows must produce the same result as empty input; this is independent
of whether the result type itself is nullable.
Audit every marked built-in across all signatures and modes. Generic unary,
multi-argument, and variadic backend wrappers skip a row before nested add
when any nullable argument is NULL. Manually constructed implementations are
marked only when their add path has the same behavior.
Verify multi-argument configuration paths including the TopN family,
two-argument collect_list, intersect_count, sequence functions,
window-funnel functions, and retention; their complete argument lists pass
through the nullable wrapper.
Verify DISTINCT paths. The distinct combinator is itself wrapped using the
original nullable argument types, so NULL tuples are skipped before entering
the distinct set. Multi-phase DISTINCT retains the same invariant.
Leave known NULL-preserving functions ArrayAgg, MapAgg, and MapAggV2
unmarked. Leave CountByEnum unmarked because it has different NULL handling.
Leave GroupConcat and MultiDistinctGroupConcat unmarked because their
class-wide ORDER BY signatures are not proven by the ordinary argument
wrapper contract.

Tests

CUSTOM_MVN=/usr/local/bin/mvn ./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.InferAggNotNullTest
- 11 tests, 0 failures, 0 errors
CUSTOM_MVN=/usr/local/bin/mvn DISABLE_BUILD_UI=ON ./build.sh --fe
- full FE build and Checkstyle succeeded
infer_agg_not_null
- force-generated output and normal run succeeded
eliminate_aggregate_casewhen
- force-generated output and normal run succeeded
- covers safe nullable and not-nullable identities, unsafe NULL-preserving
  functions, and positive/negative DISTINCT behavior

Release note

Fix incorrect results when simplifying CASE expressions inside NULL-preserving
aggregate functions.

Check List (For Author)

Test: Regression test and Unit Test
Behavior changed: Yes. Aggregate CASE simplification now runs only for
functions proven to ignore NULL-argument rows.
Does this need documentation: No

hello-stephen · 2026-09-10T13:47:43Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morrySnow · 2026-09-10T13:47:47Z

run buildall

hello-stephen · 2026-09-10T17:13:50Z

TPC-H: Total hot run time: 17081 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b17809e210175d3d261017f25b832dba9664aab, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17679	3057	3060	3057
q2	2085	254	261	254
q3	10216	932	510	510
q4	4668	263	207	207
q5	7662	562	386	386
q6	137	116	97	97
q7	541	514	390	390
q8	9227	885	883	883
q9	3463	2436	2449	2436
q10	6496	859	739	739
q11	400	203	183	183
q12	606	268	204	204
q13	18162	1527	1178	1178
q14	155	149	141	141
q15	q16	458	404	370	370
q17	1436	843	756	756
q18	3118	2314	2282	2282
q19	1284	927	811	811
q20	377	291	204	204
q21	5610	1753	1859	1753
q22	331	266	240	240
Total cold run time: 94111 ms
Total hot run time: 17081 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3436	3340	3331	3331
q2	509	393	388	388
q3	2291	2301	2249	2249
q4	1212	1177	912	912
q5	2210	2141	2147	2141
q6	176	121	89	89
q7	1031	952	855	855
q8	1607	1406	1413	1406
q9	3183	3154	3161	3154
q10	1887	1839	1671	1671
q11	355	271	258	258
q12	462	434	341	341
q13	1491	1546	1151	1151
q14	175	175	172	172
q15	q16	395	397	357	357
q17	3848	3407	3297	3297
q18	4912	4473	4983	4473
q19	937	860	869	860
q20	1014	987	836	836
q21	3875	3187	3235	3187
q22	388	351	330	330
Total cold run time: 35394 ms
Total hot run time: 31458 ms

hello-stephen · 2026-09-10T17:20:22Z

TPC-DS: Total hot run time: 83795 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b17809e210175d3d261017f25b832dba9664aab, data reload: false

query5	4249	411	338	338
query6	390	153	136	136
query7	4910	420	231	231
query8	295	130	126	126
query9	8702	2963	2922	2922
query10	398	229	185	185
query11	5393	1076	943	943
query12	118	75	73	73
query13	1201	475	332	332
query14	6152	2283	2174	2174
query14_1	2091	2037	2038	2037
query15	181	126	115	115
query16	923	383	370	370
query17	796	461	374	374
query18	2330	330	236	236
query19	176	148	124	124
query20	74	72	75	72
query21	204	102	91	91
query22	5434	5475	5417	5417
query23	6982	6335	6283	6283
query23_1	6113	6175	6156	6156
query24	7355	1113	787	787
query24_1	753	787	802	787
query25	426	306	262	262
query26	1240	243	135	135
query27	2759	421	249	249
query28	4689	1514	1530	1514
query29	929	436	362	362
query30	251	163	132	132
query31	818	407	341	341
query32	131	77	78	77
query33	463	223	189	189
query34	993	842	498	498
query35	412	407	355	355
query36	572	610	545	545
query37	122	84	74	74
query38	1012	880	855	855
query39	510	492	494	492
query39_1	489	489	467	467
query40	206	103	95	95
query41	61	60	58	58
query42	77	108	73	73
query43	241	249	213	213
query44	973	533	537	533
query45	110	105	102	102
query46	749	862	503	503
query47	749	766	728	728
query48	310	309	231	231
query49	539	244	197	197
query50	790	260	199	199
query51	8221	8296	8244	8244
query52	67	79	59	59
query53	246	232	149	149
query54	214	168	157	157
query55	73	63	55	55
query56	196	163	170	163
query57	692	670	678	670
query58	205	169	169	169
query59	1237	1271	1100	1100
query60	243	197	176	176
query61	126	137	102	102
query62	360	208	179	179
query63	174	141	137	137
query64	2622	700	660	660
query65	1656	1609	1655	1609
query66	1784	271	209	209
query67	9683	10055	10013	10013
query68	2776	1270	757	757
query69	343	229	212	212
query70	678	577	617	577
query71	243	187	172	172
query72	2258	1687	1492	1492
query73	657	612	344	344
query74	1581	1243	1144	1144
query75	1190	1113	983	983
query76	2273	708	518	518
query77	258	246	213	213
query78	3971	3779	3268	3268
query79	2850	795	593	593
query80	1608	317	264	264
query81	490	159	139	139
query82	612	119	97	97
query83	277	212	190	190
query84	292	110	90	90
query85	798	344	282	282
query86	385	171	171	171
query87	1012	985	904	904
query88	2842	2122	2126	2122
query89	284	197	176	176
query90	1945	120	131	120
query91	129	122	100	100
query92	84	72	67	67
query93	1595	1142	713	713
query94	628	241	232	232
query95	536	322	231	231
query96	838	591	275	275
query97	1037	1065	1063	1063
query98	166	137	139	137
query99	427	342	309	309
Total cold run time: 178120 ms
Total hot run time: 83795 ms

hello-stephen · 2026-09-10T17:24:46Z

ClickBench: Total hot run time: 14.7 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b17809e210175d3d261017f25b832dba9664aab, data reload: false

query1	0.01	0.00	0.00
query2	0.08	0.04	0.04
query3	0.24	0.10	0.11
query4	1.60	0.10	0.10
query5	0.18	0.16	0.15
query6	1.25	0.71	0.68
query7	0.03	0.00	0.01
query8	0.04	0.03	0.03
query9	0.29	0.22	0.21
query10	0.36	0.35	0.36
query11	0.16	0.11	0.12
query12	0.16	0.13	0.12
query13	0.31	0.30	0.31
query14	0.47	0.45	0.47
query15	0.36	0.36	0.34
query16	0.23	0.21	0.23
query17	0.73	0.72	0.69
query18	0.19	0.17	0.17
query19	1.17	1.25	1.12
query20	0.02	0.01	0.01
query21	15.44	0.16	0.13
query22	5.08	0.04	0.05
query23	16.17	0.25	0.10
query24	3.00	0.32	0.25
query25	0.12	0.04	0.04
query26	0.83	0.16	0.11
query27	0.03	0.02	0.04
query28	3.65	0.52	0.29
query29	12.49	3.23	2.57
query30	0.26	0.12	0.13
query31	2.75	0.36	0.17
query32	3.52	0.33	0.24
query33	1.56	1.65	1.40
query34	15.37	2.21	1.80
query35	1.80	1.77	1.74
query36	0.47	0.28	0.30
query37	0.06	0.04	0.04
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.12	0.07	0.06
query41	0.08	0.03	0.02
query42	0.03	0.03	0.03
query43	0.03	0.02	0.02
Total cold run time: 90.83 s
Total hot run time: 14.7 s

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Rewriting an aggregate over a CASE expression into the same aggregate over a filtered input is correct only when the aggregate ignores every row containing a NULL aggregate argument. The previous rule did not encode this semantic requirement, so NULL-preserving functions such as array_agg could return a different number of elements after the rewrite. Require an audited null-row trait before applying the rewrite. Classify built-in aggregates from their backend nullable-wrapper or explicit add-path behavior, keep NULL-preserving and incompletely proven functions unmarked, and document that the contract applies to every argument, signature, mode, and DISTINCT execution. Add plan and result coverage for both CASE simplification and aggregate-driven not-null inference. ### Release note Fix incorrect results when simplifying CASE expressions inside NULL-preserving aggregate functions. ### Check List (For Author) - Test: Regression test and Unit Test - InferAggNotNullTest - infer_agg_not_null - eliminate_aggregate_casewhen - Full FE build and Checkstyle - Behavior changed: Yes. Aggregate CASE simplification now runs only for functions proven to ignore NULL-argument rows. - Does this need documentation: No

morrySnow · 2026-09-12T10:58:45Z

run buildall

hello-stephen · 2026-09-12T11:23:02Z

TPC-H: Total hot run time: 16925 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 90b73ef6fce06cabaec2b22ae4f0c5783a367c92, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17608	3065	3053	3053
q2	2077	264	235	235
q3	10236	856	509	509
q4	4674	250	207	207
q5	7666	572	375	375
q6	136	111	92	92
q7	526	496	377	377
q8	9247	910	951	910
q9	3495	2389	2355	2355
q10	6499	840	730	730
q11	391	195	176	176
q12	608	263	196	196
q13	18139	1575	1157	1157
q14	154	144	136	136
q15	q16	434	400	374	374
q17	1353	847	789	789
q18	3065	2273	2232	2232
q19	1261	924	791	791
q20	383	284	194	194
q21	5600	1814	1813	1813
q22	323	261	224	224
Total cold run time: 93875 ms
Total hot run time: 16925 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3416	3324	3326	3324
q2	509	385	362	362
q3	2205	2340	2182	2182
q4	1193	1167	900	900
q5	2206	2483	2342	2342
q6	171	117	88	88
q7	1086	971	876	876
q8	1753	1453	1424	1424
q9	3334	3255	3314	3255
q10	1874	1819	1644	1644
q11	362	270	256	256
q12	462	439	339	339
q13	1536	1561	1209	1209
q14	166	167	175	167
q15	q16	420	403	372	372
q17	3778	3439	3387	3387
q18	4837	4517	4805	4517
q19	896	822	811	811
q20	986	958	823	823
q21	3681	2932	3113	2932
q22	367	328	316	316
Total cold run time: 35238 ms
Total hot run time: 31526 ms

hello-stephen · 2026-09-12T11:29:28Z

TPC-DS: Total hot run time: 82041 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 90b73ef6fce06cabaec2b22ae4f0c5783a367c92, data reload: false

query5	4244	417	346	346
query6	382	138	122	122
query7	4946	423	209	209
query8	290	123	117	117
query9	8676	2892	2897	2892
query10	399	217	183	183
query11	5368	1032	918	918
query12	124	69	66	66
query13	1181	436	298	298
query14	6062	2174	2062	2062
query14_1	1979	1954	1948	1948
query15	180	126	107	107
query16	910	365	338	338
query17	762	424	334	334
query18	2310	320	224	224
query19	155	134	99	99
query20	70	71	69	69
query21	203	99	83	83
query22	5397	5249	5336	5249
query23	6672	6208	5931	5931
query23_1	5953	6167	6050	6050
query24	7303	1079	780	780
query24_1	759	745	782	745
query25	397	269	231	231
query26	1255	232	126	126
query27	2794	401	251	251
query28	4711	1542	1500	1500
query29	902	405	329	329
query30	254	155	129	129
query31	819	398	334	334
query32	127	74	78	74
query33	468	209	179	179
query34	975	819	471	471
query35	406	403	346	346
query36	574	557	538	538
query37	121	79	71	71
query38	1004	845	821	821
query39	501	498	495	495
query39_1	473	462	475	462
query40	198	92	79	79
query41	56	65	54	54
query42	79	74	74	74
query43	239	241	209	209
query44	992	533	541	533
query45	112	106	103	103
query46	771	842	508	508
query47	748	770	710	710
query48	293	320	240	240
query49	544	248	203	203
query50	742	255	195	195
query51	8304	8435	8277	8277
query52	69	75	60	60
query53	194	198	144	144
query54	215	170	153	153
query55	73	60	56	56
query56	218	176	171	171
query57	689	701	645	645
query58	225	172	164	164
query59	1173	1165	1057	1057
query60	240	190	167	167
query61	124	121	124	121
query62	342	198	183	183
query63	166	139	140	139
query64	2692	662	534	534
query65	1581	1587	1526	1526
query66	1790	265	209	209
query67	9744	9903	9958	9903
query68	2764	1154	745	745
query69	336	240	190	190
query70	681	640	634	634
query71	255	183	163	163
query72	2255	1669	1563	1563
query73	652	574	330	330
query74	1573	1220	1143	1143
query75	1183	1110	963	963
query76	2278	694	510	510
query77	257	261	219	219
query78	3936	3746	3236	3236
query79	1197	808	568	568
query80	1382	338	271	271
query81	486	157	134	134
query82	619	121	101	101
query83	303	221	205	205
query84	293	110	90	90
query85	992	347	278	278
query86	386	175	173	173
query87	1007	985	894	894
query88	2788	2091	2101	2091
query89	285	197	173	173
query90	1806	125	127	125
query91	131	117	97	97
query92	79	70	71	70
query93	1345	1104	735	735
query94	635	255	227	227
query95	522	263	294	263
query96	817	561	255	255
query97	1075	1039	1018	1018
query98	148	153	130	130
query99	421	342	308	308
Total cold run time: 175028 ms
Total hot run time: 82041 ms

hello-stephen · 2026-09-12T11:33:50Z

ClickBench: Total hot run time: 14.84 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 90b73ef6fce06cabaec2b22ae4f0c5783a367c92, data reload: false

query1	0.00	0.01	0.00
query2	0.07	0.03	0.04
query3	0.25	0.12	0.11
query4	1.61	0.10	0.10
query5	0.17	0.16	0.15
query6	1.26	0.69	0.71
query7	0.03	0.00	0.00
query8	0.04	0.03	0.03
query9	0.28	0.21	0.21
query10	0.35	0.35	0.33
query11	0.16	0.12	0.12
query12	0.14	0.12	0.11
query13	0.31	0.31	0.29
query14	0.44	0.45	0.45
query15	0.37	0.36	0.34
query16	0.23	0.22	0.23
query17	0.70	0.68	0.68
query18	0.19	0.17	0.17
query19	1.22	1.23	1.12
query20	0.02	0.01	0.01
query21	15.44	0.16	0.11
query22	5.09	0.05	0.04
query23	16.21	0.25	0.10
query24	3.09	0.32	0.28
query25	0.10	0.04	0.03
query26	0.75	0.17	0.11
query27	0.04	0.03	0.04
query28	3.73	0.57	0.29
query29	12.43	3.21	2.57
query30	0.24	0.11	0.11
query31	2.76	0.36	0.17
query32	3.54	0.32	0.22
query33	1.55	1.56	1.59
query34	15.34	2.15	1.80
query35	1.81	1.76	1.76
query36	0.45	0.30	0.28
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.12	0.08	0.07
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 90.81 s
Total hot run time: 14.84 s

hello-stephen · 2026-09-12T12:48:19Z

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

morrySnow requested review from 924060929, englefly and starocean999 as code owners September 10, 2026 13:47

morrySnow changed the title ~~[fix](nereids) Respect aggregate null semantics in CASE rewrite~~ [fix](expr opt) Respect aggregate null semantics in CASE rewrite Sep 11, 2026

morrySnow marked this pull request as draft September 12, 2026 07:44

morrySnow force-pushed the fix/null-ignoring-aggregate-rewrite branch from 8b17809 to 90b73ef Compare September 12, 2026 10:58

morrySnow changed the title ~~[fix](expr opt) Respect aggregate null semantics in CASE rewrite~~ [fix](nereids) Respect aggregate null-row semantics in CASE rewrite Sep 12, 2026

morrySnow marked this pull request as ready for review September 12, 2026 10:58

Navigation Menu

Uh oh!

[fix](nereids) Respect aggregate null-row semantics in CASE rewrite - #67818

[fix](nereids) Respect aggregate null-row semantics in CASE rewrite#67818
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/null-ignoring-aggregate-rewrite

morrySnow commented Sep 10, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Sep 10, 2026

Uh oh!

morrySnow commented Sep 10, 2026

Uh oh!

hello-stephen commented Sep 10, 2026

Uh oh!

hello-stephen commented Sep 10, 2026

Uh oh!

hello-stephen commented Sep 10, 2026

Uh oh!

morrySnow commented Sep 12, 2026

Uh oh!

hello-stephen commented Sep 12, 2026

Uh oh!

hello-stephen commented Sep 12, 2026

Uh oh!

hello-stephen commented Sep 12, 2026

Uh oh!

hello-stephen commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

morrySnow commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Problem

Root cause

Reproduction

Fix

Tests

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Sep 10, 2026

Uh oh!

morrySnow commented Sep 10, 2026

Uh oh!

hello-stephen commented Sep 10, 2026

Uh oh!

hello-stephen commented Sep 10, 2026

Uh oh!

hello-stephen commented Sep 10, 2026

Uh oh!

morrySnow commented Sep 12, 2026

Uh oh!

hello-stephen commented Sep 12, 2026

Uh oh!

hello-stephen commented Sep 12, 2026

Uh oh!

hello-stephen commented Sep 12, 2026

Uh oh!

hello-stephen commented Sep 12, 2026

FE UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

morrySnow commented Sep 10, 2026 •

edited

Loading