Skip to content

[fix](analysis) Hide internal HAVING helper outputs - #67812

Merged
morrySnow merged 1 commit into
apache:masterfrom
morrySnow:fix/hide-having-helper-outputs
Sep 11, 2026
Merged

[fix](analysis) Hide internal HAVING helper outputs#67812
morrySnow merged 1 commit into
apache:masterfrom
morrySnow:fix/hide-having-helper-outputs

Conversation

@morrySnow

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

A scalar subquery that exposes one expression can be rejected as returning two columns when an aggregate is used only by its HAVING clause.

For example:

CREATE TABLE t (id SMALLINT) DISTRIBUTED BY RANDOM PROPERTIES ('replication_num' = '1');
INSERT INTO t VALUES (1);
SELECT (SELECT 1 FROM t HAVING SUM(id) > 0);

The fill-up analysis phase adds aggregate and missing-slot expressions to an internal project so that HAVING and ORDER BY can resolve them. In the aggregate path, it did not restore the original project afterwards. The helper aggregate therefore leaked into the public output, and scalar-subquery validation reported Found 2.

The rule now always adds a final project with the original output after HAVING and ORDER BY consume their helper slots. This keeps internal dependencies available during analysis while preserving the query's public output contract.

Tests cover the valid scalar subquery, an empty result caused by HAVING, and a genuine two-column scalar subquery that must still be rejected.

Release note

Fix false multi-column errors for scalar subqueries whose aggregate appears only in HAVING.

Check List (For Author)

  • Test: Unit Test and Regression Test
  • Behavior changed: Yes. Internal helper slots no longer appear in query output.
  • Does this need documentation: No

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16909 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e4e7818e62b95d04737e761ed45bc2367b80437, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17557	3113	3078	3078
q2	2069	254	221	221
q3	10255	869	517	517
q4	4669	247	210	210
q5	7674	559	386	386
q6	134	116	93	93
q7	523	512	383	383
q8	9245	891	948	891
q9	3387	2397	2380	2380
q10	6498	874	734	734
q11	398	203	187	187
q12	621	261	197	197
q13	18141	1509	1150	1150
q14	167	152	137	137
q15	q16	439	396	372	372
q17	1403	934	758	758
q18	3117	2228	2208	2208
q19	1258	898	820	820
q20	381	280	204	204
q21	5602	1748	1838	1748
q22	319	270	235	235
Total cold run time: 93857 ms
Total hot run time: 16909 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3457	3398	3396	3396
q2	508	383	376	376
q3	2197	2760	2133	2133
q4	1200	1157	893	893
q5	2140	2121	2124	2121
q6	164	119	87	87
q7	1050	918	856	856
q8	1585	1398	1389	1389
q9	3096	3067	3083	3067
q10	1855	1797	1613	1613
q11	353	272	252	252
q12	452	423	340	340
q13	1482	1518	1154	1154
q14	169	172	153	153
q15	q16	393	389	351	351
q17	3559	3268	3176	3176
q18	4778	4446	4671	4446
q19	838	891	864	864
q20	993	965	823	823
q21	3789	3050	3244	3050
q22	410	356	325	325
Total cold run time: 34468 ms
Total hot run time: 30865 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81323 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e4e7818e62b95d04737e761ed45bc2367b80437, data reload: false

query5	4271	415	338	338
query6	387	138	122	122
query7	4947	406	234	234
query8	289	129	117	117
query9	8686	2874	2889	2874
query10	397	220	186	186
query11	5374	1047	940	940
query12	121	70	72	70
query13	1208	467	304	304
query14	6065	2173	2068	2068
query14_1	1952	1964	1936	1936
query15	181	117	111	111
query16	910	379	367	367
query17	807	445	358	358
query18	2327	329	229	229
query19	165	139	112	112
query20	75	68	71	68
query21	201	105	91	91
query22	5394	5371	5272	5272
query23	6829	6181	6100	6100
query23_1	6178	6030	6209	6030
query24	7256	1080	768	768
query24_1	773	742	761	742
query25	404	287	240	240
query26	1219	227	129	129
query27	2775	410	256	256
query28	4685	1481	1484	1481
query29	914	418	330	330
query30	250	152	130	130
query31	837	393	332	332
query32	121	69	78	69
query33	461	213	179	179
query34	983	825	464	464
query35	387	402	327	327
query36	576	536	527	527
query37	116	82	67	67
query38	1034	851	837	837
query39	486	485	457	457
query39_1	440	444	449	444
query40	199	88	73	73
query41	54	50	53	50
query42	70	72	70	70
query43	240	235	209	209
query44	1005	521	529	521
query45	108	101	101	101
query46	729	860	526	526
query47	753	761	696	696
query48	304	309	224	224
query49	531	243	188	188
query50	711	259	194	194
query51	8067	7939	8148	7939
query52	68	68	63	63
query53	214	199	143	143
query54	220	154	157	154
query55	78	66	55	55
query56	213	163	169	163
query57	679	768	650	650
query58	212	161	163	161
query59	1212	1239	1097	1097
query60	243	180	154	154
query61	116	104	103	103
query62	353	198	180	180
query63	175	139	137	137
query64	2694	696	577	577
query65	1599	1644	1548	1548
query66	1919	285	207	207
query67	9982	9697	9626	9626
query68	2793	1134	716	716
query69	346	260	199	199
query70	669	605	615	605
query71	247	176	167	167
query72	2256	1673	1470	1470
query73	631	648	320	320
query74	1559	1202	1104	1104
query75	1179	1074	961	961
query76	2290	713	513	513
query77	250	257	205	205
query78	4012	3586	3219	3219
query79	1193	814	554	554
query80	1199	317	278	278
query81	489	160	136	136
query82	575	124	100	100
query83	315	210	191	191
query84	296	112	91	91
query85	948	332	288	288
query86	392	173	169	169
query87	1009	971	882	882
query88	2753	2095	2111	2095
query89	276	195	174	174
query90	1881	130	121	121
query91	127	126	112	112
query92	85	75	75	75
query93	1166	1067	713	713
query94	652	254	227	227
query95	531	343	238	238
query96	814	543	275	275
query97	1055	1072	990	990
query98	144	142	131	131
query99	423	338	311	311
Total cold run time: 175193 ms
Total hot run time: 81323 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.59 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5e4e7818e62b95d04737e761ed45bc2367b80437, data reload: false

query1	0.00	0.01	0.00
query2	0.08	0.04	0.04
query3	0.25	0.11	0.11
query4	1.61	0.09	0.10
query5	0.18	0.15	0.16
query6	1.27	0.68	0.69
query7	0.02	0.00	0.00
query8	0.05	0.02	0.02
query9	0.29	0.21	0.21
query10	0.34	0.37	0.36
query11	0.16	0.11	0.12
query12	0.15	0.13	0.12
query13	0.30	0.31	0.31
query14	0.44	0.45	0.45
query15	0.35	0.34	0.34
query16	0.23	0.22	0.20
query17	0.70	0.70	0.67
query18	0.17	0.17	0.17
query19	1.18	1.17	1.14
query20	0.01	0.01	0.01
query21	15.43	0.16	0.13
query22	5.07	0.04	0.05
query23	16.15	0.26	0.10
query24	2.98	0.31	0.24
query25	0.10	0.04	0.04
query26	0.80	0.16	0.12
query27	0.03	0.03	0.04
query28	3.59	0.54	0.26
query29	12.53	3.16	2.55
query30	0.24	0.11	0.12
query31	2.75	0.37	0.17
query32	3.53	0.32	0.23
query33	1.42	1.41	1.40
query34	15.33	2.19	1.78
query35	1.76	1.72	1.69
query36	0.44	0.29	0.28
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.12	0.07	0.08
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 90.33 s
Total hot run time: 14.59 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow changed the title [fix](nereids) Hide internal HAVING helper outputs [fix](error) Hide internal HAVING helper outputs Sep 11, 2026
@morrySnow morrySnow changed the title [fix](error) Hide internal HAVING helper outputs [fix](analysis) Hide internal HAVING helper outputs Sep 11, 2026
### What problem does this PR solve?

Problem Summary:
When a scalar subquery selected one public expression while an aggregate appeared only in HAVING, fill-up analysis exposed the aggregate helper slot as a second output. The scalar-subquery validator then rejected a valid one-column query.

This change restores the original projection output after HAVING or ORDER BY consumes helper slots, for both aggregate and non-aggregate branches. It preserves internal dependencies while keeping the public output contract unchanged. Tests cover a legal scalar subquery, an empty-result HAVING condition, and genuine multi-column rejection.

### Release note

Fix false multi-column errors for scalar subqueries whose aggregate appears only in HAVING.

### Check List (For Author)

- Test: Unit Test and Regression Test
- Behavior changed: Yes. Internal helper slots no longer appear in query output.
- Does this need documentation: No
@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@morrySnow
morrySnow force-pushed the fix/hide-having-helper-outputs branch from 5e4e781 to b7066be Compare September 11, 2026 06:15
@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16772 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7066be6095823fdd70af7aecc5009f8c9a89544, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17579	3097	3055	3055
q2	2071	258	223	223
q3	10269	854	498	498
q4	4667	244	201	201
q5	7683	552	387	387
q6	135	112	92	92
q7	515	489	389	389
q8	9219	892	961	892
q9	3420	2389	2363	2363
q10	6513	865	710	710
q11	429	194	175	175
q12	610	259	199	199
q13	18138	1522	1166	1166
q14	164	151	137	137
q15	q16	425	402	367	367
q17	1368	837	805	805
q18	3089	2241	2240	2240
q19	1249	913	774	774
q20	366	285	200	200
q21	5621	1663	1872	1663
q22	322	267	236	236
Total cold run time: 93852 ms
Total hot run time: 16772 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3495	3404	3399	3399
q2	496	401	390	390
q3	2233	2273	2189	2189
q4	1179	1157	885	885
q5	2164	2137	2090	2090
q6	173	119	87	87
q7	1037	932	862	862
q8	1577	1404	1394	1394
q9	3073	3081	3065	3065
q10	1856	1784	1587	1587
q11	353	267	247	247
q12	452	427	339	339
q13	1484	1512	1155	1155
q14	173	174	149	149
q15	q16	395	394	369	369
q17	3614	3280	3151	3151
q18	4754	4374	4721	4374
q19	875	833	898	833
q20	1012	978	853	853
q21	3841	3094	3247	3094
q22	403	352	329	329
Total cold run time: 34639 ms
Total hot run time: 30841 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81001 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7066be6095823fdd70af7aecc5009f8c9a89544, data reload: false

query5	4262	394	338	338
query6	377	134	126	126
query7	4967	413	225	225
query8	290	123	115	115
query9	8695	2859	2871	2859
query10	394	217	187	187
query11	5381	1028	896	896
query12	117	74	71	71
query13	1205	421	324	324
query14	6151	2189	2084	2084
query14_1	1977	1962	1975	1962
query15	168	115	112	112
query16	918	380	344	344
query17	768	441	359	359
query18	2312	315	226	226
query19	151	130	99	99
query20	69	66	69	66
query21	208	99	83	83
query22	5295	5206	5248	5206
query23	6751	6214	5986	5986
query23_1	6180	5932	6130	5932
query24	7252	1084	752	752
query24_1	772	774	767	767
query25	400	276	222	222
query26	1217	226	127	127
query27	2788	415	240	240
query28	4711	1490	1487	1487
query29	917	431	332	332
query30	254	151	132	132
query31	815	394	332	332
query32	129	67	69	67
query33	459	228	179	179
query34	992	814	470	470
query35	411	400	351	351
query36	565	561	494	494
query37	123	78	71	71
query38	1017	842	807	807
query39	486	472	480	472
query39_1	466	480	485	480
query40	201	94	79	79
query41	59	57	57	57
query42	75	74	71	71
query43	245	241	223	223
query44	1013	548	542	542
query45	126	109	106	106
query46	761	871	543	543
query47	768	773	708	708
query48	323	308	226	226
query49	546	255	198	198
query50	706	256	199	199
query51	8016	8153	7855	7855
query52	68	67	57	57
query53	196	198	149	149
query54	220	162	146	146
query55	74	62	57	57
query56	187	160	152	152
query57	662	633	650	633
query58	202	159	162	159
query59	1270	1243	1094	1094
query60	235	177	170	170
query61	119	117	122	117
query62	362	198	176	176
query63	169	137	135	135
query64	2697	701	659	659
query65	1649	1688	1601	1601
query66	1838	252	201	201
query67	9793	9785	9582	9582
query68	2989	1159	741	741
query69	349	225	206	206
query70	674	605	622	605
query71	254	171	166	166
query72	2315	1680	1481	1481
query73	635	612	329	329
query74	2003	1213	1120	1120
query75	1186	1091	961	961
query76	2380	712	498	498
query77	255	262	205	205
query78	3919	3634	3146	3146
query79	2308	835	568	568
query80	1608	324	263	263
query81	513	152	136	136
query82	803	124	99	99
query83	288	205	192	192
query84	312	106	90	90
query85	778	338	291	291
query86	389	181	165	165
query87	1015	953	889	889
query88	2780	2106	2122	2106
query89	288	191	174	174
query90	1993	127	127	127
query91	127	116	92	92
query92	72	72	59	59
query93	1435	1083	733	733
query94	650	248	181	181
query95	516	319	225	225
query96	827	592	264	264
query97	1056	1021	999	999
query98	165	143	128	128
query99	439	339	313	313
Total cold run time: 177715 ms
Total hot run time: 81001 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.59 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b7066be6095823fdd70af7aecc5009f8c9a89544, data reload: false

query1	0.00	0.00	0.00
query2	0.08	0.03	0.04
query3	0.24	0.11	0.09
query4	1.60	0.10	0.10
query5	0.17	0.16	0.16
query6	1.27	0.69	0.69
query7	0.03	0.01	0.01
query8	0.05	0.03	0.03
query9	0.29	0.21	0.21
query10	0.35	0.36	0.35
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.32	0.30	0.32
query14	0.45	0.45	0.44
query15	0.37	0.35	0.35
query16	0.21	0.21	0.23
query17	0.66	0.68	0.66
query18	0.18	0.15	0.15
query19	1.14	1.17	1.16
query20	0.02	0.00	0.01
query21	15.47	0.16	0.11
query22	5.06	0.04	0.04
query23	16.17	0.26	0.10
query24	3.01	0.33	0.27
query25	0.11	0.04	0.04
query26	0.82	0.16	0.11
query27	0.04	0.02	0.03
query28	3.74	0.52	0.27
query29	12.49	3.17	2.59
query30	0.25	0.12	0.11
query31	2.75	0.38	0.17
query32	3.52	0.33	0.23
query33	1.59	1.49	1.36
query34	15.35	2.24	1.77
query35	1.74	1.71	1.70
query36	0.49	0.30	0.29
query37	0.06	0.04	0.04
query38	0.05	0.03	0.02
query39	0.03	0.02	0.02
query40	0.11	0.08	0.09
query41	0.08	0.03	0.03
query42	0.03	0.03	0.03
query43	0.03	0.02	0.02
Total cold run time: 90.73 s
Total hot run time: 14.59 s

@morrySnow
morrySnow merged commit c017770 into apache:master Sep 11, 2026
33 of 35 checks passed
@morrySnow
morrySnow deleted the fix/hide-having-helper-outputs branch September 11, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants