Skip to content

[fix](subquery) Preserve generate child outputs for subqueries - #67807

Merged
morrySnow merged 1 commit into
apache:masterfrom
morrySnow:fix/preserve-generate-child-output
Sep 11, 2026
Merged

[fix](subquery) Preserve generate child outputs for subqueries#67807
morrySnow merged 1 commit into
apache:masterfrom
morrySnow:fix/preserve-generate-child-output

Conversation

@morrySnow

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

When a table-generating function contains a scalar subquery, NormalizeGenerate inserts a LogicalProject to materialize the subquery result. That project exposed only the new scalar-subquery aliases and discarded every output from the original child. Consequently, a projection above the LATERAL VIEW could no longer resolve base-table columns. A generating expression that referenced both a base-table column and the scalar subquery also lost its input slot.

For example, both of these valid forms failed analysis:

SELECT b.id, e
FROM base_table b
LATERAL VIEW explode_numbers((SELECT MAX(n) FROM base_table)) t AS e;

SELECT b.id, e
FROM base_table b
LATERAL VIEW explode_numbers(b.n + (SELECT MAX(n) FROM base_table)) t AS e;

What is changed and how does it work?

The materialization project now retains the original child outputs, in their existing order, before appending aliases for scalar-subquery results. This preserves the child slots and their expression IDs for both the generator and projections above it, while later projection pruning can still remove unused internal slots.

Focused unit tests cover both dependencies: a base-table column consumed above the generate node, and a base-table column consumed inside the generating expression. A result regression covers the same two query shapes end to end.

Release note

Fix analysis failures for LATERAL VIEW generating functions that combine scalar subqueries with base-table columns.

Check List (For Author)

  • Test:
    • Unit Test: NormalizeGenerateTest (2 tests)
    • Regression Test: nereids_p0/test_generate_subquery_output
    • Full FE build and Checkstyle
  • Behavior changed: Yes. Valid generating functions retain all required child slots during scalar-subquery normalization.
  • Does this need documentation: No

### What problem does this PR solve?

Problem Summary:
When a table-generating function contains a scalar subquery, NormalizeGenerate inserts a project that materializes the subquery. That project previously exposed only the new subquery aliases and discarded every output from the original child. As a result, an outer projection could no longer read base-table columns, and a generator expression that also referenced a base-table column had a missing input slot.

The normalization project now preserves the original child output before appending the subquery aliases. Existing slots keep their identities and order, while the generator can still consume the materialized subquery slots. Unit and regression coverage exercise both outer-output and generator-input dependencies.

### Release note

Fix LATERAL VIEW analysis failures when a generating function combines scalar subqueries with base-table columns.

### Check List (For Author)

- Test: Unit Test and Regression Test
- Behavior changed: Yes. Valid LATERAL VIEW queries retain the base-table slots required above and inside the generator.
- Does this need documentation: No
@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow morrySnow changed the title [fix](nereids) Preserve generate child outputs for subqueries [fix](subquery) Preserve generate child outputs for subqueries Sep 10, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16854 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f9d38f3c8800a4b50039696bb450c0a7d31a9050, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17565	3043	3033	3033
q2	2090	255	218	218
q3	10248	922	503	503
q4	4661	259	210	210
q5	7666	548	394	394
q6	145	119	99	99
q7	548	509	389	389
q8	9244	844	945	844
q9	3473	2479	2397	2397
q10	6508	854	725	725
q11	392	196	181	181
q12	618	256	198	198
q13	18128	1529	1176	1176
q14	158	147	138	138
q15	q16	437	403	371	371
q17	1486	922	769	769
q18	3099	2286	2283	2283
q19	1270	845	753	753
q20	394	292	203	203
q21	5643	1735	1886	1735
q22	342	275	235	235
Total cold run time: 94115 ms
Total hot run time: 16854 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3404	3338	3337	3337
q2	508	393	386	386
q3	2294	2353	2170	2170
q4	1203	1176	925	925
q5	2236	2161	2163	2161
q6	170	129	90	90
q7	1021	986	897	897
q8	1617	1402	1411	1402
q9	3172	3164	3143	3143
q10	1878	1842	1643	1643
q11	366	272	254	254
q12	452	440	344	344
q13	1500	1535	1184	1184
q14	185	170	164	164
q15	q16	408	396	364	364
q17	3611	3394	3270	3270
q18	4853	4501	4807	4501
q19	1016	852	867	852
q20	1012	961	828	828
q21	3890	3231	3251	3231
q22	406	344	331	331
Total cold run time: 35202 ms
Total hot run time: 31477 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 83227 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f9d38f3c8800a4b50039696bb450c0a7d31a9050, data reload: false

query5	4254	416	333	333
query6	385	138	127	127
query7	4949	422	218	218
query8	287	127	116	116
query9	8695	2924	2916	2916
query10	420	218	185	185
query11	5370	1043	943	943
query12	117	74	70	70
query13	1195	440	340	340
query14	6099	2227	2118	2118
query14_1	2010	1985	1978	1978
query15	175	126	111	111
query16	921	385	313	313
query17	793	475	356	356
query18	2341	330	240	240
query19	166	142	110	110
query20	75	74	80	74
query21	203	103	88	88
query22	5368	5346	5400	5346
query23	6783	6272	6273	6272
query23_1	6132	6302	6172	6172
query24	7320	1131	786	786
query24_1	788	796	809	796
query25	427	297	263	263
query26	1234	241	132	132
query27	2772	433	261	261
query28	4680	1514	1503	1503
query29	921	444	361	361
query30	247	159	129	129
query31	833	407	342	342
query32	127	77	73	73
query33	462	238	189	189
query34	976	841	498	498
query35	414	411	355	355
query36	575	573	511	511
query37	127	81	76	76
query38	1010	859	836	836
query39	498	494	472	472
query39_1	474	463	472	463
query40	204	93	84	84
query41	64	56	57	56
query42	76	76	82	76
query43	241	242	214	214
query44	985	576	545	545
query45	115	104	98	98
query46	773	804	539	539
query47	763	779	731	731
query48	321	317	232	232
query49	545	246	189	189
query50	758	270	195	195
query51	8113	8276	8202	8202
query52	72	73	64	64
query53	196	204	175	175
query54	249	183	156	156
query55	78	62	59	59
query56	180	169	160	160
query57	746	666	635	635
query58	199	169	156	156
query59	1231	1251	1105	1105
query60	228	234	191	191
query61	115	114	112	112
query62	340	206	194	194
query63	167	144	142	142
query64	2706	690	605	605
query65	1634	1666	1620	1620
query66	1806	256	192	192
query67	10254	9883	9865	9865
query68	2875	1151	760	760
query69	339	208	193	193
query70	669	624	637	624
query71	246	176	158	158
query72	2266	1664	1508	1508
query73	634	557	333	333
query74	1849	1248	1144	1144
query75	1196	1121	980	980
query76	2306	718	535	535
query77	252	252	213	213
query78	4230	3713	3319	3319
query79	2736	815	575	575
query80	1601	334	270	270
query81	519	165	134	134
query82	615	124	99	99
query83	278	209	190	190
query84	294	109	89	89
query85	799	338	271	271
query86	470	172	176	172
query87	1015	990	918	918
query88	2873	2108	2113	2108
query89	278	200	178	178
query90	1992	134	134	134
query91	129	116	97	97
query92	91	72	73	72
query93	1613	1187	701	701
query94	621	249	184	184
query95	516	251	234	234
query96	814	608	280	280
query97	1080	1097	1051	1051
query98	164	142	131	131
query99	420	351	313	313
Total cold run time: 179074 ms
Total hot run time: 83227 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f9d38f3c8800a4b50039696bb450c0a7d31a9050, data reload: false

query1	0.01	0.01	0.00
query2	0.08	0.04	0.04
query3	0.26	0.12	0.10
query4	1.60	0.10	0.09
query5	0.18	0.16	0.18
query6	1.25	0.71	0.72
query7	0.03	0.01	0.01
query8	0.05	0.02	0.03
query9	0.28	0.21	0.22
query10	0.37	0.37	0.34
query11	0.17	0.11	0.12
query12	0.14	0.12	0.12
query13	0.32	0.31	0.31
query14	0.46	0.45	0.46
query15	0.38	0.36	0.35
query16	0.22	0.20	0.21
query17	0.70	0.71	0.68
query18	0.19	0.17	0.18
query19	1.24	1.20	1.17
query20	0.01	0.01	0.01
query21	15.45	0.16	0.12
query22	5.06	0.04	0.04
query23	16.18	0.27	0.10
query24	3.02	0.35	0.24
query25	0.11	0.05	0.04
query26	0.72	0.17	0.14
query27	0.04	0.03	0.03
query28	3.64	0.54	0.27
query29	12.49	3.19	2.58
query30	0.25	0.11	0.12
query31	2.75	0.38	0.18
query32	3.50	0.32	0.25
query33	1.40	1.47	1.53
query34	15.36	2.20	1.80
query35	1.79	1.75	1.76
query36	0.46	0.29	0.29
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.12	0.08	0.08
query41	0.07	0.02	0.02
query42	0.04	0.02	0.03
query43	0.03	0.02	0.03
Total cold run time: 90.55 s
Total hot run time: 14.89 s

@morrySnow
morrySnow merged commit 30d83e5 into apache:master Sep 11, 2026
42 of 44 checks passed
@morrySnow
morrySnow deleted the fix/preserve-generate-child-output branch September 11, 2026 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants