Skip to content

[fix](be) Reuse file block columns during v2 finalization#65589

Open
xylaaaaa wants to merge 1 commit into
apache:masterfrom
xylaaaaa:codex/fix-v2-block-ownership-transfer
Open

[fix](be) Reuse file block columns during v2 finalization#65589
xylaaaaa wants to merge 1 commit into
apache:masterfrom
xylaaaaa:codex/fix-v2-block-ownership-transfer

Conversation

@xylaaaaa

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #65560

Problem Summary:

FileScannerV2 reads physical columns into a file-local Block and then maps them into the table output Block. For a direct mapping, both blocks temporarily own the same ColumnPtr. The old finalization path calls IColumn::mutate() while the file-local owner is still alive, so COW recursively clones the column tree. For a large MAP this duplicates key, value, and offset payloads even though no schema conversion is needed.

A normal-read A/B reproduction using large_string_map.brotli.parquet showed that applying only this ownership fix reduced retained RSS from 8.74 GiB to 6.68 GiB, reduced process HWM from 8.74 GiB to 8.15 GiB, and reduced runtime from 56.4s to 46.2s.

The fix first materializes immutable output mappings, keeps file-local columns alive while table virtual columns are evaluated, then releases the file-local owners before requesting mutable output columns. Direct mappings therefore transfer their existing payload instead of deep-cloning it.

This PR intentionally does not add COUNT(column) pushdown or change Parquet/ORC decoding.

Release note

Reduce memory use when FileScannerV2 finalizes directly mapped external file columns.

Check List (For Author)

  • Test: Manual test
    • A/B normal MAP read verified payload pointer reuse and lower RSS
    • Added STRING payload ownership unit test
    • Added MAP key/value/offset ownership unit test
    • New clean-worktree unit-test build is pending
  • Behavior changed: No. Output values are unchanged; direct mappings avoid an unnecessary COW clone.
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: None

Related PR: apache#65560

Problem Summary: FileScannerV2 materializes file-local columns into a table output Block while the file-local Block still owns the same ColumnPtr. Converting the output to mutable columns at that point triggers copy-on-write deep clones, including MAP key, value, and offset subcolumns. A normal-read A/B reproduction with large_string_map.brotli.parquet retained about 2.1 GiB less RSS and reduced process high-water memory by about 606 MiB when only this ownership issue was fixed. Materialize all output mappings first, release the file-local owners, and only then request mutable output columns so direct mappings transfer their existing payload.

### Release note

Reduce memory use when FileScannerV2 finalizes directly mapped external file columns.

### Check List (For Author)

- Test: Manual test
    - A/B normal MAP read verified payload pointer reuse and lower RSS; STRING and MAP ownership unit tests added, pending execution in this clean worktree
- Behavior changed: No. Output values are unchanged; direct mappings avoid an unnecessary COW clone.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xylaaaaa xylaaaaa marked this pull request as ready for review July 14, 2026 09:34
Copilot AI review requested due to automatic review settings July 14, 2026 09:34
@xylaaaaa

Copy link
Copy Markdown
Contributor Author

run buildall

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29751 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 874483418dd9d68b9cfae67354b3da8503efbea1, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17591	4096	4108	4096
q2	2043	336	205	205
q3	10231	1424	821	821
q4	4686	477	340	340
q5	7504	891	574	574
q6	175	169	134	134
q7	780	830	623	623
q8	9339	1578	1583	1578
q9	5583	4347	4370	4347
q10	6822	1794	1477	1477
q11	500	370	315	315
q12	736	553	435	435
q13	18078	3394	2792	2792
q14	273	260	231	231
q15	q16	797	792	714	714
q17	1033	989	1005	989
q18	6990	5807	5682	5682
q19	1370	1235	1102	1102
q20	738	624	585	585
q21	6000	2745	2409	2409
q22	433	352	302	302
Total cold run time: 101702 ms
Total hot run time: 29751 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4375	4324	4315	4315
q2	298	320	210	210
q3	4628	5038	4422	4422
q4	2057	2146	1402	1402
q5	4383	4288	4277	4277
q6	231	173	129	129
q7	1746	1971	1803	1803
q8	2563	2222	2176	2176
q9	7979	8166	7785	7785
q10	4992	4651	4183	4183
q11	565	418	399	399
q12	774	777	561	561
q13	3263	3659	2944	2944
q14	309	302	280	280
q15	q16	709	758	657	657
q17	1359	1342	1330	1330
q18	7911	7357	7296	7296
q19	1123	1125	1078	1078
q20	2223	2211	1942	1942
q21	5722	4599	4383	4383
q22	515	444	406	406
Total cold run time: 57725 ms
Total hot run time: 51978 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 178751 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 874483418dd9d68b9cfae67354b3da8503efbea1, data reload: false

query5	4316	672	495	495
query6	465	242	215	215
query7	4928	607	325	325
query8	347	195	183	183
query9	8803	4058	4071	4058
query10	484	376	292	292
query11	5946	2351	2179	2179
query12	153	101	101	101
query13	1256	598	432	432
query14	6141	5273	4884	4884
query14_1	4262	4253	4215	4215
query15	223	207	179	179
query16	1035	447	444	444
query17	913	725	571	571
query18	2470	459	341	341
query19	202	187	149	149
query20	106	110	105	105
query21	231	154	138	138
query22	13517	13447	13414	13414
query23	17340	16560	16199	16199
query23_1	16481	16275	16281	16275
query24	7517	1785	1290	1290
query24_1	1310	1307	1301	1301
query25	595	466	396	396
query26	1319	351	208	208
query27	2645	561	394	394
query28	4494	2005	2012	2005
query29	1132	663	502	502
query30	345	255	227	227
query31	1124	1092	978	978
query32	131	64	66	64
query33	544	334	263	263
query34	1175	1118	645	645
query35	776	792	676	676
query36	1202	1173	1101	1101
query37	165	114	99	99
query38	1882	1711	1678	1678
query39	881	875	858	858
query39_1	852	834	838	834
query40	258	176	146	146
query41	72	74	68	68
query42	98	94	95	94
query43	335	333	281	281
query44	1414	788	762	762
query45	199	192	185	185
query46	1024	1196	764	764
query47	2190	2189	2067	2067
query48	421	425	290	290
query49	594	418	312	312
query50	1072	422	317	317
query51	10613	10661	10662	10661
query52	90	92	78	78
query53	267	274	206	206
query54	318	253	228	228
query55	76	73	69	69
query56	325	324	309	309
query57	1338	1323	1250	1250
query58	309	276	268	268
query59	1602	1619	1432	1432
query60	322	285	272	272
query61	203	153	154	153
query62	548	500	443	443
query63	237	198	209	198
query64	2853	1071	870	870
query65	4709	4666	4660	4660
query66	1815	502	377	377
query67	29321	29224	29162	29162
query68	3424	1507	893	893
query69	426	297	272	272
query70	1049	976	959	959
query71	343	328	325	325
query72	3077	2879	2345	2345
query73	827	768	444	444
query74	5071	4946	4736	4736
query75	2529	2514	2142	2142
query76	2368	1200	788	788
query77	347	372	283	283
query78	11983	11954	11310	11310
query79	1370	1134	718	718
query80	1294	577	496	496
query81	544	316	284	284
query82	904	158	120	120
query83	366	317	292	292
query84	325	163	128	128
query85	988	628	552	552
query86	435	298	297	297
query87	1832	1827	1752	1752
query88	3698	2770	2762	2762
query89	449	376	321	321
query90	1929	211	199	199
query91	204	192	161	161
query92	67	58	61	58
query93	1680	1530	986	986
query94	764	385	314	314
query95	796	483	480	480
query96	1018	789	362	362
query97	2642	2622	2515	2515
query98	222	207	201	201
query99	1095	1126	989	989
Total cold run time: 264738 ms
Total hot run time: 178751 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 874483418dd9d68b9cfae67354b3da8503efbea1, data reload: false

query1	0.00	0.01	0.00
query2	0.11	0.05	0.04
query3	0.25	0.14	0.14
query4	1.60	0.14	0.16
query5	0.24	0.22	0.22
query6	1.24	1.02	1.05
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.31	0.32
query10	0.53	0.54	0.55
query11	0.20	0.14	0.14
query12	0.19	0.15	0.14
query13	0.48	0.49	0.47
query14	1.01	1.00	1.03
query15	0.62	0.59	0.60
query16	0.30	0.34	0.31
query17	1.13	1.10	1.08
query18	0.22	0.21	0.21
query19	2.03	2.00	1.97
query20	0.02	0.02	0.01
query21	15.44	0.19	0.14
query22	4.98	0.06	0.05
query23	16.14	0.30	0.12
query24	2.98	0.42	0.32
query25	0.10	0.04	0.05
query26	0.72	0.20	0.16
query27	0.04	0.04	0.04
query28	3.52	0.96	0.54
query29	12.49	3.98	3.20
query30	0.28	0.16	0.15
query31	2.77	0.59	0.31
query32	3.23	0.59	0.49
query33	3.19	3.18	3.17
query34	15.41	4.23	3.52
query35	3.52	3.46	3.51
query36	0.56	0.45	0.44
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.04	0.03
query40	0.18	0.17	0.16
query41	0.08	0.03	0.04
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.55 s
Total hot run time: 24.92 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants