[fix](be) Reuse file block columns during v2 finalization#65589
Open
xylaaaaa wants to merge 1 commit into
Open
[fix](be) Reuse file block columns during v2 finalization#65589xylaaaaa wants to merge 1 commit into
xylaaaaa wants to merge 1 commit into
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#65560 Problem Summary: FileScannerV2 materializes file-local columns into a table output Block while the file-local Block still owns the same ColumnPtr. Converting the output to mutable columns at that point triggers copy-on-write deep clones, including MAP key, value, and offset subcolumns. A normal-read A/B reproduction with large_string_map.brotli.parquet retained about 2.1 GiB less RSS and reduced process high-water memory by about 606 MiB when only this ownership issue was fixed. Materialize all output mappings first, release the file-local owners, and only then request mutable output columns so direct mappings transfer their existing payload. ### Release note Reduce memory use when FileScannerV2 finalizes directly mapped external file columns. ### Check List (For Author) - Test: Manual test - A/B normal MAP read verified payload pointer reuse and lower RSS; STRING and MAP ownership unit tests added, pending execution in this clean worktree - Behavior changed: No. Output values are unchanged; direct mappings avoid an unnecessary COW clone. - Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29751 ms |
Contributor
TPC-DS: Total hot run time: 178751 ms |
Contributor
ClickBench: Total hot run time: 24.92 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #65560
Problem Summary:
FileScannerV2 reads physical columns into a file-local
Blockand then maps them into the table outputBlock. For a direct mapping, both blocks temporarily own the sameColumnPtr. The old finalization path callsIColumn::mutate()while the file-local owner is still alive, so COW recursively clones the column tree. For a large MAP this duplicates key, value, and offset payloads even though no schema conversion is needed.A normal-read A/B reproduction using
large_string_map.brotli.parquetshowed that applying only this ownership fix reduced retained RSS from 8.74 GiB to 6.68 GiB, reduced process HWM from 8.74 GiB to 8.15 GiB, and reduced runtime from 56.4s to 46.2s.The fix first materializes immutable output mappings, keeps file-local columns alive while table virtual columns are evaluated, then releases the file-local owners before requesting mutable output columns. Direct mappings therefore transfer their existing payload instead of deep-cloning it.
This PR intentionally does not add COUNT(column) pushdown or change Parquet/ORC decoding.
Release note
Reduce memory use when FileScannerV2 finalizes directly mapped external file columns.
Check List (For Author)