[SPARK-56892][SQL] Bulk read optimization for Parquet DELTA_BINARY_PACKED decoding#56543
Open
iemejia wants to merge 2 commits into
Open
[SPARK-56892][SQL] Bulk read optimization for Parquet DELTA_BINARY_PACKED decoding#56543iemejia wants to merge 2 commits into
iemejia wants to merge 2 commits into
Conversation
…BINARY_PACKED decoding" This reverts commit c13302a.
### What changes were proposed in this pull request? Fix readIntegersAsLongs and readIntegersAsDoubles in VectorizedDeltaBinaryPackedReader to truncate each prefix-sum result to int before widening, matching the INT32 encoder's modular arithmetic. ### Why are the changes needed? The Parquet INT32 delta encoder (DeltaBinaryPackingValuesWriterForInteger) computes deltas using Java int arithmetic, which wraps on overflow. The bulk widened reader was performing the prefix sum in long space and writing the raw long result without narrowing back to int. When the delta overflows 32 bits (e.g. a sequence containing Int.MinValue), the reconstructed long value has the wrong sign -- Int.MinValue appears as +2147483648 instead of -2147483648. ### Does this PR introduce _any_ user-facing change? No. This fixes a correctness bug introduced by the bulk read optimization for readIntegersAsLongs/readIntegersAsDoubles. ### How was this patch tested? - ParquetTypeWideningSuite: IntegerType -> LongType, IntegerType -> DoubleType - ParquetDeltaEncodingInteger: new focused tests for modular delta overflow with both single-batch and split reads - VectorizedDeltaReaderBenchmark: new overflow-pattern cases for readIntegersAsLongs and readIntegersAsDoubles - Full ParquetDeltaEncodingInteger/Long suites (30 tests) - ParquetIOSuite UINT_64 tests ### Was this patch authored or co-authored using generative AI tooling? Yes. Assisted-by: GitHub Copilot:claude-opus-4.6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Re-apply the bulk read optimization for
VectorizedDeltaBinaryPackedReader(reverted in c13302a) with a fix for the INT32 widening bug that caused the CI failure.Commit 1 — Reapply the original optimization (revert of the revert):
readIntegers/readLongsvia prefix-sum +putInts/putLongsencodeUnsignedLongBigEndian)readIntegersAsLongsandreadIntegersAsDoublesoverridesCommit 2 — Fix the INT32 widening bug:
DeltaBinaryPackingValuesWriterForInteger) computes deltas using Java int arithmetic with modular overflow. The bulk widened readers (readIntegersAsLongs,readIntegersAsDoubles) were performing the prefix sum in long space and writing raw long results without truncating back to int. When delta overflow occurs (e.g. a sequence containingInt.MinValue), the reconstructed long has the wrong sign.This is the same content as #55919, which was merged and reverted due to this bug.
Why are the changes needed?
The bulk read path eliminates per-value lambda dispatch overhead and enables the JIT to better vectorize the inner unpacking loop. See #55919 for full benchmark results.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
ParquetTypeWideningSuite: IntegerType -> LongType, IntegerType -> DoubleTypeParquetDeltaEncodingInteger: new focused tests for modular delta overflowParquetDeltaEncodingInteger/Long: full suites (30 tests)ParquetIOSuite: UINT_64 testsVectorizedDeltaReaderBenchmark: full suite including new overflow casesWas this patch authored or co-authored using generative AI tooling?
Yes.
Assisted-by: GitHub Copilot:claude-opus-4.6
cc @LuciferYang @sunchao