Repository navigation
Support i128 and i256 decimals in DecimalByteParts encoding - #9834
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DecimalBytePartsArraystored the whole unscaled decimal value in one signed integer child. That capped it at values that fit in 64 bits. This PR adds support fori128andi256decimals.Each value is now split into a signed most significant part (MSP) plus up to three unsigned 64-bit lower parts. Every part is an independent child array, so each one compresses on its own. The frozen
vortex.decimal_byte_partsfile format is untouched. Arrays with lower parts serialize under a newvortex.decimal_byte_parts.v2format owned by a plugin.This is the integration branch for three reviewed sub-PRs: #9808 (splitting and assembly), #9809 (array and kernels), and #9810 (serde plugin).
Representation
i8,i16,i32,i64i128i64MSP holding the high 64 bits, plus oneu64lower part.i256i64MSP holding the high 64 bits, plus threeu64lower parts.Parts are ordered most significant first. The MSP carries the sign and the null mask. Lower parts are non-nullable unsigned integers. A lower part may use a narrower dtype such as
u8,u16, oru32when its values fit. Its position still counts as a full 64-bit window. Splitting writes zeroes at null positions so stray bytes in null slots do not hurt compression of the lower parts.Splitting and assembly
DecimalByteParts::encodesplits aDecimalArrayinto parts.split_decimalexposes the raw parts for callers that want to build the array themselves.Assembly picks a path from the number of lower parts:
i128.i256. The MSP is sign-extended into the high 128 bits.Assembly casts narrowed lower parts back to
u64first. Thei256assembly loop vectorizes on local ARM64 builds. Markingi256's shifts#[inline]removed three out-of-line calls per row.#[inline]Medians of five alternating release runs.
From<i64>andFrom<u64>fori256are added invortex-array.Compute
execute::<DecimalArray>reassembles the canonical array from all parts. The compare, filter, is-constant, and take kernels understand lower parts. Slice and mask apply per child.Two limits are documented in code. Take with nullable indices on an array with lower parts falls back to canonical execution, because taking each part would make the lower parts nullable. The CUDA kernel rejects arrays with lower parts, because GPU reassembly is not implemented yet.
Serialization
DecimalBytePartsPluginowns both wire formats and picks one from the array layout.vortex.decimal_byte_partsvortex.decimal_byte_parts.v2The v1 format is frozen. Its metadata and decoder live in
plugin/v1.rsand are byte-identical to what shipped. The v1 decoder rejects any payload that claims lower parts.The v2 format records the MSP's integer type and one integer type per lower part. The lower part count is the length of that list. The decoder validates every type and restores each child with its recorded dtype. The v2 format itself accepts zero lower parts. The plugin only chooses it when lower parts are present, so files stay readable by older readers whenever possible.
The in-memory encoding ID is now
vortex.decimal_byte_parts.v2. The registry maps both wire IDs to the plugin, so existing v1 files read through it with no migration.No edition declares the v2 format yet. Writing an array with lower parts under an edition that does not permit v2 fails with an explicit error rather than silently falling back.
Compression
The BtrBlocks decimal scheme still narrows decimals that fit in
i64and wraps them in a single-part array. Wide decimals stay canonical. Nothing in this PR writes the v2 format through the compressor.The scheme now declares
vortex.decimal_byte_partsas its produced encoding rather than the in-memory ID. Since #9914 that list holds the serialized IDs a scheme writes, and this scheme only ever writes the frozen format. Without that change every writer that filters schemes by edition would drop the decimal scheme, because no edition permits the in-memory v2 name.API Changes
Breaking. Registering
DecimalBytePartsdirectly no longer supports serde for either format. Replacesession.arrays().register(DecimalByteParts)withsession.arrays().register(DecimalBytePartsPlugin).vortex_decimal_byte_parts::initializealready does this.Breaking.
dbp_encodeis replaced byDecimalByteParts::encode.Breaking.
DecimalBytesPartsMetadatais no longer public.DecimalBytePartsV2Metadatais exposed instead.The in-memory encoding ID string changed from
vortex.decimal_byte_partstovortex.decimal_byte_parts.v2. This affects display and trace output, not files.New public items:
DecimalByteParts::try_new_with_lower_parts,DecimalByteParts::encode,split_decimal,DecimalParts,DecimalBytePartsPlugin,decimal_byte_parts_v1_id, anddecimal_byte_parts_v2_id.