Skip to content

Support i128 and i256 decimals in DecimalByteParts encoding - #9834

Merged
mhk197 merged 5 commits into
developfrom
mk/dbp-v2-feature
Sep 18, 2026
Merged

mhk197 merged 5 commits into
developfrom
mk/dbp-v2-feature

Conversation

@mhk197

@mhk197 mhk197 commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

DecimalBytePartsArray stored the whole unscaled decimal value in one signed integer child. That capped it at values that fit in 64 bits. This PR adds support for i128 and i256 decimals.

Each value is now split into a signed most significant part (MSP) plus up to three unsigned 64-bit lower parts. Every part is an independent child array, so each one compresses on its own. The frozen vortex.decimal_byte_parts file format is untouched. Arrays with lower parts serialize under a new vortex.decimal_byte_parts.v2 format owned by a plugin.

This is the integration branch for three reviewed sub-PRs: #9808 (splitting and assembly), #9809 (array and kernels), and #9810 (serde plugin).

Representation

Decimal storage Children
i8, i16, i32, i64 Signed MSP only. Shares the original value buffer.
i128 i64 MSP holding the high 64 bits, plus one u64 lower part.
i256 i64 MSP holding the high 64 bits, plus three u64 lower parts.

Parts are ordered most significant first. The MSP carries the sign and the null mask. Lower parts are non-nullable unsigned integers. A lower part may use a narrower dtype such as u8, u16, or u32 when its values fit. Its position still counts as a full 64-bit window. Splitting writes zeroes at null positions so stray bytes in null slots do not hurt compression of the lower parts.

Splitting and assembly

DecimalByteParts::encode splits a DecimalArray into parts. split_decimal exposes the raw parts for callers that want to build the array themselves.

Assembly picks a path from the number of lower parts:

  • None. Reuse the MSP buffer as decimal storage without copying.
  • One. Combine the MSP and the lower part into an i128.
  • Two. The lower parts form the low 128 bits of an i256. The MSP is sign-extended into the high 128 bits.
  • Three. The MSP and the first lower part form the high 128 bits. The remaining two form the low 128 bits.

Assembly casts narrowed lower parts back to u64 first. The i256 assembly loop vectorizes on local ARM64 builds. Marking i256's shifts #[inline] removed three out-of-line calls per row.

Rows With #[inline] Without
1,024 0.917 µs 6.207 µs
8,192 6.332 µs 48.540 µs

Medians of five alternating release runs. From<i64> and From<u64> for i256 are added in vortex-array.

Compute

execute::<DecimalArray> reassembles the canonical array from all parts. The compare, filter, is-constant, and take kernels understand lower parts. Slice and mask apply per child.

Two limits are documented in code. Take with nullable indices on an array with lower parts falls back to canonical execution, because taking each part would make the lower parts nullable. The CUDA kernel rejects arrays with lower parts, because GPU reassembly is not implemented yet.

Serialization

DecimalBytePartsPlugin owns both wire formats and picks one from the array layout.

Array layout Serialized ID
MSP only vortex.decimal_byte_parts
MSP plus one to three lower parts vortex.decimal_byte_parts.v2

The v1 format is frozen. Its metadata and decoder live in plugin/v1.rs and are byte-identical to what shipped. The v1 decoder rejects any payload that claims lower parts.

The v2 format records the MSP's integer type and one integer type per lower part. The lower part count is the length of that list. The decoder validates every type and restores each child with its recorded dtype. The v2 format itself accepts zero lower parts. The plugin only chooses it when lower parts are present, so files stay readable by older readers whenever possible.

The in-memory encoding ID is now vortex.decimal_byte_parts.v2. The registry maps both wire IDs to the plugin, so existing v1 files read through it with no migration.

No edition declares the v2 format yet. Writing an array with lower parts under an edition that does not permit v2 fails with an explicit error rather than silently falling back.

Compression

The BtrBlocks decimal scheme still narrows decimals that fit in i64 and wraps them in a single-part array. Wide decimals stay canonical. Nothing in this PR writes the v2 format through the compressor.

The scheme now declares vortex.decimal_byte_parts as its produced encoding rather than the in-memory ID. Since #9914 that list holds the serialized IDs a scheme writes, and this scheme only ever writes the frozen format. Without that change every writer that filters schemes by edition would drop the decimal scheme, because no edition permits the in-memory v2 name.

API Changes

Breaking. Registering DecimalByteParts directly no longer supports serde for either format. Replace session.arrays().register(DecimalByteParts) with session.arrays().register(DecimalBytePartsPlugin). vortex_decimal_byte_parts::initialize already does this.

Breaking. dbp_encode is replaced by DecimalByteParts::encode.

Breaking. DecimalBytesPartsMetadata is no longer public. DecimalBytePartsV2Metadata is exposed instead.

The in-memory encoding ID string changed from vortex.decimal_byte_parts to vortex.decimal_byte_parts.v2. This affects display and trace output, not files.

New public items: DecimalByteParts::try_new_with_lower_parts, DecimalByteParts::encode, split_decimal, DecimalParts, DecimalBytePartsPlugin, decimal_byte_parts_v1_id, and decimal_byte_parts_v2_id.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants