Skip to content

row-spine: reach more dictionary values via overflow escape tags - #38198

Draft
antiguru wants to merge 2 commits into
mainfrom
claude/dict-compression-overflow-vi0vum
Draft

row-spine: reach more dictionary values via overflow escape tags#38198
antiguru wants to merge 2 commits into
mainfrom
claude/dict-compression-overflow-vi0vum

Conversation

@antiguru

@antiguru antiguru commented Aug 13, 2026

Copy link
Copy Markdown
Member

Motivation

The per-column dictionary codec in arrangements can only compress as many
values as it has spare one-byte tags. On the safe-install path that is
256 - SAFE_TAG_BASE (134) values per column, which is well under the ~512
heavy hitters the MisraGries summary identifies, so popular values past the
cutoff stay uncompressed even though we know they are popular.

Description

Two commits.

row-spine: reach more dictionary values via overflow escape tags

Reserve the top two structurally safe tags as escapes rather than handing
them out as direct entries. An escape plus one index byte addresses a
256-entry block of a new per-column overflow table, so a codec can reference
132 values in one byte and a further 512 in two. Heavy hitters past that fall
through raw, as they did before.

Capacity per column goes from 134 to 132 direct + 512 overflow (= 644) on the
safe path, and to 253 + 512 on the merge path where dynamically free tags are
also available. A const assertion ties OVERFLOW_CAPACITY to
MisraGries::DEFAULT_K, so the codec can now absorb every heavy hitter the
summary retains.

Notes on the non-obvious parts:

  • Soundness. Escapes live in the structurally safe range, so no literal
    datum's first byte can be mistaken for one, and the escape slots are
    excluded from the direct table on both construction paths. encode's
    raw-fall-through debug_assert now also checks the byte is not an escape.
  • Decoding. Resolution moved into DictionaryCodec::lookup, which returns
    the referenced value together with the width of the reference (one byte for
    a direct tag, two for an escape). ColumnsIter::next advances by that
    width. The hot path gains one compare against a constant per column.
  • Why escape blocks instead of a varint index. With a ~512-entry ceiling a
    LEB128 index needs three bytes past 127, whereas a fixed escape block is two
    bytes for all 512, and decoding is a shift-or instead of a loop. Reserving a
    further escape tag buys another 256 slots if that is ever wanted.
  • Assignment order is already optimal. A direct entry saves
    count * (len - 1) bytes and an overflow entry count * (len - 2), so the
    choice between them is worth exactly one byte per occurrence. Giving direct
    tags to the highest-count values, which is the summary's existing order,
    maximizes the saving with no re-sorting.
  • Overflow entries require len > 2. A two-byte value cannot shrink under
    a two-byte reference, so those are skipped rather than consuming a slot that
    cannot pay for itself.

repr: prove the datum tag upper bound exhaustively

This change leans harder on SAFE_TAG_BASE (the escape tags come out of the
same range), and nothing actually held that constant to the Tag enum.
test_safe_tag_base samples: it packs interesting_datums() for every scalar
type and inspects first bytes, so it cannot reach StringHuge, BytesHuge or
ListHuge (those need values larger than 4GiB), and a newly added tag goes
unnoticed until some datum exercises it.

Since every datum begins with a Tag discriminant, the real invariant is a
property of the enum: no byte at or above the boundary decodes as a Tag.
TryFromPrimitive makes that checkable over the whole byte range, but only
where Tag is visible, so the bound is now named in mz_repr as
TAG_UPPER_BOUND and checked exhaustively there. SAFE_TAG_BASE is defined
from it so the two cannot drift, and the sampling test stays as an independent
check from the consumer's side.

Happy to drop this second commit if you would rather not touch mz_repr here.

Verification

Two new tests in src/row-spine/src/lib.rs:

  • test_overflow_codec_round_trip drives more distinct popular values than
    either construction path has direct tags for, asserts every value compresses
    to a dictionary code, and asserts both widths are exercised.
  • test_overflow_capacity_saturation overshoots the overflow table and checks
    the surplus falls through raw rather than taking an unaddressable index.

Both round-trip two-column rows, so a reference decoded at the wrong width
corrupts the following column rather than going unnoticed. I confirmed this by
mutating lookup to report a one-byte width for overflow references: both
tests fail, with the leftover index byte surfacing as a spurious extra column.

One new test in src/repr/src/row.rs, tag_upper_bound_is_exclusive. I
confirmed it is load-bearing by lowering TAG_UPPER_BOUND to 93, a live tag:
it fails with expected Err found Ok(UInt64_64).

The existing codec tests (test_codec_round_trip,
test_safe_codec_merge_bitmap_carryover, test_safe_tag_base,
push_done_promotion_avoids_merge_poison) still pass, as does the rest of
mz_repr's lib suite.

Not yet measured: the effect on a real arrangement's memory and scan
throughput. The net win depends on how heavy the tail of the popular set is in
practice, and each codec grows by up to 512 entries' worth of bytes per
column. A feature-benchmark scenario over a wide low-cardinality string column
would put numbers on it before this earns the dyncfg's trust.

Release notes

This change does not require release notes: the behavior sits behind the
enable_arrangement_dictionary_compression_alpha dyncfg, which defaults off.

claude added 2 commits August 13, 2026 14:55
The per-column dictionary codec could only compress as many values as it had
spare one-byte tags. On the safe-install path that is `256 - SAFE_TAG_BASE`
(134) values, well under the ~512 heavy hitters the MisraGries summary
identifies, so popular values past the cutoff stayed uncompressed.

Reserve the top two safe tags as escapes instead of handing them out as direct
entries. An escape plus one index byte addresses a 256-entry block of a new
per-column overflow table, so a codec can now reference 132 values in one byte
and a further 512 in two, past which the remaining heavy hitters fall through
raw as before. Escapes are structurally safe tags, so no literal datum's first
byte can be mistaken for one, and the escape slots are excluded from the direct
table on both construction paths, keeping decoding unambiguous.

Direct tags go to the highest-count values, which is optimal: the choice
between a one-byte and a two-byte reference is worth exactly one byte per
occurrence. Values of two bytes or less are skipped when filling the overflow
table, since a two-byte reference cannot shrink them.

Adds `test_overflow_codec_round_trip`, which drives more distinct popular
values than either construction path has direct tags for and checks that all of
them compress and round-trip, and `test_overflow_capacity_saturation`, which
overshoots the overflow table and checks that the surplus falls through raw
rather than taking an unaddressable index. Both fail if an escape reference is
decoded at the wrong width, since a mis-sized reference corrupts the following
column.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgsuFbWokpbhSPU4ojvMy1
`mz_row_spine`'s dictionary codec repurposes byte values that no datum tag
occupies, using `SAFE_TAG_BASE` (122) as the boundary. Nothing checked that
boundary against the `Tag` enum. The only check, `test_safe_tag_base`, packs
`interesting_datums()` for every scalar type and inspects the first byte, which
samples rather than proves: it cannot reach `StringHuge`, `BytesHuge` or
`ListHuge` (those need values larger than 4GiB), and a newly added tag goes
unnoticed until some datum exercises it.

Every datum begins with a `Tag` discriminant, so the real invariant is a
property of the enum: no byte at or above the boundary decodes as a `Tag`.
`TryFromPrimitive` makes that checkable over the whole byte range, but only
where `Tag` is visible, so name the bound here as `TAG_UPPER_BOUND` and check it
in `tag_upper_bound_is_exclusive`. `SAFE_TAG_BASE` is now defined from it, so
the two cannot drift, and the existing sampling test stays as an independent
check from the consumer's side.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgsuFbWokpbhSPU4ojvMy1
@antiguru
antiguru force-pushed the claude/dict-compression-overflow-vi0vum branch from 9706da8 to 0864920 Compare August 13, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants