row-spine: reach more dictionary values via overflow escape tags - #38198
Draft
antiguru wants to merge 2 commits into
Draft
row-spine: reach more dictionary values via overflow escape tags#38198antiguru wants to merge 2 commits into
antiguru wants to merge 2 commits into
Conversation
The per-column dictionary codec could only compress as many values as it had spare one-byte tags. On the safe-install path that is `256 - SAFE_TAG_BASE` (134) values, well under the ~512 heavy hitters the MisraGries summary identifies, so popular values past the cutoff stayed uncompressed. Reserve the top two safe tags as escapes instead of handing them out as direct entries. An escape plus one index byte addresses a 256-entry block of a new per-column overflow table, so a codec can now reference 132 values in one byte and a further 512 in two, past which the remaining heavy hitters fall through raw as before. Escapes are structurally safe tags, so no literal datum's first byte can be mistaken for one, and the escape slots are excluded from the direct table on both construction paths, keeping decoding unambiguous. Direct tags go to the highest-count values, which is optimal: the choice between a one-byte and a two-byte reference is worth exactly one byte per occurrence. Values of two bytes or less are skipped when filling the overflow table, since a two-byte reference cannot shrink them. Adds `test_overflow_codec_round_trip`, which drives more distinct popular values than either construction path has direct tags for and checks that all of them compress and round-trip, and `test_overflow_capacity_saturation`, which overshoots the overflow table and checks that the surplus falls through raw rather than taking an unaddressable index. Both fail if an escape reference is decoded at the wrong width, since a mis-sized reference corrupts the following column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MgsuFbWokpbhSPU4ojvMy1
`mz_row_spine`'s dictionary codec repurposes byte values that no datum tag occupies, using `SAFE_TAG_BASE` (122) as the boundary. Nothing checked that boundary against the `Tag` enum. The only check, `test_safe_tag_base`, packs `interesting_datums()` for every scalar type and inspects the first byte, which samples rather than proves: it cannot reach `StringHuge`, `BytesHuge` or `ListHuge` (those need values larger than 4GiB), and a newly added tag goes unnoticed until some datum exercises it. Every datum begins with a `Tag` discriminant, so the real invariant is a property of the enum: no byte at or above the boundary decodes as a `Tag`. `TryFromPrimitive` makes that checkable over the whole byte range, but only where `Tag` is visible, so name the bound here as `TAG_UPPER_BOUND` and check it in `tag_upper_bound_is_exclusive`. `SAFE_TAG_BASE` is now defined from it, so the two cannot drift, and the existing sampling test stays as an independent check from the consumer's side. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MgsuFbWokpbhSPU4ojvMy1
antiguru
force-pushed
the
claude/dict-compression-overflow-vi0vum
branch
from
August 13, 2026 15:52
9706da8 to
0864920
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The per-column dictionary codec in arrangements can only compress as many
values as it has spare one-byte tags. On the safe-install path that is
256 - SAFE_TAG_BASE(134) values per column, which is well under the ~512heavy hitters the MisraGries summary identifies, so popular values past the
cutoff stay uncompressed even though we know they are popular.
Description
Two commits.
row-spine: reach more dictionary values via overflow escape tagsReserve the top two structurally safe tags as escapes rather than handing
them out as direct entries. An escape plus one index byte addresses a
256-entry block of a new per-column overflow table, so a codec can reference
132 values in one byte and a further 512 in two. Heavy hitters past that fall
through raw, as they did before.
Capacity per column goes from 134 to 132 direct + 512 overflow (= 644) on the
safe path, and to 253 + 512 on the merge path where dynamically free tags are
also available. A const assertion ties
OVERFLOW_CAPACITYtoMisraGries::DEFAULT_K, so the codec can now absorb every heavy hitter thesummary retains.
Notes on the non-obvious parts:
datum's first byte can be mistaken for one, and the escape slots are
excluded from the direct table on both construction paths.
encode'sraw-fall-through
debug_assertnow also checks the byte is not an escape.DictionaryCodec::lookup, which returnsthe referenced value together with the width of the reference (one byte for
a direct tag, two for an escape).
ColumnsIter::nextadvances by thatwidth. The hot path gains one compare against a constant per column.
LEB128 index needs three bytes past 127, whereas a fixed escape block is two
bytes for all 512, and decoding is a shift-or instead of a loop. Reserving a
further escape tag buys another 256 slots if that is ever wanted.
count * (len - 1)bytes and an overflow entrycount * (len - 2), so thechoice between them is worth exactly one byte per occurrence. Giving direct
tags to the highest-count values, which is the summary's existing order,
maximizes the saving with no re-sorting.
len > 2. A two-byte value cannot shrink undera two-byte reference, so those are skipped rather than consuming a slot that
cannot pay for itself.
repr: prove the datum tag upper bound exhaustivelyThis change leans harder on
SAFE_TAG_BASE(the escape tags come out of thesame range), and nothing actually held that constant to the
Tagenum.test_safe_tag_basesamples: it packsinteresting_datums()for every scalartype and inspects first bytes, so it cannot reach
StringHuge,BytesHugeorListHuge(those need values larger than 4GiB), and a newly added tag goesunnoticed until some datum exercises it.
Since every datum begins with a
Tagdiscriminant, the real invariant is aproperty of the enum: no byte at or above the boundary decodes as a
Tag.TryFromPrimitivemakes that checkable over the whole byte range, but onlywhere
Tagis visible, so the bound is now named inmz_reprasTAG_UPPER_BOUNDand checked exhaustively there.SAFE_TAG_BASEis definedfrom it so the two cannot drift, and the sampling test stays as an independent
check from the consumer's side.
Happy to drop this second commit if you would rather not touch
mz_reprhere.Verification
Two new tests in
src/row-spine/src/lib.rs:test_overflow_codec_round_tripdrives more distinct popular values thaneither construction path has direct tags for, asserts every value compresses
to a dictionary code, and asserts both widths are exercised.
test_overflow_capacity_saturationovershoots the overflow table and checksthe surplus falls through raw rather than taking an unaddressable index.
Both round-trip two-column rows, so a reference decoded at the wrong width
corrupts the following column rather than going unnoticed. I confirmed this by
mutating
lookupto report a one-byte width for overflow references: bothtests fail, with the leftover index byte surfacing as a spurious extra column.
One new test in
src/repr/src/row.rs,tag_upper_bound_is_exclusive. Iconfirmed it is load-bearing by lowering
TAG_UPPER_BOUNDto 93, a live tag:it fails with
expected Err found Ok(UInt64_64).The existing codec tests (
test_codec_round_trip,test_safe_codec_merge_bitmap_carryover,test_safe_tag_base,push_done_promotion_avoids_merge_poison) still pass, as does the rest ofmz_repr's lib suite.Not yet measured: the effect on a real arrangement's memory and scan
throughput. The net win depends on how heavy the tail of the popular set is in
practice, and each codec grows by up to 512 entries' worth of bytes per
column. A feature-benchmark scenario over a wide low-cardinality string column
would put numbers on it before this earns the dyncfg's trust.
Release notes
This change does not require release notes: the behavior sits behind the
enable_arrangement_dictionary_compression_alphadyncfg, which defaults off.