feat[buffer]: specialized collect_bool per arch

[joseph-isaacs]

Rationale for this change

Currently LLVM cannot create a bitcast <64 x i1> -> i64 instruction (https://llvm.org/docs/LangRef.html#bitcast-to-instruction), so the auto-vectorizer never turns collect_bool's bit-at-a-time shift-OR loop into byte→bit packing instructions. This means we must add arch-specific collect_bool impls, as done here: modern ISAs pack 64 bytes → 64 bits in 1–4 instructions (vptestmb, vpmovmskb, pmovmskb, NEON ushl+addp), while the scalar loop pays ~1 cycle per element on every collect_bool-based kernel (compare, between, filter/take, LIKE, boolean decode, From<&[bool]>).

The catch: runtime-detected wide kernels must live behind a #[target_feature] function boundary that cannot inline into callers, and an early draft that put every predicate behind that boundary regressed FSST's DFA-scan fsst_prefix benchmarks by 13–63%. The design therefore has two tiers, with the boundary trade-off documented on every entry point.

What changes are included in this PR?

• New vortex-buffer/src/bit/pack.rs with byte→bit pack kernels per architecture: AVX-512BW (vptestmb), AVX2 (vpcmpeqb+vpmovmskb), SSE2 (pcmpeqb+pmovmskb), aarch64 NEON (ushl+addp tree), and a portable SWAR multiply fallback (also used under Miri). Each kernel is tested against the scalar reference; the NEON reduction tree was additionally validated against a reference model on 10,004 random inputs.
• Default tier (collect_bool / collect_bool_words, unchanged signatures): the word loop compiles once with the widest compile-time-enabled kernel (SSE2/NEON on stock targets, AVX2/AVX-512 under -C target-cpu=native), so the predicate, the [bool; 64] materialization, and the pack all inline into the caller. Safe for arbitrary predicates — fsst_prefix is unchanged-to-slightly-better vs develop.
• Opt-in tier (collect_bool_multiversioned / collect_bool_words_multiversioned): compiles the loop with the predicate inside once per CPU feature level and selects a clone by runtime detection. Docs state the caller's assertion: the predicate must be trivially cheap, because the boundary deoptimizes non-trivial ones. Lengths under one word skip detection entirely.
• Routed the provably-cheap call sites through the multiversioned tier: From<&[bool]> / From<&[u8]>, ByteBool boolean decode, and the primitive/decimal between kernels (now reading via get_unchecked, sound because the predicate only receives indices 0..len — a contract now documented on the entry points).
• New benches/collect_bool.rs covering each kernel and both tiers; only the shipped entry points (from_bool_slice, collect_bool_u32_gt) are tracked under CodSpeed — baselines and per-variant loops compile out.

Measured on AVX-512 hardware (divan, 64Ki bits, medians): the fused word loop with a bool-gather predicate runs 14–24× faster than the scalar loop (SSE2 1.7µs / AVX2 1.2µs / AVX-512 1.1µs vs 26.5µs); BitBufferMut::from(&[bool]) improves ~43×; a between-shaped predicate doubles under the multiversioned tier; generated asm verified per kernel (the fused AVX-512 loop is vmovdqu64 + vptestmb + kmovq per 64 bools).

What APIs are changed? Are there any user-facing changes?

No existing signatures change. New public API in vortex-buffer: BitBuffer::collect_bool_multiversioned, BitBufferMut::collect_bool_multiversioned, collect_bool_words_multiversioned, collect_bool_word_scalar, the per-arch pack kernels (pack_bool_word_{swar,sse2,avx2,avx512,neon}), and the per-arch word loops (collect_bool_words_{sse2,avx2,avx512,neon}, kept public so each variant stays individually testable and benchable). All documented, with the multiversioned entries steering readers back to collect_bool unless their specific predicate has been checked (ideally benchmarked).

🤖 Generated with Claude Code

https://claude.ai/code/session_01TcsiAHXQc4e11yYcP3a3NX

[codspeed-hq]

Merging this PR will improve performance by 17.23%

⚡ 23 improved benchmarks
✅ 1637 untouched benchmarks
🆕 4 new benchmarks
⏩ 47 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	copy_nullable[65536]	1,028 µs	573.4 µs	+79.3%
⚡	Simulation	new_bp_prim_test_between[i32, 32768]	155 µs	123.8 µs	+25.23%
⚡	Simulation	new_bp_prim_test_between[i16, 32768]	150.8 µs	122.1 µs	+23.48%
⚡	Simulation	new_alp_prim_test_between[f32, 32768]	169.4 µs	138.6 µs	+22.24%
⚡	Simulation	new_bp_prim_test_between[i32, 16384]	95.6 µs	80 µs	+19.52%
⚡	Simulation	new_bp_prim_test_between[i16, 16384]	92.2 µs	77.9 µs	+18.32%
⚡	Simulation	new_bp_prim_test_between[i64, 32768]	162.5 µs	138.1 µs	+17.67%
⚡	Simulation	new_alp_prim_test_between[f32, 16384]	106 µs	90.6 µs	+17.02%
⚡	Simulation	new_raw_prim_test_between[f32, 32768]	121.1 µs	103.6 µs	+16.89%
⚡	Simulation	new_alp_prim_test_between[f64, 32768]	177.5 µs	152.9 µs	+16.05%
⚡	Simulation	copy_non_nullable[65536]	908.6 µs	794 µs	+14.44%
⚡	Simulation	new_raw_prim_test_between[u32, 32768]	116.5 µs	101.9 µs	+14.38%
⚡	Simulation	new_raw_prim_test_between[i32, 32768]	114.3 µs	100.1 µs	+14.22%
⚡	Simulation	new_bp_prim_test_between[i64, 16384]	101.5 µs	89.2 µs	+13.78%
⚡	Simulation	new_raw_prim_test_between[f32, 16384]	74.9 µs	66.1 µs	+13.37%
⚡	Simulation	new_alp_prim_test_between[f64, 16384]	113.3 µs	101.2 µs	+12.01%
⚡	Simulation	new_raw_prim_test_between[u32, 16384]	72.7 µs	65.2 µs	+11.48%
⚡	Simulation	new_raw_prim_test_between[i32, 16384]	71.5 µs	64.3 µs	+11.27%
⚡	Simulation	slice_vortex_buffer[1024]	959.2 ns	871.7 ns	+10.04%
⚡	Simulation	slice_vortex_buffer[16384]	959.2 ns	871.7 ns	+10.04%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/collect-bool-arch-optimizations-bo09ws (e64d06d) with develop (5abaf98)}

Footnotes

1. 47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩