Support out of order grouped aggregate accumulation

[gatesn]

Our original API for grouped aggregate functions only supported groups that we pre-sorted into complete lists. That meant large groups had to be globally sorted before we could start accumulating their internal state, unlike the grouped-aggregate APIs used by query engines.

This PR updates the grouped accumulator API to accept caller-assigned dense GroupIds directly.

Summary

• Accept GroupIds { ids: ArrayRef, num_groups }, including repeated, out-of-order, and absent IDs.
• Dispatch grouped kernels on both the value encoding and group-ID encoding.
• Cache materialized and validated group IDs across clones and aggregate functions.
• Keep a list-group compatibility adapter for existing list_sum callers.
• Let each aggregate own its dense grouped-state representation, with the existing generic partial-state implementation as the fallback.
• Store grouped count state as one u64 per group.
• Store grouped sum state in monomorphic u64, i64, f64, or decimal vectors, with overflow and empty state kept separately.
• Preserve SQL null-on-empty sum semantics, including all-null groups and groups absent from a batch.
• Select exact decimal state width (i8 through i256) once when the accumulator is created.
• Merge count, primitive-sum, float-sum, and decimal-sum partial arrays without execute_scalar per row.
• Adapt between direct typed scatter for unclustered IDs and per-run reduction for clustered IDs.
• Use one stable counting-sort gather per morsel in the universal fallback instead of one take per group.

Spiral impact

Spiral already interns group keys into dense u32 state slots, so it can pass those IDs directly without rearranging values into ordered list groups. This covers the common count/sum machinery while allowing Spiral to retain its stronger integer/decimal mean states and other specialized paths.

Local performance

Medians below use 65,536 non-null i32 rows and include canonical result materialization. Shuffled inputs contain the same multiset of IDs as their clustered counterpart.

Groups	Count clustered	Count shuffled	Sum clustered	Sum shuffled
128	16.6 μs	16.5 μs	22.0 μs	36.3 μs
4,096	18.5 μs	16.1 μs	30.2 μs	26.9 μs
65,536	18.3 μs	37.2 μs	60.4 μs	90.2 μs

The original 65,536-row / 4,096-group shuffled sum benchmark was approximately 215 μs with per-group SumPartial state. Typed storage plus adaptive scatter reduces it to approximately 26.9 μs while also materializing the result and tracking SQL empty-state semantics, an approximately 8× improvement.

On the existing roughly 1K-row cases, representative medians are 1.9 μs for clustered-null count, 3.7 μs for nullable all-valid i32 sum, 3.9 μs for all-valid f64 sum, and 6.2 μs for clustered-null i32 sum.

Validation

• cargo +nightly fmt --all
• cargo test -p vortex-array --lib (3,376 passed, 1 ignored)
• cargo bench -p vortex-array --bench aggregate_grouped
• cargo clippy -p vortex-array --all-targets --all-features
• cargo clippy --all-targets --all-features
• RUSTDOCFLAGS="-D warnings" cargo doc --profile ci --no-deps -p vortex-array

[codspeed]

Merging this PR will regress 16 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 14 improved benchmarks
❌ 16 regressed benchmarks
✅ 2105 untouched benchmarks
🆕 18 new benchmarks
⏩ 110 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	fsl_sum_large	167.8 µs	231.8 µs	-27.59%
❌	Simulation	listview_sum_large	191.8 µs	254.7 µs	-24.69%
❌	WallTime	mul_u64_nonnull_neon	15.2 µs	19.6 µs	-22.51%
❌	Simulation	fsl_sum_medium	144.6 µs	183.2 µs	-21.1%
❌	Simulation	listview_sum_medium	159.5 µs	198.4 µs	-19.58%
❌	Simulation	fsl_sum_small	131.7 µs	163.1 µs	-19.28%
❌	Simulation	list_sum_nullable_elements_large	414.4 µs	512.2 µs	-19.09%
❌	Simulation	listview_sum_small	140.2 µs	172.4 µs	-18.67%
❌	Simulation	list_sum_large	272.7 µs	335.3 µs	-18.67%
❌	Simulation	sum_v2_i32_nullable_all_valid	110.9 µs	133.4 µs	-16.86%
❌	WallTime	mul_i64_nonnull_neon	17 µs	20.4 µs	-16.69%
❌	Simulation	list_sum_nullable_elements_medium	308.3 µs	368 µs	-16.22%
❌	WallTime	multiply_shapes_neon[(16384, PerRowPerRow)]	17.2 µs	20.5 µs	-15.98%
❌	Simulation	list_sum_medium	235.8 µs	278 µs	-15.16%
❌	Simulation	list_sum_small	213.2 µs	244.9 µs	-12.91%
❌	Simulation	sum_v2_f64_all_valid	117.6 µs	131.5 µs	-10.57%
⚡	Simulation	count_varbinview	159 µs	35 µs	×4.5
⚡	Simulation	sum_i32_clustered_nulls	124 µs	57.6 µs	×2.2
⚡	Simulation	sum_f64_clustered_nulls	131.1 µs	61.8 µs	×2.1
⚡	Simulation	count_i32_clustered_nulls	89.8 µs	43.8 µs	×2.1
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ngates/grouped-aggregate (3c4d65f) with develop (f880575)}

Footnotes

1. 110 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[joseph-isaacs]

We wait that vortex was a linear scan engine. This breaks it correct?

[onursatici]

I prefer this API, not only that it supports out of order groups but also it can keep the memory footprint lower by not having us to materialise each group before calling aggregate.

I think you are conflicting with #8314
Sum and count kernels you added make sense to me but we went with having a encoding agnostic kernel support on the registry on the merged PR instead of having the kernel in the aggregate function's vtable

[gatesn]

We wait that vortex was a linear scan engine. This breaks it correct?

No, the idea is that I can take an array, compute the group indices (i.e. using a scalar function), then update the aggregate state per group. This still doesn't require a shuffle / sort. But it does mean aggregate state may grow large. So in the future this is where we would want the ability to spill partials / reconstruct later.

[onursatici]

I think we should choose either to have encoding agnostic grouped kernels that we register in the registry, like develop does with AggregateFnSession::register_grouped_kernel, or have the fast path as methods on the aggregate function vtable, like this PR does with AggregateFnVTable::try_accumulate_grouped

If we merge this as is, we will have two ways that do very similar things

[gatesn]

Agreed, in general I want to move everything towards session kernels, including both aggregate functions and scalar functions.

[github-actions]

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

[github-actions]

This PR was closed because it has been inactive for 7 days since being marked as stale.

[github-actions]

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

[github-actions]

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days