Add StandardSum: a sum aggregate where zero valid values yields null

[mhk197]

Adds a new aggregate function StandardSum whose sum over zero valid values (empty or all-null input) is null rather than zero, matching DuckDB, arrow-rs, and DataFusion. Overflow still yields null. list_sum now uses it, which makes its empty/all-null-list second pass unnecessary.

The existing Sum is untouched. Since the zoned layout persists AggregateSpecProto { id: "vortex.sum", .. } and Sum's partial is wire format, it makes more sense to keep its name for now. Sum keeps being what statistics store and merge (identity-on-empty is what stat merging requires now).

StandardSum's partial aggregate is{sum, seen} rather than just seen. A null sum already means overflow and must poison merges, while "empty" must be the merge identity so one nullable scalar can't encode both. So StandardSum's partial carries a seen flag like DuckDB's isset. The identity is {0, false}, and merge is is null-poisoning add of sum and union of seen., Finalize maps seen == 0 to null.

Implementation notes:

• The accumulation kernels are shared with Sum (re-exported, unchanged)
• finalize produces a lazy array rather than materializing eagerly
• The grouped kernel emits {sum, seen} rows directly

This also fixes StructGetItemRule. get_item on a nullable struct has a nullable result dtype even when every row is valid, but the rule returned the raw non-nullable field, causing a reduced-dtype mismatch (surfaced by the lazy finalize over null groups; it's a standalone bug).

[codspeed-hq]

Merging this PR will not alter performance

⚡ 6 improved benchmarks
❌ 1 regressed benchmark
✅ 1653 untouched benchmarks
🆕 12 new benchmarks
⏩ 47 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	fsl_sum_small	57.1 µs	74.3 µs	-23.1%
⚡	Simulation	listview_sum_large	108.1 ms	88.8 ms	+21.7%
⚡	Simulation	list_sum_large	135 ms	111.6 ms	+20.96%
⚡	Simulation	list_sum_nullable_elements_large	493.3 ms	425.5 ms	+15.92%
⚡	Simulation	sum_i32_nullable_all_valid	41.8 µs	36.4 µs	+14.77%
⚡	Simulation	list_sum_nullable_elements_medium	4.9 ms	4.3 ms	+12.35%
⚡	Simulation	compress[(100000, 4)]	1.8 ms	1.6 ms	+11.92%
🆕	Simulation	standard_sum_f64_all_valid	N/A	54.9 µs	N/A
🆕	Simulation	standard_sum_f64_clustered_nulls	N/A	83.6 µs	N/A
🆕	Simulation	standard_sum_i32_clustered_nulls	N/A	80.6 µs	N/A
🆕	Simulation	standard_sum_i32_nullable_all_valid	N/A	51.4 µs	N/A
🆕	Simulation	canonical_standard_sum_f64_all_valid	N/A	75.3 µs	N/A
🆕	Simulation	canonical_standard_sum_f64_clustered_nulls	N/A	103.2 µs	N/A
🆕	Simulation	canonical_standard_sum_i32_clustered_nulls	N/A	100.6 µs	N/A
🆕	Simulation	canonical_standard_sum_i32_nullable_all_valid	N/A	76.7 µs	N/A
🆕	Simulation	canonical_sum_f64_all_valid	N/A	49.9 µs	N/A
🆕	Simulation	canonical_sum_f64_clustered_nulls	N/A	78.9 µs	N/A
🆕	Simulation	canonical_sum_i32_clustered_nulls	N/A	75.9 µs	N/A
🆕	Simulation	canonical_sum_i32_nullable_all_valid	N/A	47.4 µs	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/sum-empty-null (1ca56ff) with develop (5abaf98)}

Footnotes

1. 47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[gatesn]

I think... with the new Zoned layout, the aggregate partial has become a wire API.

That said, we haven't updated FileStats to use aggregate partials. And I don't believe zone maps use Sum?

A safer option would be to add a separate Sum function with this new behavior and different partial. @robert3005 any thoughts on a preferred direction?

Either way, we should separately make sure we have forward/backward compatibility tests for zoned statistics!

[robert3005]

we have SUM in zonemaps

[robert3005]

@mhk197 I think simpler would be to add another function. Maybe even rename them such that the new sql semantic sum is called sum and the other is called sum_<I have no idea what a good name is>

[joseph-isaacs]

Why can we not use Sum options?

[robert3005]

We can but in order to support it you have to change the partial and the partial is being written to the files therefore you have to handle partial being in the old form even if you want to compute the new sum, maybe the semantics are clear but I thought it might be easier this way.

[robert3005]

I feel like this was a long standing bug

[joseph-isaacs]

How can this happen?

[mhk197]

e.g. partial sum calculated over [-1,1], [0, 0, 0, 0] vs [null, null], []

[joseph-isaacs]

Lets make sure we get this right

[joseph-isaacs]

Can we instead store a overflowed and a nullable partial sum

{partial: T, overflowed: bool} that way you can first check overflow of both and return, then use NULL op x = x semantics. This seems much simpler.

I guess overflowed == seen /\ is_null(value). But this seems more complicated to me

[mhk197]

Yep makes sense, this has been done.

[mhk197]

Wrt comment below, we're changing the semantics of partial sum field (NULL op x = x instead of poisoning merges) and partials are already persisted on wire. This change would result in old stored Sum stats/zone maps creating a lot of work, since would have to fetch and canonicalize underlying arrays to fetch validity + count non-null bits

[joseph-isaacs]

I really want to avoid 100s of very similar partial. I am not sure we want to create a new one here vs using a good default for the other.

[gatesn]

The other one is part of the serialized ZonedLayout. You cannot change an aggregate partial