Add `UnionScalar` by connortsui20 · Pull Request #8739 · vortex-data/vortex

connortsui20 · 2026-07-13T15:25:52Z

Rationale for this change

Tracking Issue: #7882

UnionScalar is the row representation needed for scalar access and constant arrays before the canonical UnionArray. Non-null values store the selected type ID and child ScalarValue; null remains on the outer Scalar, avoiding a second null layer.

What changes are included in this PR?

add UnionValue, UnionScalar, construction, downcasts, validation, display, equality, hashing, defaults, and sizing
add scalar protobuf round trips and keep non-identity Union casts unsupported
add arbitrary generation and ConstantArray scalar access coverage

Tested with cargo nextest run -p vortex-array and cargo clippy --all-targets --all-features.

What APIs are changed? Are there any user-facing changes?

Adds ScalarValue::Union, UnionValue, UnionScalar, Scalar::union, and the Union scalar downcasts.

codspeed-hq · 2026-07-13T15:31:30Z

Merging this PR will not alter performance

✅ 1660 untouched benchmarks
⏩ 47 skipped benchmarks¹

_{Comparing ct/union-scalar (1e93238) with develop (8ea5dcd)}

47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

connortsui20 · 2026-07-13T18:57:15Z

+        // Union nullability is part of its variant dtypes, so the generic nullability-only cast
+        // below is not valid for unions. Keep all non-identity union casts unsupported until their
+        // semantics are defined.
+        if matches!(self.dtype(), DType::Union(_)) || matches!(target_dtype, DType::Union(_)) {
+            vortex_bail!(
+                "non-identity union scalar cast from {} to {target_dtype} is not supported",
+                self.dtype()
+            );
+        }


Interested in what people think about how we should deal with this

connortsui20 · 2026-07-13T19:43:23Z

+/// We implement `Hash` manually so top-level dtype nullability does not affect the hash.
 impl Hash for Scalar {
    fn hash<H: Hasher>(&self, state: &mut H) {
+        // TODO(connor): Recursively ignore dtype nullability to match Scalar equality (#8744).


robert3005 · 2026-07-14T14:00:50Z

+        // Union nullability is part of its variant dtypes, so the generic nullability-only cast
+        // below is not valid for unions. Keep all non-identity union casts unsupported until their
+        // semantics are defined.
+        if self.dtype().is_union() || target_dtype.is_union() {


I think you have to gate the other branch and make sure this falls back to union scalar

what is the other branch?

the one the comment refers to

I don't think I understand, are you saying that it can somehow get past this gate?

robert3005 · 2026-07-14T15:37:44Z

            DType::List(..) | DType::FixedSizeList(..) => self.as_list().cast(target_dtype),
            DType::Struct(..) => self.as_struct().cast(target_dtype),
-            DType::Union(..) => todo!("TODO(connor)[Union]: unimplemented"),
+            DType::Union(..) => unreachable!("union casts are handled before scalar dispatch"),


@connortsui20 I am saying you should fallback here in your cast

I still don't really understand here, what exactly do you mean by "falls back to union scalar"? We already have a union scalar in the input

robert3005 · 2026-07-14T15:38:40Z

+    }
+
+    /// Returns the non-null zero value for `dtype`, or [`None`] if no such value exists.
+    fn try_zero_value(dtype: &DType) -> Option<Self> {


I don't get why this needs to return option, Null is a weird type

Null isn't the only type that returns None here though (see FSL, Struct, and now Union)? This just improves the error message in general

because null returns none, I think the develop behaviour is better

this is just an implementation detail though? this is a private helper so the behavior is the same (just different error message)?

robert3005 · 2026-07-14T15:39:42Z

                Self::Tuple(field_values)
            }
-            DType::Union(..) => todo!("TODO(connor)[Union]: unimplemented"),
+            DType::Union(_) => vortex_panic!("{dtype} has no default value"),


this could pick first variant

its not clear what to do if there are no variants (empty union / void type)

robert3005 · 2026-07-14T15:46:02Z

I think as_nullable for union has to traverse all the children and make them nullable. Since there's no top level nullability there's no other implementation that makes sense imho

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 · 2026-07-14T15:50:01Z

This does mean that as_nullable will have different behavior than struct for example which only does the top-level thing, but lets just do it and see how well that works

A union has no top-level nullability of its own; its nullability is derived from its variants. So DType::with_nullability on a union now propagates the requested nullability into every variant instead of being a no-op, via a new UnionVariants::with_nullability helper. This also lets Scalar::into_nullable work on non-nullable unions (it previously had to assert against the no-nullable-representation case), so the assertion and its panic path are removed. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

robert3005 · 2026-07-14T15:58:27Z

why is it different? Struct has top level nullability while the union doesn't

connortsui20 changed the title ~~Add Union scalar support~~ Add UnionScalar Jul 13, 2026

connortsui20 mentioned this pull request Jul 13, 2026

Tracking Issue: Add DType::Union #7882

Open

12 tasks

connortsui20 force-pushed the ct/union-scalar branch 4 times, most recently from 7e619d3 to 243969b Compare July 13, 2026 16:08

connortsui20 commented Jul 13, 2026

View reviewed changes

connortsui20 force-pushed the ct/union-scalar branch 3 times, most recently from 8bb8db6 to 7925944 Compare July 13, 2026 19:41

connortsui20 commented Jul 13, 2026

View reviewed changes

connortsui20 force-pushed the ct/union-scalar branch 3 times, most recently from 43d2fc9 to 9de7663 Compare July 13, 2026 20:01

connortsui20 added the changelog/feature A new feature label Jul 13, 2026

Base automatically changed from codex/reject-negative-union-type-ids to develop July 13, 2026 20:34

connortsui20 force-pushed the ct/union-scalar branch from 9de7663 to 4893177 Compare July 13, 2026 20:56

connortsui20 marked this pull request as ready for review July 13, 2026 22:05

joseph-isaacs reviewed Jul 14, 2026

View reviewed changes

Comment thread vortex-array/src/scalar/typed_view/union.rs Outdated

robert3005 reviewed Jul 14, 2026

View reviewed changes

Comment thread vortex-array/src/scalar/cast.rs Outdated

robert3005 reviewed Jul 14, 2026

View reviewed changes

Add Union scalar support

8ee294c

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 force-pushed the ct/union-scalar branch from 4893177 to 8ee294c Compare July 14, 2026 15:47

connortsui20 requested a review from robert3005 July 14, 2026 16:09

Uh oh!

Conversation

connortsui20 commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

What APIs are changed? Are there any user-facing changes?

Uh oh!

codspeed-hq Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert3005 commented Jul 14, 2026

Uh oh!

connortsui20 commented Jul 14, 2026

Uh oh!

robert3005 commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

connortsui20 commented Jul 13, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 13, 2026 •

edited

Loading