Skip to content

Add UnionArray compute functions#8884

Draft
connortsui20 wants to merge 1 commit into
ct/union-arrayfrom
ct/union-array-compute
Draft

Add UnionArray compute functions#8884
connortsui20 wants to merge 1 commit into
ct/union-arrayfrom
ct/union-array-compute

Conversation

@connortsui20

@connortsui20 connortsui20 commented Jul 21, 2026

Copy link
Copy Markdown
Member

Rationale for this change

Depends on #8834. Tracking issue: #8769.

This is the deferred compute layer for UnionArray and remains a draft for later review.

What changes are included in this PR?

  • Adds UnionArray filter, take, slice, and mask execution.
  • Adds constant and chunked canonicalization.
  • Adds Union casting, uncompressed-size accounting, and recursive compression.
  • Adds focused tests for structural operations and canonicalization.

Nullable-index take remains explicitly unsupported and needs resolution before this layer is ready.

What APIs are changed? Are there any user-facing changes?

No additional public array representation APIs beyond #8834. This layer wires UnionArray into existing compute paths.

@connortsui20 connortsui20 changed the title ct/union array compute Add UnionArray compute functions Jul 21, 2026
@connortsui20 connortsui20 mentioned this pull request Jul 21, 2026
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done ec92eef 1 Explore Profiling Data
Previous Runs (7)
Status Commit Job Attempt Link
🟢 Done ab33ec0 1 Explore Profiling Data
🟢 Done f2427ce 1 Explore Profiling Data
🟢 Done e32db9e 1 Explore Profiling Data
🟢 Done 98f8f06 1 Explore Profiling Data
🟢 Done 1e867dd 1 Explore Profiling Data
🟢 Done a6b7e6c 1 Explore Profiling Data
🟢 Done 6fe53ba 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +1.2%
Engines: DataFusion No clear signal (+4.0%, low confidence) · DuckDB No clear signal (-1.5%, low confidence)
Vortex (geomean): 1.014x ➖
Parquet (geomean): 1.008x ➖
Shifts: Parquet (control) +0.8% · Median polish +0.8%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.045x ➖, 0↑ 0↓)
name PR ec92eef (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 8414693 8197567 1.03
vortex_q01/datafusion:vortex-file-compressed 5201551 4885078 1.06
datafusion / parquet (1.006x ➖, 0↑ 0↓)
name PR ec92eef (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 20201445 20638806 0.98
vortex_q01/datafusion:parquet 4731033 4580085 1.03
duckdb / vortex-file-compressed (0.996x ➖, 0↑ 0↓)
name PR ec92eef (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 9982079 10061599 0.99
vortex_q01/duckdb:vortex-file-compressed 6163313 6167184 1.00
duckdb / parquet (1.010x ➖, 0↑ 0↓)
name PR ec92eef (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23447372 23195009 1.01
vortex_q01/duckdb:parquet 9434025 9340266 1.01

No file size changes detected.

@connortsui20
connortsui20 force-pushed the ct/union-array-compute branch from 6fe53ba to a6b7e6c Compare July 21, 2026 17:48
@connortsui20
connortsui20 force-pushed the ct/union-array branch 2 times, most recently from 96923e7 to 3233233 Compare July 21, 2026 17:59
@connortsui20
connortsui20 force-pushed the ct/union-array-compute branch from a6b7e6c to 1e867dd Compare July 21, 2026 17:59
@codspeed-hq

codspeed-hq Bot commented Jul 21, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 28.39%

⚡ 1 improved benchmark
✅ 1846 untouched benchmarks
⏩ 46 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decode_varbin[(1000, 8)] 34.3 µs 26.7 µs +28.39%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/union-array-compute (ec92eef) with ct/union-array (772b738)2

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on ct/union-array (f51f2dd) during the generation of this report, so 8307911 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.09910% with 115 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.13%. Comparing base (cd92358) to head (98f8f06).

Files with missing lines Patch % Lines
...rtex-array/src/arrays/constant/vtable/canonical.rs 49.42% 44 Missing ⚠️
vortex-array/src/scalar_fn/fns/cast/mod.rs 0.00% 17 Missing ⚠️
vortex-array/src/arrays/union/tests.rs 93.59% 13 Missing ⚠️
...gregate_fn/fns/uncompressed_size_in_bytes/union.rs 0.00% 11 Missing ⚠️
vortex-array/src/arrays/masked/execute.rs 0.00% 10 Missing ⚠️
vortex-compressor/src/compressor/cascade.rs 0.00% 8 Missing ⚠️
vortex-array/src/arrays/filter/execute/mod.rs 0.00% 6 Missing ⚠️
vortex-array/src/arrays/dict/execute.rs 0.00% 3 Missing ⚠️
...aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs 0.00% 1 Missing ⚠️
...ortex-array/src/arrays/chunked/vtable/canonical.rs 97.14% 1 Missing ⚠️
... and 1 more

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@connortsui20
connortsui20 force-pushed the ct/union-array-compute branch 2 times, most recently from e32db9e to f2427ce Compare July 21, 2026 20:31
@connortsui20
connortsui20 force-pushed the ct/union-array branch 2 times, most recently from dd86118 to 772b738 Compare July 21, 2026 21:08
@connortsui20
connortsui20 force-pushed the ct/union-array-compute branch from f2427ce to ab33ec0 Compare July 21, 2026 21:11
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant