feat(vortex-geo): per-row bounding-box scalar function (vortex.geo.envelope)#8831
Merged
Conversation
Merging this PR will not alter performance
Performance Changes
Comparing 
Footnotes
|
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (9)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: Vortex queries 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.019x ➖, 0↑ 0↓)
datafusion / parquet (0.969x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.029x ➖, 0↑ 0↓)
duckdb / parquet (1.009x ➖, 0↑ 0↓)
No file size changes detected. |
HarukiMoriarty
force-pushed
the
nemo/geo-bounding-rect
branch
from
July 17, 2026 20:26
488a97c to
0c8823f
Compare
HarukiMoriarty
changed the base branch from
develop
to
nemo/geo-null-propagation
July 17, 2026 20:26
HarukiMoriarty
force-pushed
the
nemo/geo-bounding-rect
branch
from
July 20, 2026 20:05
0c8823f to
b269fa9
Compare
connortsui20
requested changes
Jul 20, 2026
connortsui20
left a comment
connortsui20
left a comment
Member
There was a problem hiding this comment.
this should be a scalar function?
…velope) `GeoEnvelope` computes the per-row 2-D axis-aligned bounding box of a native geometry column, returned as a native geoarrow.box (`Rect`) column — the per-row counterpart of the `GeometryAabb` aggregate (whole-column). Intended for row-oriented consumers such as bulk-loading an in-memory R-tree in a spatial-join operator, which read the resulting box column back row by row. Computed directly over the native coordinate storage — no `geo_types` decode and no Arrow round-trip: walk the nested `ListView` storage down to the leaf x/y buffers, tracking which top-level row owns each coordinate, then min/max per row. A `Rect` input is the identity; a null row or an empty geometry yields a null box, so the output is always nullable. Hoist the shared `f64_field` coordinate-column accessor into `extension::coordinate`, used by both this function and the aggregate. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
HarukiMoriarty
force-pushed
the
nemo/geo-bounding-rect
branch
from
July 21, 2026 14:13
b269fa9 to
72aa947
Compare
HarukiMoriarty
changed the title
joseph-isaacs
added
the
action/benchmark-sql-full
github-actions
Bot
removed
the
action/benchmark-sql-full
Contributor
Benchmarks: PolarSignals Profiling 📖Vortex (geomean): 1.016x ➖ How to read Verdict and Engines
datafusion / vortex-file-compressed (1.016x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.998x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.995x ➖, 0↑ 0↓)
datafusion / parquet (1.016x ➖, 1↑ 4↓)
datafusion / arrow (0.983x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.004x ➖, 0↑ 0↓)
duckdb / parquet (0.994x ➖, 1↑ 1↓)
duckdb / duckdb (1.006x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.033x ➖, 0↑ 2↓)
datafusion / vortex-compact (1.042x ➖, 0↑ 1↓)
datafusion / parquet (1.024x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.041x ➖, 0↑ 1↓)
duckdb / vortex-compact (1.016x ➖, 0↑ 1↓)
duckdb / parquet (1.038x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb S3 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.870x ➖, 1↑ 0↓)
datafusion / vortex-compact (1.090x ➖, 0↑ 1↓)
datafusion / parquet (1.031x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.969x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.996x ➖, 0↑ 0↓)
duckdb / parquet (1.026x ➖, 0↑ 0↓)
|
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.062x ➖, 0↑ 3↓)
datafusion / parquet (1.061x ➖, 0↑ 3↓)
duckdb / vortex-file-compressed (1.057x ➖, 0↑ 3↓)
duckdb / parquet (1.018x ➖, 0↑ 0↓)
duckdb / duckdb (1.035x ➖, 0↑ 1↓)
File Size Changes (201 files changed, +0.0% overall, 102↑ 99↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.008x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.002x ➖, 0↑ 0↓)
datafusion / parquet (0.992x ➖, 0↑ 0↓)
datafusion / arrow (1.009x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.001x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.004x ➖, 0↑ 0↓)
duckdb / parquet (0.990x ➖, 0↑ 0↓)
duckdb / duckdb (0.990x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.929x ➖, 19↑ 0↓)
datafusion / parquet (0.958x ➖, 7↑ 0↓)
duckdb / vortex-file-compressed (0.951x ➖, 9↑ 1↓)
duckdb / parquet (0.969x ➖, 4↑ 0↓)
duckdb / duckdb (0.959x ➖, 6↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.108x ➖, 0↑ 2↓)
datafusion / vortex-compact (1.125x ➖, 0↑ 4↓)
datafusion / parquet (1.084x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.052x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.059x ➖, 0↑ 1↓)
duckdb / parquet (1.039x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Appian on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.972x ➖, 0↑ 0↓)
datafusion / parquet (0.976x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.986x ➖, 0↑ 0↓)
duckdb / parquet (0.998x ➖, 0↑ 0↓)
duckdb / duckdb (0.996x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
| } | ||
|
|
||
| impl ScalarFnVTable for GeoEnvelope { | ||
| type Options = EmptyOptions; |
Contributor
There was a problem hiding this comment.
type of covering bound
Contributor
Benchmarks: TPC-H SF=10 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.150x ➖, 0↑ 4↓)
datafusion / vortex-compact (1.153x ➖, 0↑ 6↓)
datafusion / parquet (1.134x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed (1.167x ➖, 0↑ 1↓)
duckdb / vortex-compact (1.101x ➖, 0↑ 0↓)
duckdb / parquet (1.087x ➖, 0↑ 0↓)
|
connortsui20
requested changes
Jul 21, 2026
|
|
||
| /// Materialize a named non-nullable `f64` field of a coordinate `Struct` column as a | ||
| /// [`PrimitiveArray`], for bulk per-ordinate reads. | ||
| pub(crate) fn f64_field( |
Member
There was a problem hiding this comment.
If you are going to execute this into primitive, I feel like you might as well just return the Buffer<f64> here so you dont have to call as_slice after?
| /// row owns each coordinate, then min/maxes x/y per row over the raw `f64` buffers. The ring/part | ||
| /// nesting is irrelevant to a bounding box, only which coordinates belong to a row matters. A null | ||
| /// row, or a valid row that owns no coordinate (an empty geometry), yields a null box. | ||
| fn envelopes(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult<ArrayRef> { |
Member
There was a problem hiding this comment.
Nit: I feel like you could modularize this better? And I think I would prefer to see the logic in the actual execute method instead of pulling out the entire function (for example, just move this back into the execute method, and then you can pull the else list case into a separate function because it has a while loop inside it)
Comment on lines
+99
to
+101
| if ext.is::<Rect>() { | ||
| // The bounding box of a box is itself: read its min/max fields straight from storage. | ||
| let coords = storage.execute::<StructArray>(ctx)?; |
Member
There was a problem hiding this comment.
Maybe I am missing something, but if the array is already a Rect array, can't we just return it directly and say that a rectangle is itself its envelope?
| // Per-row min/max accumulators; `seen[r]` marks a row that owns at least one coordinate. | ||
| let mut lo = vec![(f64::INFINITY, f64::INFINITY); len]; | ||
| let mut hi = vec![(f64::NEG_INFINITY, f64::NEG_INFINITY); len]; | ||
| let mut seen = vec![false; len]; |
Member
There was a problem hiding this comment.
its not clear to me even after tracing through this code what the purpose of seen is here
Comment on lines
+135
to
+153
| for (i, &row) in owner.iter().enumerate() { | ||
| let start = usize::try_from(offsets[i]) | ||
| .map_err(|_| vortex_err!("geo: list offset exceeds usize"))?; | ||
| let size = usize::try_from(sizes[i]) | ||
| .map_err(|_| vortex_err!("geo: list size exceeds usize"))?; | ||
| child[start..start + size].fill(row); | ||
| } | ||
| owner = child; | ||
| node = elements; | ||
| } | ||
| let coords = node.execute::<StructArray>(ctx)?; | ||
| let xs = f64_field(&coords, "x", ctx)?; | ||
| let ys = f64_field(&coords, "y", ctx)?; | ||
| for ((&r, &x), &y) in owner | ||
| .iter() | ||
| .zip(xs.as_slice::<f64>()) | ||
| .zip(ys.as_slice::<f64>()) | ||
| { | ||
| lo[r] = (lo[r].0.min(x), lo[r].1.min(y)); |
Member
There was a problem hiding this comment.
this all seems very complicated to me... IIUC we just want to get the min and the max of each of the rows' elements right? Why do we need to make so many intermediate steps? Maybe this is the correct way to do this, but because it is very hard to trace through (no comments) it is hard for me to reason if this is ideal.
Address review feedback on the envelope scalar function: - Replace the per-element owner-scatter with flatten_row_offsets: each List level converts through list_from_list_view (exact ListArray layout), row boundaries compose through the offsets, and each row then min/maxes a contiguous slice of the raw f64 buffers. This also fixes sliced/scattered views, where unreferenced elements were previously attributed to row 0. - Short-circuit Rect operands: a box is its own envelope, so project the 2-D corner fields lazily and keep the operand validity lazy too. - ordinates (was f64_field) returns Buffer<f64> directly; share box_corners and box_field_names between the envelope scalar fn and the GeometryAabb aggregate instead of duplicating them. - New tests: sliced operands, uneven nesting across levels, inner-level empty geometries, and 3-D Rect operands. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Member
|
talked offline, we should add a microbenchmark I am pretty sure there is a way to write this function in a branchless way (after some preprocessing, like finding all of the empty lists and converting to a list from a list view), and I believe that would be much faster. But we can't really tell without seeing some numbers. |
Reduce every row unconditionally instead of branching on validity per row: a null or empty row folds its (usually empty) coordinate range into meaningless corners that sit under a null in the output and are never observed. The output validity is computed separately as a bulk mask combine — the operand's null mask AND a non-empty mask built with BitBuffer::collect_bool over the row offsets. Benchmarked against the branchy and hybrid alternatives across geometry shapes and null densities (none / 10% periodic / 50% pseudo-random): the branch-free form is best or tied on list-backed geometries and makes runtime independent of the null distribution. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
BufferMut::push stays an out-of-line call (its grow path defeats inlining), so the envelope row loop paid four calls plus capacity checks per row. Pre-size the four corner buffers and write by index, and iterate offset pairs to drop the per-row bounds check on row_offsets[r + 1]. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Point storage is the coordinate struct itself, so each row's box is degenerate: project the x/y ordinate arrays straight into the corner columns (both corners alias the same buffers) and keep the operand's validity lazy, skipping the flatten and reduction entirely. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
push (BufferMut::with_capacity plus four out-of-line push calls per row, the shape the pre-sized loop replaced) vs the shipped indexed stores over pre-sized buffers: 2x at one vertex per row, 1.6x at four, a wash by 64 vertices, and no size regime where indexed loses (Apple M5 Max). Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Comment on lines
+85
to
+87
| let non_empty = Mask::from(BitBuffer::collect_bool(len, |r| { | ||
| row_offsets[r] < row_offsets[r + 1] | ||
| })); |
Member
There was a problem hiding this comment.
why not just calculate the final validity mask here instead of &ing it later with the valid? You could iterate the valid mask (I believe there is a method for that)? Maybe it won't affect perf at all though
Contributor
Author
There was a problem hiding this comment.
It will make it slower 6%-33%.
- Execute list levels straight to ListViewArray instead of Canonical::into_listview (same canonicalization, no panicking downcast), in both flatten walks. - dtype().is_list() over matches!(.., DType::List(..)) (review nit), swept across envelope.rs and extension/mod.rs. - Document why row_boxes combines two masks word-at-a-time rather than folding validity into the collect_bool closure: both the per-index and Mask::iter forms bench 6-33% slower end-to-end. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Replace the write-path shape comparison (its numbers are preserved in that commit's message) with an end-to-end benchmark of vortex.geo.envelope: the lazy ScalarFnArray executed to Canonical through a geo session, over harness-built columns. Cases cross nesting depth (Point, MultiPoint 32 pts/row, MultiPolygon 2 polygons x 2 rings x 8 vertices) with validity (non-nullable, ~10% periodic nulls, ~50% pseudo-random nulls). A _test-harness feature plus a self dev-dependency exposes test_harness to benches, the vortex-array pattern. Medians on an Apple M5 Max, 131072 rows per case: point ~0.8us at any validity (the zero-copy fast path makes execute O(1), so this is dispatch overhead), multipoint 1.39ms, multipolygon 2.17ms non-nullable / 2.11ms periodic nulls / 1.37ms random nulls (null rows are empty in storage and cost nothing). Row count scales linearly: throughput per row is identical at 1M rows. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
connortsui20
enabled auto-merge (squash)
July 21, 2026 21:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
GeoEnvelope(vortex.geo.envelope): the per-row 2-D axis-aligned bounding box of a native geometry column, returned as a nativegeoarrow.box(Rect) column.The consumer is a row-oriented spatial-join operator that bulk-loads an in-memory R-tree (
rstar::RTree::bulk_loadtakes individual objects), so it needs one box per geometry, read back row by row.How
ListViewstorage down to the leaf coordinateStruct, tracking which top-level row owns each coordinate, then min/max x/y per row over the rawf64buffers.Rectinput is the identity (the bounding box of a box is itself).x/yleaf ordinates are read; anyz/mare ignored — matching theGeometryAabbaggregate.validity()returnsNone, since the output null mask is not derivable from the operand's validity (empty geometries introduce nulls even over a non-nullable operand).