bench(predicate_eval): add expensive-first and wide-column shapes

[adriangb]

Which issue does this PR close?

• Part of [EPIC] Adaptive predicate evaluation #22883
• Related to Reorder boolean expressions (including filter predicates) according to evaluation cost / selectivity #11262
• Prequel split out of Adaptive (runtime, stats-based) conjunct reordering for FilterExec #22698: these are the benchmark-suite additions from that PR, carried on their own so they can land (and be run) independently of the adaptive-filter work.

Rationale for this change

The predicate_eval suite (the conjunctive-filter microbenchmarks added for adaptive predicate ordering in #11262) was missing a few shapes that distinguish evaluation strategies for AND chains, and it had no result checking at all:

1. An expensive-selective conjunct written before a cheap-unselective one. The suite already had the reverse (costsel q03: cheap c0 < 90 at ~90% first, expensive regexp_like(s, 'rare') at ~0.1% second). Without the mirror shape there is no case in the suite where the as-written order is already the optimal one, so the suite cannot tell "reordered well" apart from "reordered at all". It also exercises a distinct code path: measuring a cheap conjunct on the small survivor batch left after pre-selection by an expensive one, rather than measuring an expensive conjunct on a full batch.

2. A many-column table. Every existing cardinality query runs over the 16-column ints dataset, so the cost of materializing a filtered batch per level is small and roughly constant across the suite. Re-running the k = 8 predicate over a 64-column table separates the cost of evaluating the conjuncts from the cost of filtering the batch at each level, which is what per-level batch filtering actually pays for on wide inputs.

3. No result validation. The suite's template had no result directive, so --result-mode validate verified nothing on it. A reordering under test must not change which rows survive the filter, and every query here is a count(*) whose value is fixed by the generated data — a checked-in expected count is therefore a free correctness check on top of the timing.

4. Nullable conjuncts. Every predicate in the suite was non-nullable, so the suite never exercised the one case where writing the selective conjunct first buys nothing: BinaryExpr's AND pre-selection bails out when the left-hand boolean array contains any NULL.

5. Drift that a warm-up cannot absorb. The existing drift dataset flips which conjunct is selective 10% of the way through the table, which at the default PRED_ROWS is within the first few batches — a one-shot warm-up already lands on the post-flip order and is right for the remaining ~90% of the scan. There was no shape where a decide-once strategy is wrong for a large part of the scan, and none at all where the right order differs between concurrently running partitions rather than over time.

What changes are included in this PR?

Three commits, all entirely under benchmarks/ (plus the suite's own docs).

1. Expensive-first and wide-column shapes — five new files under benchmarks/sql_benchmarks/predicate_eval/:

• queries/costsel/q04.sql + benchmarks/costsel/q04.benchmark — costsel_q04_expensive_selective_then_cheap_unselective: the mirror of q03. Same two predicates, written the other way round, so the as-written order is already the best one. Together q03/q04 bracket a reorderer's behaviour when cost and selectivity point the same way.
• queries/cardinality/q34.sql + benchmarks/cardinality/q34.benchmark — cardinality_q34_k8_wide64: q32's k = 8 predicate (seven ~90% compares followed by one ~5% compare) run over the new 64-column dataset.
• load/ints_wide.sql — ints widened from 16 to 64 Int64 columns. It extends ints.sql's multiplier sequence rather than replacing it: c0..c15 keep exactly the same multipliers (all coprime to 100, so residues stay uniform and columns stay mutually decorrelated), which means q34's predicate has exactly the same hidden selectivities as q32's. Only the width of the batches flowing through the filter changes.

2. Result validation, a nulls subgroup, and a late-flip drift shape:

• Result validation. The shared template gains result sql_benchmarks/predicate_eval/results/${NAME}.csv, and the expected count(*) for all 32 benchmarks (existing and new) is persisted under predicate_eval/results/. --result-mode validate now checks the row counts as well as running the timing. Caveat, documented in the template and the suite's README entry: the counts were persisted at the suite defaults (PRED_ROWS=1000000, PRED_FILL=30), so validation assumes those. The scale and width subgroups pin their own PRED_ROWS / PRED_FILL as template parameters (which win over the environment), so those validate at any setting of the knobs.
• nulls subgroup (load/nulls.sql, queries/nulls/q90.sql, q91.sql, and their .benchmark files). c_sel is NULL on exactly 10% of rows and uniform on [0,100) elsewhere, so c_sel < 5 is true on exactly 4% of rows and NULL on 10%. check_short_circuit returns ShortCircuitStrategy::None as soon as the left-hand boolean array has null_count() > 0, so the nullable selective conjunct gates nothing wherever it is written: q90 writes it first, q91 writes it last. A reorderer that ranks by selectivity alone will hoist it to the front and gain nothing, which is what these two measure.
• drift q82 — late flip (load/drift_half.sql). The same mirrored predicates as q80/q81, but the flip is at the halfway point of the table, so a warm-up-and-freeze decision is wrong for half the scan.
• Docs: the predicate_eval row in benchmarks/sql_benchmarks/README.md now lists the nulls subgroup and the result-validation behaviour with its default-PRED_ROWS caveat; benchmarks/bench.sh and the suite description list nulls too, and bench.sh's run_predicate_eval comment notes that q83 writes into predicate_eval/scratch/ and pins its own target_partitions.

3. Per-partition drift (q83) — replaces the alternating-block shape from commit 2, which could only approximate it:

• drift q83 — per-partition skew (load/drift_files.sql). Each scan partition gets its own fixed selectivity profile, so one global ordering decision is necessarily backwards for half of the streams — the case a decide-once or re-sample-a-shared-decision strategy cannot get right, and a per-stream decision can. That shape is not reachable from a CREATE TABLE ... AS SELECT MemTable: a generate_series scan is one partition fanned out by RepartitionExec with RoundRobinBatch(target_partitions), which deals whole batches round-robin, so partition p holds batches p, p+P, p+2P, ... and any contiguous block of rows is sprayed across every partition. The load script therefore writes 16 small Parquet files instead — PRED_ROWS / 16 rows each, odd files favouring a_sel = 0 (~0.1%, against b_sel = 0 at ~50%) and even files the mirror, at the same rates q80/q81/q82 use — into a gitignored predicate_eval/scratch/ (mirroring parquet_row_filter_skip), and registers the directory as one external table.
  Two read-side settings pin one file per partition, and neither alone is enough. target_partitions = 16 matches the file count, so FileGroup::split_files puts one file in each group and EnforceDistribution adds no RoundRobinBatch above the scan; at the machine default (12 cores here) the 16 files chunk into 8 groups of 2 — one file of each profile per group — and a RoundRobinBatch(12) then re-deals those batches anyway. repartition_file_scans = false stops FileGroupPartitioner re-deriving the groups as byte ranges over the total file bytes, which ignores file identity and at the default yields groups like [f00:0..224901, f01:0..74967]. load and the benchmarked run share one SessionContext (SqlBenchmark::initialize), so both settings reach the measured query. Pinning target_partitions also makes the shape identical on every machine, at the cost of over- or under-subscribing cores relative to the other drift queries; the load script and query comment say all of this.

No Rust changes: the suite discovers the new .benchmark files automatically, and the only non-benchmark edits are the README/bench.sh/.suite text above. The suite still sets no engine config globally — the two settings above live in q83's own load SQL, on that benchmark's own SessionContext, because they are part of the data shape rather than a strategy under test.

What is the testing strategy for this PR?

No unit tests: these are benchmark definitions, not library code. Verification is that the suite parses, every query runs and produces a result row, and every query's result matches its checked-in expected count.

All ten subgroups pass --result-mode validate at the default PRED_ROWS (32 benchmarks, -i 1, release build, on this branch on top of main with no other changes):

$ cargo run --release --bin benchmark_runner -- predicate_eval --subgroup nulls -i 1 --result-mode validate
predicate_eval/nulls_q90_nullable_selective_first/nulls iteration 0: 0.7 ms, 1 rows
predicate_eval/nulls_q91_nullable_selective_last/nulls iteration 0: 0.6 ms, 1 rows

$ cargo run --release --bin benchmark_runner -- predicate_eval --subgroup drift -i 1 --result-mode validate
predicate_eval/drift_q80_a_then_b/drift iteration 0: 0.5 ms, 1 rows
predicate_eval/drift_q81_b_then_a/drift iteration 0: 0.3 ms, 1 rows
predicate_eval/drift_q82_late_flip/drift iteration 0: 0.4 ms, 1 rows
predicate_eval/drift_q83_per_partition_skew/drift iteration 0: 1.0 ms, 1 rows

$ cargo run --release --bin benchmark_runner -- predicate_eval --subgroup costsel -i 1 --result-mode validate
predicate_eval/costsel_q01_regexp_selective_last/costsel iteration 0: 5.8 ms, 1 rows
predicate_eval/costsel_q02_regexp_selective_first/costsel iteration 0: 2.0 ms, 1 rows
predicate_eval/costsel_q03_cheap_unselective_then_expensive_selective/costsel iteration 0: 1.3 ms, 1 rows
predicate_eval/costsel_q04_expensive_selective_then_cheap_unselective/costsel iteration 0: 1.4 ms, 1 rows

$ cargo run --release --bin benchmark_runner -- predicate_eval --subgroup cardinality -i 1 --result-mode validate
predicate_eval/cardinality_q30_k2/cardinality iteration 0: 0.5 ms, 1 rows
predicate_eval/cardinality_q31_k4/cardinality iteration 0: 0.8 ms, 1 rows
predicate_eval/cardinality_q32_k8/cardinality iteration 0: 1.2 ms, 1 rows
predicate_eval/cardinality_q33_k16/cardinality iteration 0: 2.7 ms, 1 rows
predicate_eval/cardinality_q34_k8_wide64/cardinality iteration 0: 1.4 ms, 1 rows

cost, selectivity, width, scale, neutral and correlation pass the same way, as does a single run of the whole suite (predicate_eval -i 1 --result-mode validate: 32 benchmarks, exit 0).

Validation was checked to actually be checking: overwriting one expected count (nulls_q90 20000 → 999999) makes the run fail with

Error: Execution error: Error in result on row 1, column 1 running query "": expected value "999999" but got value "20000" in row: ["20000"]

and restoring the file makes it pass again.

The new datasets were also verified directly. drift_half has the flip exactly at the halfway row (a_sel = 0 at 0.0998% / 50.0% over the two halves, b_sel = 0 mirrored). nulls has exactly 100000 NULLs and 40000 rows with c_sel < 5 at PRED_ROWS=1000000.

drift_files was checked both per file and in the plan. Per file, f00 (even) has 62500 rows with a_sel = 0 on 31250 of them and b_sel = 0 on 62; f01 (odd) is the exact mirror; over the whole table count(*) = 1000000, a_sel = 0 and b_sel = 0 each on 250496 rows, and both on 992 — the persisted count.

The partitioning was confirmed with EXPLAIN (release build of this branch, 12-core box). One whole file per group, no byte ranges, and the filter directly on the scan:

physical_plan
ProjectionExec: expr=[count(Int64(1))@0 as count(*)]
  AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))]
    CoalescePartitionsExec
      AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))]
        FilterExec: a_sel@0 = 0 AND b_sel@1 = 0, projection=[]
          DataSourceExec: file_groups={16 groups: [[.../scratch/drift_files/f00.parquet], [.../scratch/drift_files/f01.parquet], [.../scratch/drift_files/f02.parquet], [.../scratch/drift_files/f03.parquet], [.../scratch/drift_files/f04.parquet], ...]}, projection=[a_sel, b_sel], file_type=parquet, predicate=a_sel@1 = 0 AND b_sel@2 = 0, ...

Both settings were checked to be load-bearing by dropping them. Without the target_partitions pin the scan plans as 8 groups of 2 files with a RoundRobinBatch(12) on top; with repartition_file_scans left on it plans as file_groups={12 groups: [[.../f00.parquet:0..224901, .../f01.parquet:0..74967], [.../f01.parquet:74967..224901, .../f02.parquet:0..149934], ...]} — both profiles in one stream either way.

Validation was checked to be checking on the new query too: setting drift_q83_per_partition_skew.csv to 991 fails with expected value "991" but got value "992", and restoring it passes.

The extra load cost is small. At the default PRED_ROWS, --query 83 -i 1 takes ~60 ms wall from a cleared scratch directory against ~10 ms for --query 80, and the whole drift subgroup (-i 1, cold scratch) runs in ~0.08 s.

That q34 and q32 have identical selectivities was checked separately: both return count(*) = 5000 at PRED_ROWS=100000.

Are there any user-facing changes?

No. Benchmark definitions only; nothing in this PR is part of any public API or affects query execution.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.72%. Comparing base (6ab4ce6) to head (8a1565a).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25032      +/-   ##
==========================================
- Coverage   81.72%   81.72%   -0.01%     
==========================================
  Files        1127     1127              
  Lines      416237   416310      +73     
  Branches   416237   416310      +73     
==========================================
+ Hits       340156   340213      +57     
- Misses      56092    56104      +12     
- Partials    19989    19993       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.