Adaptive (runtime, stats-based) conjunct reordering for FilterExec - #22698
Open
adriangb wants to merge 21 commits into
Open
Adaptive (runtime, stats-based) conjunct reordering for FilterExec#22698adriangb wants to merge 21 commits into
adriangb wants to merge 21 commits into
Conversation
github-actions
Bot
added
documentation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
adriangb
force-pushed
the
lift-selectivity-stats
branch
3 times, most recently
from
June 2, 2026 02:29
a24471d to
4d7b733
Compare
Contributor
Author
Contributor
Author
|
Benchmark summary for head 12c9a05 (no custom evaluator; learned order evaluated by Flag off → on
Main vs PR, flag off both sides
|
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 12c9a05 (12c9a05) to 12c9a05 diff Run configurationrun benchmark tpch10
baseline:
ref: "12c9a05490090905317181368e0915b02e7d6924"
changed:
ref: "12c9a05490090905317181368e0915b02e7d6924"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 12c9a05 (12c9a05) to 12c9a05 diff Run configurationrun benchmark tpch10
baseline:
ref: "12c9a05490090905317181368e0915b02e7d6924"
changed:
ref: "12c9a05490090905317181368e0915b02e7d6924"CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
Contributor
Author
|
A/A control on tpch10 (trigger: same PR head, same config on both sides) shows ±7–15% per-query swings (Q7 1.14x, Q9 1.13x, Q17 1.15x, Q22 1.12x) and a 4% total difference between two runs of the identical binary. So the flag-off tpch "regression" in the main-vs-PR run above is noise; nothing in the flag-off path shows up above the floor. The same floor applies to single per-query flags in the other suites, so the tpcds +1.8% net with the flag on is suggestive, not proven; the microbenchmark result behind it (cheap-predicate reorders adopted with no payoff) is the more reliable evidence. |
…act-once core) Add runtime, statistics-based conjunct reordering for `FilterExec`, off by default behind `datafusion.execution.adaptive_filter_reordering`. A conjunctive predicate is evaluated through a compact-once loop: conjunct masks are AND-combined and the working batch is physically compacted to the surviving rows once the accumulated mask is selective enough, so a selective conjunct shrinks the batch the conjuncts after it must decode. This compaction — not reordering a fused `BinaryExpr` AND, which does not compact between conjuncts — is the source of the win, and reordering compounds it. Each conjunct is timed and counted on the rows that reach it during a short warm-up; the conjuncts are then ranked by rows discarded per nanosecond (`(1 - pass_rate) / cost_per_row`) and, if the ranked order is materially cheaper than the written one, it is adopted and frozen. Results, plan, and EXPLAIN are unchanged; volatile predicates are never reordered. This is the minimal core. Benchmarks (predicate_eval) confirm it captures the "buried selective conjunct" wins (costsel_q01 ~-14%, width ~-12%) but also that compact-once regresses cheap-predicate conjunctions (cardinality k8 ~+37%) where the compaction overhead is not repaid — a guard that keeps the plain fused evaluation for those is added in the next commit. Cross-stream sharing, drift re-measurement, and confidence-interval statistics are later layers. Tested by unit tests (compact-once result-equivalence in any order, ranking, expected-cost weighting, adopt/keep decisions) and an end-to-end `adaptive_filter.slt` asserting identical results and EXPLAIN with the flag on and off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lh7i9DyeFWuTFWjogrVNkb
A `FilterExec` is split across many partition streams, each seeing a slice of the data. With per-stream warm-up, every stream pays its own measurement cost, and when each stream is only a handful of batches long that warm-up is most of its work — so the reordering win never materialises (and the warm-up overhead shows up as a regression). Benchmarked: at 12 partitions the costsel_q01 win collapsed from -67% (single stream) to -14%. Share the measurements. `AdaptiveFilterShared` holds a per-conjunct stats pool plus a settled-order epoch, common to every stream of one `FilterExec`. Each stream measures a batch into a local accumulator and folds it into the pool; the first stream to reach `WARMUP_BATCHES` pooled batches decides the order and publishes it by bumping the epoch. Other streams poll the epoch with one relaxed atomic load per batch and adopt the published order without paying warm-up. The warm-up is thus paid ~once per query, not once per stream. Restores the recovered win at default partitioning: width -56..-66%, costsel_q01 -58%, cardinality k16 -26% (was -12%, -14%, -6% without sharing), matching or beating the full design. Steady-state regressions on conjunctions where compaction does not pay (neutral_q61 ~+11%, cardinality k4 ~+5%, costsel_q02/q03 ~+3-5%) remain — a Fused-vs-CompactOnce guard addresses those in the next commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lh7i9DyeFWuTFWjogrVNkb
The compact-once loop wins by gating expensive conjuncts behind a selective one, but its per-conjunct bookkeeping (mask AND, true_count, the compaction copy) is pure overhead when there is nothing to gate. On a conjunction of interchangeable predicates — several equally expensive, equally unselective regexps, say — the warm-up settles on the written order (nothing to reorder) yet still paid compact-once on every batch, regressing ~11% vs the plain predicate (predicate_eval neutral_q61). Guard it: compact-once is adopted only when the warm-up actually reorders the conjuncts. When the settled order equals the written order, evaluate the predicate as-is — byte-for-byte the flag-off path, zero overhead. Since every real win reorders (a selective conjunct moves toward the front), this keeps the full win while removing the no-reorder regression. predicate_eval (vs flag off): neutral_q61 +11% -> ~0; wins preserved (costsel_q01 -60%, width -58..-66%, cardinality k16 -31%). A small residual remains on low-cardinality cheap conjunctions that do reorder (k4/k8 ~+3-4%), where compaction's cost is not repaid by gating so few/cheap predicates; the full champion/challenger arbiter regresses these more (~+10%), so a heavier guard is not worth it here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lh7i9DyeFWuTFWjogrVNkb
Two points raised by @xudong963 that carried over into the compact-once rewrite of the adaptive `FilterExec` conjunct evaluator: - Replace `.expect("u32 live")` on the live-row index downcast with a let-else returning `internal_err!`, so a broken invariant surfaces as a clean error rather than a panic. - Add a `debug_assert!` documenting that live-row indices are tracked in arrow's `u32` `filter`/`take` index space, making the `num_rows as u32` cast's precondition explicit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R1mvYrjFyTy2kbBoGrWzT6
@xudong963 noted the pooled adaptive-conjunct measurements live on the `FilterExec` plan node and are reused by every `execute()` call, leaking the learned conjunct order across independent executions. Implement `ExecutionPlan::reset_state` for `FilterExec` — the sanctioned mechanism for exactly this (its trait docs cite `DynamicFilterPhysicalExpr`; `CrossJoinExec`, `HashJoinExec`, and `SortExec` use it for their build-side / dynamic-filter state). It returns a fresh node with a new `AdaptiveFilterShared` (and fresh metrics), so a re-executed plan re-learns from scratch, while preserving the still-valid predicate, input, and cached plan properties. Reordering only ever affects performance, never results, so this closes a perf-staleness gap, not a correctness bug. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R1mvYrjFyTy2kbBoGrWzT6
- An empty batch no longer consumes the warm-up: a run of empty batches would settle the written order on no evidence, permanently disabling adaptation for the stream. - A conjunct evaluated faster than the timer's resolution now clamps its cost to 1ns instead of dropping out of the ranking as unmeasured (which sorted the cheapest conjunct last — backwards). - The u32::MAX row-count guard is now a real internal error instead of a debug_assert; in release the indices would have silently wrapped. Also documents the known limitations of the one-shot, conditional-stats settle (correlated conjuncts, no drift re-measurement) in the module doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DgXQmKrKgre4epbcSxUHNo
The previous test stored the table as a single batch, so with WARMUP_BATCHES = 8 the flag-on queries only ever exercised the measuring path. Store 4000 rows as 64-row batches so the warm-up completes and the settled (possibly reordered) path runs end-to-end, and add a query whose conjunct produces NULLs to exercise the null-mask path through real SQL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DgXQmKrKgre4epbcSxUHNo
The config docs claimed reordering was the only observable difference; in fact side effects of fallible predicates can change even when no reorder is adopted, because while measuring (and after a reorder) conjuncts are evaluated only on rows that survived the conjuncts before them. Say so explicitly, with an example, and note in FilterExec that Clone sharing the pooled measurements is deliberate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DgXQmKrKgre4epbcSxUHNo
…atch strategy Two review responses: - AdaptiveConjunction::try_new no longer takes an `enabled` bool that short-circuits to None; whether the feature is on is FilterExec's policy, so the flag check moves to FilterExec::execute and try_new answers only the structural question (reorderable, non-volatile conjunction). - The evaluator's per-batch behaviour is now observable: evaluate is a thin wrapper over evaluate_traced, which also reports the BatchStrategy used (Measure / Fused / Reordered). Two scenario tests exercise the input/output contract end to end — batches in, masks + strategy trace out — with per-conjunct costs injected by seeding the shared pool with synthetic measurements (the stand-in for a mocked clock), so which strategy gets adopted is deterministic: warm-up settles on a reorder for cheap-unselective + expensive-selective conjuncts, and on the written fused predicate for interchangeable ones. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DgXQmKrKgre4epbcSxUHNo
`AdaptiveFilterShared` carried an `epoch` atomic that unsettled streams polled once per batch, taking the mutex only when it changed. The atomic bought nothing: every unsettled stream already locks `inner` on each non-empty measured batch to pool its counts, so it was one extra word of state and a second synchronisation point for the same information. Unsettled streams now read the published decision straight from the mutex at the top of `evaluate_traced` — the same adoption point the epoch had, before the current batch is evaluated — and `pool_and_maybe_settle` adopts a decision another stream published between the two lock acquisitions instead of returning early. A settled stream never touches the shared state at all, as before. Also replace the defensive stats-length reset in `pool_and_maybe_settle` with lazy init plus a `debug_assert_eq!`: no site shares one `AdaptiveFilterShared` across different predicates (the builder, predicate rewrites and `reset_state` all allocate fresh; `Clone`, `with_fetch` and `with_batch_size` share it for the same predicate), so a length mismatch is a bug, not a case to paper over. No behavioural change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…metrics Adaptive conjunct reordering was invisible from the outside: with the flag on there was no way to tell from a real query whether the runtime actually adopted a reordered evaluation order, or settled on the written one. Add an `adaptive_reorders` counter to `FilterExecMetrics`, incremented once per partition stream at the batch on which that stream adopts a reordered (compacting) decision — including streams that pick up a decision another stream published. `AdaptiveConjunction` stays free of metrics types: it exposes a one-shot `take_adopted_reorder()` transition signal and the `FilterExec` stream does the counting. The counter is registered only when adaptive reordering is enabled for the execution, so the default flag-off path's metrics are unchanged. Also assert in `adaptive_filter.slt` that the reorder happens, via `EXPLAIN ANALYZE` on a predicate written selective-conjunct-last, and add the flag-off `EXPLAIN` that the file's "identical on and off" claim was asserting against nothing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…FilterExec coverage `streams_pool_measurements_and_share_settled_order` derived its `order == [1, 0]` / `compact` assertions from the real `Instant` timings of eight hundred-row batches, so the settle guard made it depend on the measured cost ratio staying inside the material-win window — a scheduling hiccup on a shared runner flips it. Seed the shared pool two batches short of the warm-up, as the scenario tests already do, so the two real batches cannot move the ranking. Also correct its comment: `rem_euclid(25) < 5` keeps 5 rows in 25 (20%), exactly the compact-once threshold. `no_reorder_evaluates_plain_predicate` still measures real timings and is left that way on purpose: both conjuncts pass ~96% of rows, and above `1 - TIE_COST_FRACTION` the material-win guard cannot hold for any positive costs, so no timing can produce a reorder there. Its doc now says so. New tests: - `adaptive_filter_reordering_end_to_end` (filter.rs): a real four- partition, sixteen-batch-per-partition `FilterExec` run with `adaptive_filter_reordering` on. Asserts the rows equal the flag-off output, NULLs are dropped, `adaptive_reorders` is registered only with the flag on and counts an adoption, and that re-executing the node (state kept) and re-executing after `reset_state` (state dropped) both produce identical rows. - `adopted_reorder_can_introduce_a_divide_by_zero` and `adopted_reorder_can_avoid_a_divide_by_zero_the_written_order_raises`: the two directions of the side effect the config option's doc warns about. The fused `BinaryExpr` `AND` pre-selects at its own 20% threshold, so `b <> 0 AND 1 / b > 2` succeeds flag-off when `b <> 0` holds for 15% of rows and errors once the conjuncts are reordered; mirrored, `1 / b > 2 AND a < 10` errors as written and succeeds once the selective conjunct is promoted and compacts the zeros away. - `first_measured_batch_initialises_the_shared_pool`: the lazy `stats.is_empty()` sizing of the pooled registry, including that an empty batch does not size it. The `FilterExec` conjuncts are too cheap for real timings to separate reliably, so the pool is seeded through a new `#[cfg(test)]` `AdaptiveFilterShared::seed_one_batch_short_of_warmup` — the same mocked-clock stand-in the scenario tests use, reachable from filter.rs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Documentation-only pass over the adaptive filter, applying alamb's review nits and correcting the stated baseline. - Module doc: describe what conjunct evaluation already does today — `reorder_predicates` orders conjuncts cheap-before-expensive by a static cost class, and `BinaryExpr`'s `AND` pre-selects when the conjuncts so far keep <= 20% of the rows and produce no nulls. Spell out what pre-selection cannot do (gate on a later conjunct, fire through nulls, carry survivors compacted across a nested chain) instead of claiming the `AND` evaluates every conjunct on every row regardless of order. - Add an intra-doc link to `BinaryExpr`, drop the "left-deep fused" jargon, expand the `regexp_like` example into a before/after evaluation order, and leave the flag's default value documented on the flag itself. - Drop the unsupported "compact-once is itself a win even without reordering" claim; point at the PR for the measurements rather than quoting numbers. - Fold the side-effect caveat, the conditional-statistics caveat and the one-shot caveat into a single "Known limitations" list instead of repeating them across the module; settle on one vocabulary (a decision is *settled*, a stream *adopts* it) and remove the leftover "publishes", "frozen" and "A/B-validated" wording. - Config doc: shorter, and the side-effect caveat is now bidirectional (a divide-by-zero can appear *or* disappear). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… AND Once the warm-up adopts a reorder, materialise the learned order once as a right-nested `AND` chain, `(c_first AND (c_second AND (... AND c_last)))`, and let `BinaryExpr` evaluate it like any other predicate, instead of running the settled path through the per-conjunct compact-once loop. Right-nesting is what makes this cheap: `BinaryExpr`'s pre-selection filters the batch it is handed before evaluating its right-hand side, so the survivors of the first (most selective) conjunct stay compacted for the entire remainder of the chain. A left-nested chain -- what `conjunction()` builds -- re-filters the original batch and scatters at every level, which measures materially slower than the flag off on cheap 4-8 conjunct predicates; right-nested is within noise of the compact-once loop. The measuring path keeps the per-conjunct loop with compaction: it has to time each conjunct on exactly the rows that reach it. `Settled` now carries the expression to evaluate alongside the order, and `settle` builds it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… of a private evaluation loop The warm-up used to walk the conjuncts itself: a private `eval_conjuncts` loop that AND-ed the masks, compacted the working batch past its own selectivity threshold, tracked live row indices and scattered the result back to full length. That was a second conjunction-evaluation engine living next to `BinaryExpr`'s, with its own compaction policy to keep in step and its own null and index handling to get right. Delete it. Each conjunct is now wrapped in a small measuring `PhysicalExpr` that times the call and counts the rows it was handed and the rows it kept, and the wrapped conjuncts are assembled into the written order as the same right-nested `AND` chain the settled path uses. `BinaryExpr` evaluates and pre-selects exactly as it would for the plain predicate, so the compaction is its own and every conjunct is measured on precisely the rows it hands over. The wrapper returns the conjunct's result unchanged, nulls included; three-valued logic stays `BinaryExpr`'s business. The module now holds no evaluation logic of its own on any path: stats and the shared pool, the ranking and cost model, `settle` plus the right-nested chain builder, the measuring wrapper, and the per-stream glue. Behaviour is unchanged where it was observable: the pooled counts for the first measured batch are identical, both divide-by-zero side-effect tests still hold (the warm-up's pre-selection keeps `1 / b` away from the zeros exactly as the old loop's compaction did), and the sqllogictests are unaffected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ith_adaptive_reorder_metrics Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…a pre-evaluation lock When the warm-up keeps the written order, evaluate it as the same right-nested AND chain an adopted order uses instead of the planner's left-nested expression: a left-nested chain pre-selects on the accumulated prefix and pays a whole-batch filter and scatter at every level where that prefix crosses the threshold. Unsettled streams no longer take the shared lock before evaluating a batch; a decision made meanwhile is picked up when the batch's counts are pooled, and those counts are discarded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Remove the test-only tracing machinery from the production path, simplify the metric plumbing and cut the documentation down to saying each thing once, where a reader first meets it. No behaviour change. - delete `BatchStrategy` and `evaluate_traced`; `evaluate` is the only entry point, and `order` is gone from `Settled` and `AdaptiveConjunction` (`reordered` is the decision). The tests that used the trace now assert on `settled`/`reordered` and on the shape of `settled_predicate`. - `AdaptiveConjunction::try_new` takes the `adaptive_reorders` `Count` and increments it in `adopt`, so `adopted_reorder`, `take_adopted_reorder` and `FilterExecMetrics::record_adaptive_reorder` are gone and the poll loop is a plain two-arm match. `execute` decides `AdaptiveConjunction::applies` first so the counter exists before the evaluator that increments it, and is still registered exactly when the adaptive path is active. - `right_nested_conjunction` is infallible, which lets `settle` drop its `predicate` fallback and `AdaptiveConjunction` drop the field. - one seeding helper (`seed_one_batch_short_of_warmup`) for both the module tests and the `FilterExec` end-to-end test. - drop `AdaptiveFilterShared::new` and `MeasuredConjunct::return_field` (the trait default derives the same field from `data_type`/`nullable`). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
adriangb
force-pushed
the
lift-selectivity-stats
branch
from
September 7, 2026 16:38
8ae31d3 to
be499fa
Compare
Contributor
Author
|
I pulled out benchmarks so that they can outlive this PR and serve as a baseline: #25032 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Predicate evaluation order matters: a selective conjunct run first gates the
work of the conjuncts after it. Two mechanisms already order and gate
conjuncts, and both decide statically:
cost class (perf: Reorder predicates in conjuncts via simple heuristic #22343). It is blind to selectivity, so a cheap-but-unselective
conjunct still sorts ahead of an expensive-but-very-selective one, and
conjuncts within one class keep their written order;
BinaryExpr'sANDpre-selects: when the conjuncts evaluated so far keepat most 20% of rows (and produce no nulls) it filters the batch before
evaluating the next one. It cannot gate a conjunct on a more selective one
written after it.
This PR adds runtime, statistics-based reordering for
FilterExec: eachconjunct's selectivity and per-row cost are measured on the rows that reach
it, the conjuncts are ranked by rows discarded per nanosecond, and the ranking
is adopted only if it is materially cheaper than the written order. Once
adopted, the learned order is materialised once as an ordinary
ANDchain andevaluated by
BinaryExprlike any other predicate. The module contains noconjunction-evaluation logic of its own on any path; it only measures, ranks,
and builds the chain. It is off by default
(
datafusion.execution.adaptive_filter_reordering).What changes are included in this PR?
Everything lives in a new private module
datafusion/physical-plan/src/adaptive_filter.rs;FilterExecgains a sharedper-execution state field, a metric, and a two-arm match in the stream poll
loop.
operator) the written order is evaluated by
BinaryExpras a right-nestedANDchain in which each conjunct is wrapped in a small measuringexpression that records rows seen, rows passed and time.
BinaryExpr'spre-selection does the compaction, so each conjunct is measured on exactly
the rows
BinaryExprhands it. The wrappers disappear once the order issettled.
(1 + rows_in - rows_out) / time(the key Velox uses,Pedreira et al. VLDB 2022); adopt the ranking only if its expected cost is
at least 5% below the written order's. If not, the written order is kept.
AND: whichever order settles is builtonce as
c1 AND (c2 AND (... AND cn))and handed toBinaryExpr.Right-nesting is what makes this work:
BinaryExprpre-selection filtersthe batch it is given, so the survivors of the first conjunct stay compacted
through the rest of the chain. A left-nested chain (what the planner and
conjunction()build) pre-selects on the accumulated prefix and re-filtersthe original batch and scatters back at every level; see the measurements
below. This also applies when the written order is kept: on
predicate_evalit made no difference on 20 of 24 shapes and was 21–31%faster on the two many-cheap-conjunct shapes whose accumulated prefix
crosses the 20% pre-selection threshold while no single conjunct does.
adaptive_reordersonFilterExec(per partition, only presentwhen the flag is on) shows in
EXPLAIN ANALYZEwhether a reorder wasadopted.
reset_stategives re-executions fresh measurements; predicate rewrites reset the pooled
state; results are order-independent.
configs.md/information_schema.Known limitations (documented in the module): measurements are
conditional on the written order, so correlated conjuncts can be misjudged;
the settle is one-shot with no drift re-measurement; the settle cost model
does not yet include evaluation overhead, so on very cheap predicates a
reorder can be adopted that buys nothing (see k4 below).
Measurements behind the settled-path design
Same binary, settled path selected by an environment switch, 8 interleaved
rounds × 40 iterations on
predicate_eval, ratios of medians. A = flag off,B = a dedicated compact-once evaluation loop (an earlier revision of this PR),
C = learned order rebuilt as a left-nested
AND, D = learned order rebuilt asa right-nested
AND(this PR).The right-nested rebuild matches the dedicated loop everywhere except a
~10% residual on the 0.6 ms k4 query, which is evaluator fixed cost on a
reorder that buys nothing there; tightening the settle guard to account for
evaluation overhead is a follow-up.
tpch_sf10 (same binary, flag off → on): Q6 1.19× faster, Q12 1.45× faster,
all other queries unchanged; tpcds_sf1 and clickbench neutral within the A/A
noise floor. See the bot runs in the PR comments.
Are these changes tested?
the right-nested shape of the rebuilt chain, the metric transition, lazy
pool init, and both directions of the fallible-predicate side effect (an
adopted reorder can introduce or avoid a divide-by-zero).
FilterExectest (4 partitions, nullable column,repeated execution with and without
reset_state).adaptive_filter.slt: results identical on and off,EXPLAINidentical onand off, and an
EXPLAIN ANALYZEassertion thatadaptive_reordersfireson a predicate written selective-last.
Are there any user-facing changes?
One new config option,
datafusion.execution.adaptive_filter_reordering(experimental, default false), and one new
FilterExecmetric,adaptive_reorders. When enabled, query results never change, but theobservable side effects of fallible predicates can, in either direction:
reordering
b <> 0 AND 1/b > 2can make a divide-by-zero error appear ordisappear. Predicates containing volatile expressions are never reordered.