Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions datafusion/common/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -964,13 +964,14 @@ config_namespace! {
/// the new schema verification step.
pub skip_physical_aggregate_schema_check: bool, default = false

/// Temporary switch for aggregate stream implementations that are being
/// migrated from `GroupedHashAggregateStream`.
/// Temporary switch for the aggregate stream implementations that were
/// split out of `GroupedHashAggregateStream`.
///
/// When set to true, DataFusion tries the migrated implementations when
/// their preconditions are satisfied. When set to false, grouped
/// aggregation falls back to `GroupedHashAggregateStream`. This option
/// will be removed after the migration is finished.
/// The split is complete, so the default `true` plans every grouped
/// aggregation with the dedicated streams. When set to false, grouped
/// aggregation falls back to the legacy `GroupedHashAggregateStream`.
/// The fallback is kept in case of major bugs in the new streams, and
/// will be deleted after the 56.0.0 release together with this option.
///
/// See <https://github.com/apache/datafusion/issues/22710> for details.
pub enable_migration_aggregate: bool, default = true
Expand Down
8 changes: 5 additions & 3 deletions datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs
Original file line number Diff line number Diff line change
Expand Up @@ -299,9 +299,11 @@ async fn streaming_aggregate_test() {
}
}

/// Perform batch and streaming aggregation with same input
/// and verify outputs of `AggregateExec` with pipeline breaking stream `GroupedHashAggregateStream`
/// and non-pipeline breaking stream `BoundedAggregateStream` produces same result.
/// Perform batch and streaming aggregation with same input and verify that the
/// two `AggregateExec` variants produce the same result: the pipeline breaking
/// one over unordered input (`PartialHashAggregateStream`) and the
/// non-pipeline breaking one over ordered input
/// (`OrderedPartialAggregateStream`).
async fn run_aggregate_test(input1: Vec<RecordBatch>, group_by_columns: Vec<&str>) {
let schema = input1[0].schema();
let session_config = SessionConfig::new().with_batch_size(50);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -335,9 +335,9 @@ mod tests {
schema,
)?);

// This test is for `GroupByMetrics`, which are maintained by
// `GroupedHashAggregateStream`. Use a finite memory pool so the partial
// aggregate does not take the initial-partial stream path.
// This test is for `GroupByMetrics`, which every grouped aggregation
// stream records. The memory limit is large enough that the partial
// aggregate stays on the in-memory path.
let runtime = RuntimeEnvBuilder::new()
.with_memory_limit(10 * 1024 * 1024, 1.0)
.build_arc()?;
Expand Down
26 changes: 16 additions & 10 deletions datafusion/physical-plan/src/aggregates/grouped_hash_stream.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,20 @@
// specific language governing permissions and limitations
// under the License.

//! Hash aggregation
//! Legacy hash aggregation.
//!
//! # Deprecation
//!
//! [`GroupedHashAggregateStream`] handled every grouped execution path before
//! they were split into dedicated streams. It is no longer planned by default:
//! it is only reachable by setting
//! `datafusion.execution.enable_migration_aggregate` to `false`. It is kept as
//! a fallback in case of major bugs in the new streams, and will be deleted
//! after the 56.0.0 release together with that option.
//!
//! New features and improvements should go into the dedicated streams instead.
//!
//! See issue for details: <https://github.com/apache/datafusion/issues/22710>

use std::sync::Arc;
use std::task::{Context, Poll};
Expand Down Expand Up @@ -137,15 +150,8 @@ enum OutOfMemoryMode {

/// HashTable based Grouping Aggregator
///
/// # Development Note
///
/// This implementation is being incrementally refactored. See the tracking issue
/// for details.
///
/// New features and improvements should go directly into the new implementation.
/// Please coordinate through the tracking issue.
///
/// Issue: <https://github.com/apache/datafusion/issues/22710>
/// This is the legacy implementation. See the [module documentation](self) for
/// the deprecation schedule.
///
/// # Design Goals
///
Expand Down
5 changes: 0 additions & 5 deletions datafusion/physical-plan/src/aggregates/hash_stream.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,6 @@
//!
//! See comments in [`PartialHashAggregateStream`] and [`FinalHashAggregateStream`]
//! for details.
//!
//! Note these streams are an incremental migration of the existing
//! [`crate::aggregates::grouped_hash_stream::GroupedHashAggregateStream`].
//!
//! See issue for details: <https://github.com/apache/datafusion/issues/22710>

use std::mem::size_of;
use std::sync::Arc;
Expand Down
34 changes: 16 additions & 18 deletions datafusion/physical-plan/src/aggregates/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@
//! ```
//!
//! Every stage without grouping expressions uses [`AggregateStream`]. This path
//! is selected before the grouped-stream migration setting is considered.
//! is selected before any of the grouped streams are considered.
//!
//! ## 4. Grouped TopK aggregation
//!
Expand Down Expand Up @@ -135,10 +135,14 @@
//!
//! See [`PartialReduceHashAggregateStream`] for details.
//!
//! ## 6. Fallback grouped hash aggregation
//! ## 6. Legacy grouped hash aggregation
//!
//! [`GroupedHashAggregateStream`] is the legacy implementation for several of the
//! stream types above. It is being incrementally migrated to separate streams.
//! [`GroupedHashAggregateStream`] is the legacy implementation that all of the
//! grouped streams above were split out of. The split is complete, so it is no
//! longer planned: it is only reachable by setting
//! [`datafusion.execution.enable_migration_aggregate`](datafusion_common::config::ExecutionOptions::enable_migration_aggregate)
//! to `false`. That fallback is kept for one release and will then be removed
//! together with this stream.
//!
//! See the issue for details: <https://github.com/apache/datafusion/issues/22710>
#![expect(rustdoc::private_intra_doc_links)]
Expand Down Expand Up @@ -691,15 +695,11 @@ enum StreamType {
OrderedFinalAggregate(OrderedFinalAggregateStream),
/// Single stage of aggregation for ordered input.
OrderedSingleAggregate(OrderedSingleAggregateStream),
/// Hash aggregation reused for multiple stages
/// Legacy hash aggregation reused for multiple stages
///
/// Note this is being incrementally migrated to dedicated streams like
/// [`StreamType::PartialHash`], [`StreamType::FinalHash`],
/// [`StreamType::OrderedPartialAggregate`],
/// [`StreamType::OrderedFinalAggregate`], and
/// [`StreamType::OrderedSingleAggregate`]
///
/// See issue for details: <https://github.com/apache/datafusion/issues/22710>
/// Every path it handles now has a dedicated stream, so this variant is only
/// produced when `datafusion.execution.enable_migration_aggregate` is set to
/// `false`. See [`grouped_hash_stream`] for the deprecation schedule.
GroupedHash(GroupedHashAggregateStream),
/// Grouped TopK aggregate stream.
/// Input output scheme: initial input -> final result
Expand Down Expand Up @@ -1234,11 +1234,8 @@ impl AggregateExec {
// `enable_migration_aggregate` config option selects between the new
// streams and the legacy implementation.
//
// The legacy implementation is deprecated. It is kept for one more
// release as a fallback for potential bugs in the new streams, and
// will be removed after that.
//
// Issue: <https://github.com/apache/datafusion/issues/22710>
// See the `grouped_hash_stream` module documentation for the
// deprecation schedule.
if context
.session_config()
.options()
Expand Down Expand Up @@ -1292,7 +1289,8 @@ impl AggregateExec {
);
}

// Execution paths that have not been migrated use the fallback implementation
// `enable_migration_aggregate` is disabled: fall back to the legacy
// implementation, which handles every grouped execution path itself.
Ok(StreamType::GroupedHash(GroupedHashAggregateStream::new(
self, context, partition,
)?))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,14 +102,6 @@ use crate::{InputOrderMode, SendableRecordBatchStream, metrics};
/// group key and spills them as one run. After the input ends, it spills any
/// remaining states, performs a sort-preserving merge of all runs, and feeds the
/// merged input into a fully ordered final aggregate stream.
///
/// ## Implementation Note
///
/// This is intentionally kept simple and closely maps to
/// `GroupedHashAggregateStream` to finish the refactor sooner.
///
/// See issue for details: <https://github.com/apache/datafusion/issues/22710>
///
pub(crate) struct OrderedPartialAggregateStream {
schema: SchemaRef,
input: SendableRecordBatchStream,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,6 @@
// under the License.

//! Partial-reduce hash aggregation stream implementation.
//!
//! This stream is part of the incremental migration from
//! [`crate::aggregates::grouped_hash_stream::GroupedHashAggregateStream`].
//!
//! See issue for details: <https://github.com/apache/datafusion/issues/22710>

use std::ops::ControlFlow;
use std::sync::Arc;
Expand Down
5 changes: 0 additions & 5 deletions datafusion/physical-plan/src/aggregates/single_stream.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,6 @@
// under the License.

//! Single-stage hash aggregation stream implementation.
//!
//! This stream is part of the incremental migration from
//! [`crate::aggregates::grouped_hash_stream::GroupedHashAggregateStream`].
//!
//! See issue for details: <https://github.com/apache/datafusion/issues/22710>

use std::ops::ControlFlow;
use std::sync::Arc;
Expand Down
5 changes: 2 additions & 3 deletions datafusion/physical-plan/src/aggregates/skip_partial.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,7 @@ use crate::metrics;

/// Tracks if the aggregate should skip partial aggregations
///
/// See "partial aggregation" discussion on
/// [`crate::aggregates::grouped_hash_stream::GroupedHashAggregateStream`].
/// Used by [`crate::aggregates::hash_stream::PartialHashAggregateStream`].
pub(super) struct SkipAggregationProbe {
// ========================================================================
// PROPERTIES:
Expand All @@ -40,7 +39,7 @@ pub(super) struct SkipAggregationProbe {
// ========================================================================
// STATES:
// Fields changes during execution. Can be buffer, or state flags that
// influence the execution in parent `GroupedHashAggregateStream`
// influence the execution in the parent aggregate stream
// ========================================================================
/// Number of processed input rows (updated during probing)
input_rows: usize,
Expand Down
2 changes: 1 addition & 1 deletion datafusion/sqllogictest/test_files/information_schema.slt
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ datafusion.execution.coalesce_batches true When set to true, record batches will
datafusion.execution.collect_statistics true Should DataFusion collect statistics when first creating a table. Has no effect after the table is created. Defaults to true.
datafusion.execution.enable_ansi_mode false Whether to enable ANSI SQL mode. The flag is experimental and relevant only for DataFusion Spark built-in functions When `enable_ansi_mode` is set to `true`, the query engine follows ANSI SQL semantics for expressions, casting, and error handling. This means: - **Strict type coercion rules:** implicit casts between incompatible types are disallowed. - **Standard SQL arithmetic behavior:** operations such as division by zero, numeric overflow, or invalid casts raise runtime errors rather than returning `NULL` or adjusted values. - **Consistent ANSI behavior** for string concatenation, comparisons, and `NULL` handling. When `enable_ansi_mode` is `false` (the default), the engine uses a more permissive, non-ANSI mode designed for user convenience and backward compatibility. In this mode: - Implicit casts between types are allowed (e.g., string to integer when possible). - Arithmetic operations are more lenient — for example, `abs()` on the minimum representable integer value returns the input value instead of raising overflow. - Division by zero or invalid casts may return `NULL` instead of failing. # Default `false` — ANSI SQL mode is disabled by default.
datafusion.execution.enable_file_stream_work_stealing true When `true` (the default), DataFusion's built-in file scans dynamically rebalance files across partitions at query execution time: a partition that goes idle reads files (or byte-range morsels) originally assigned to a sibling partition, which keeps all partitions busy in a single process. Executors that depend on the plan-time partition assignment — such as Ballista and datafusion-distributed, which run each partition as an isolated task and never poll the siblings — should set this to `false` so each partition reads only its own file group and no runtime reassignment occurs.
datafusion.execution.enable_migration_aggregate true Temporary switch for aggregate stream implementations that are being migrated from `GroupedHashAggregateStream`. When set to true, DataFusion tries the migrated implementations when their preconditions are satisfied. When set to false, grouped aggregation falls back to `GroupedHashAggregateStream`. This option will be removed after the migration is finished. See <https://github.com/apache/datafusion/issues/22710> for details.
datafusion.execution.enable_migration_aggregate true Temporary switch for the aggregate stream implementations that were split out of `GroupedHashAggregateStream`. The split is complete, so the default `true` plans every grouped aggregation with the dedicated streams. When set to false, grouped aggregation falls back to the legacy `GroupedHashAggregateStream`. The fallback is kept in case of major bugs in the new streams, and will be deleted after the 56.0.0 release together with this option. See <https://github.com/apache/datafusion/issues/22710> for details.
datafusion.execution.enable_nlj_coordinated_fallback true Enables the memory-limited fallback for `NestedLoopJoinExec` join types that emit unmatched left rows in the final output (LEFT, LEFT SEMI, LEFT ANTI, LEFT MARK, FULL) when the right side has multiple partitions. This fallback coordinates per-chunk left state (visited bitmap and probe-thread counter) across all right-side partitions, which assumes every partition runs in the same process. Distributed engines that execute each output partition as an independent task (e.g. Ballista, datafusion-distributed) build a separate coordinator per task and poll only one partition, so the cross-partition counter never reaches zero and the fallback would stall. Such engines should set this to `false`: the coordinated fallback is then disabled for left-emitting multi-partition joins, which instead fail with a resource-exhaustion error under memory pressure rather than deadlocking. Single-partition and non-left-emitting joins are unaffected and always keep the fallback.
datafusion.execution.enable_recursive_ctes true Should DataFusion support recursive CTEs
datafusion.execution.enforce_batch_size_in_joins false Should DataFusion enforce batch size in joins or not. By default, DataFusion will not enforce batch size in joins. Enforcing batch size in joins can reduce memory usage when joining large tables with a highly-selective join filter, but is also slightly slower.
Expand Down
Loading
Loading