Integrate DataFusion as execution engine for compute-heavy operations

[qzyu999]

Feature Request / Improvement

Problem

PyIceberg cannot perform several operations at production scale due to unbounded memory requirements in the PyArrow execution path:

• Tables with equality deletes are unreadable (hard ValueError)
• CoW deletes OOM on large Parquet files (~1GB)
• CoW overwrite OOMs (same pattern as delete)
• Upsert uses O(n) row-by-row comparison
• Compaction not implemented, requires external sort (infeasible in-memory for large tables)
• Orphan file deletion OOMs (LEFT ANTI JOIN of millions of paths)

The list goes on (documented below) with operations that don't scale in the typical single-node environment of PyIceberg. These block PyIceberg from achieving feature parity with Java Iceberg for V2/V3.

Proposed Solution

Integrate Apache DataFusion (already exists via pip install 'pyiceberg[pyiceberg-core]') as an optional execution engine behind an automatic engine-resolution layer. When installed, compute-heavy operations use DataFusion's spill-to-disk execution (bounded memory). When not installed, the existing PyArrow path remains unchanged (works for small data, OOMs gracefully on large).

No existing behavior changes. No forced dependency. DuckDB-style UX where only a developer only needs to configure a memory budget if they so choose.

Update: We also choose to pivot the decision on the architectural boundaries after valuable feedback from @kevinjqliu. That is, we avoid dependency on iceberg-rust and allow PyIceberg to own the semantics and only use DataFusion as the execution engine for compute-heavy operations, like how we currently utilize PyArrow.

Design Doc

Support for PyIceberg DataFusion Integration

Relation to Existing DataFusion Integration

PyIceberg already uses DataFusion as part of a standard multi-engine interop pattern (e.g. to_duckdb(), to_ray(), etc.), these are all read-only query operations. This proposal is for an orthogonal role and the two are not to be conflated. This new role is for that of an internal implementation, for example if a user calls table.compact(), they'll never see the actions of DataFusion under the hood, similar to how PyArrow is used today.

Why DataFusion and Why Not a Pluggable Pattern

This is in indeed an opinionated architectural choice. DataFusion has strong positive attributes that don't collectively exist in any alternative:

• Apache-licensed (ASF ecosystem compatibility)
• Arrow-native (zero-copy interop with PyArrow — no serialization)
• Embeddable as a library (not a server/database)
• Spill-to-disk for sort, join, and aggregate (the core requirement)
• Python bindings exist and are maintained
  DuckDB has GPL-licensed extensions and copies at the Arrow boundary. Polars has no spill-to-disk. Spark requires a JVM. Velox has no Python bindings. PyArrow lacks the ability to temporarily spill-to-disk.

Why Not Make the Compute Engine Pluggable?

Building a pluggable compute engine abstraction layer right now introduces massive complexity with zero immediate benefit:

1. No viable alternatives exist: With only one candidate satisfying the requirements, there is no real choice to make.
2. Enormous abstraction surface: The surface area for memory configuration, SQL dialects, data registration, and object store integration differs drastically between engines. Standardizing this upfront is a massive undertaking.
3. Hidden implementation detail: The engine is an internal utility hidden behind operations like table.compact(). Users do not interact with it directly.
4. Existing escape hatches: Users who want to query data using alternative engines can already do so via the existing to_x() export methods.

If an alternative emerges, pyiceberg/execution/compute.py is a clean substitution point. Building pluggable infrastructure for theoretical future alternatives is a case of premature abstraction (YAGNI). If a viable alternative emerges in the future, the codebase contains a clean substitution point.

User Experience

DuckDB-style: sensible default memory budget, optional override via existing config mechanisms. No new initialization step required.

from pyiceberg.catalog import load_catalog

catalog = load_catalog("prod", uri="...")
table = catalog.load_table("db.events")

# Everything just works — default 512MB budget, spills to disk if needed
table.compact()                          # new method
table.delete("status = 'expired'")       # existing method, no longer OOMs on large files
df = table.scan().to_arrow()             # existing method, now resolves equality deletes

Memory budget is configurable through PyIceberg's existing config hierarchy:

# .pyiceberg.yaml
execution:
  memory-limit: 1GB

# Environment variable
export PYICEBERG_EXECUTION__MEMORY_LIMIT=2GB

Most users never configure this — the default handles typical workloads. When datafusion is not installed, PyArrow fallback is used unchanged. The install path:

pip install 'pyiceberg[datafusion]'

Implementation

We can create a new module pyiceberg/execution/ where we make it possible for the chosen query engine to resolve to either DataFusion or PyArrow, allow the memory limit to be configured, handle the mapping of PyIceberg storage properties, and encapsulate heavy data transformations safely away from the core table logic.

pyiceberg/execution/
├── __init__.py          # Re-exports
├── engine.py            # resolve_engine(), checks `import datafusion`
├── session.py           # create_bounded_session(), FairSpillPool + DiskManager
├── object_store.py      # Translate FileIO props -> DataFusion object store config
└── compute.py           # sort_batches(), anti_join(), filter_parquet()

Checklist

Foundation (no blockers)

• Engine resolution module
• Bounded-session helpers (configures RuntimeEnvBuilder.with_fair_spill_pool())
• Object store bridge (translate FileIO properties to DataFusion)

Operations Unblocked

• Equality delete read resolution
• Streaming CoW delete/overwrite
• Table compaction (sort + rewrite)
• Orphan file deletion
• Upsert via hash join
• Equality-to-positional conversion
• Position delete compaction
• Full MoR compaction
• Z-Order / Hilbert sorting
• DV compaction
• Incremental compaction
• Sort-order enforcement on write
• Dynamic partition overwrite (bounded memory)

Related Issues

PyIceberg

• [Feat] Support Merge-on-Read mode for Deletes #1078 (MoR support epic)
• [feature request] Support reading equality delete files #1210 / Equality Delete support #3270 (equality delete reads)
• Remove pyiceberg-core from the pyarrow extra #3356 (execution path isolation)
• Support data files compaction #1092 (data compaction)
• Delete orphan files #1200 (orphan file deletion)
• Add support for Equality Deletes on DeleteFileIndex #3285 (DeleteFileIndex for equality deletes)
• Add commit retry with data conflict validation #3319 / Add commit retry and concurrency validation for writes #3320 (commit retry, prerequisite for safe compaction commits)
• Feature: Add metadata-only replace API to Table for REPLACE snapshot operations #3130 / feat: Add metadata-only replace API to Table for REPLACE snapshot operations #3131 (REPLACE API, prerequisite for compaction)
• V3 Tracking issue #1818 (V3 tracking, DV compaction)
• Support producing positional deletes #1808 (positional delete write support)
• feat: Add DeleteFileIndex to improve position delete lookup #2918 (DeleteFileIndex for positional deletes, merged foundation)

Our contributions (to iceberg-rust) (all closed due to direction pivot)

• iceberg-rust#2716 (EPIC: bounded-memory execution operations for pyiceberg-core)
• iceberg-rust#2717 (bounded-memory session helper)
• iceberg-rust#2718 (pyiceberg_core.execution module)

datafusion-python

• datafusion-python#1217 (FFI boundary stability)

[qzyu999]

I'm planning to start on the first sub-task (engine resolution module).

[kevinjqliu]

I'd love to introduce datafusion to pyiceberg, but I do have some concerns with the approach outlined above.

I think PyIceberg should continue to own the Iceberg semantics, and only use DataFusion as the execution engine for data-intensive operations. If we rely on iceberg-rust / pyiceberg-core for Iceberg semantics, then PyIceberg becomes limited by iceberg-rust’s feature set. That also effectively deprecate and bypass a lot of PyIceberg’s existing functionality. This was the original reason I stopped experimenting with the datafusion table provider

I think the cleaner boundary is: PyIceberg handles everything and defers execution to dataFusion, much like how pyarrow is used today

[kris-gaudel]

Hey, this sounds like a great plan! I would love to help on this issue