fix: apply MemTable DELETE and UPDATE at execution, not planning

[michaelsembwever]

Which issue does this PR close?

• Closes EXPLAIN DELETE and EXPLAIN UPDATE execute the statement on an in-memory table #24656.

Rationale for this change

EXPLAIN DELETE and EXPLAIN UPDATE changed the rows of an in-memory table. The plan was printed, and the statement had also run:

> create table t as values (1), (2), (3);

> explain delete from t where column1 > 1;
+---------------+----------------------------------+
| plan_type     | plan                             |
+---------------+----------------------------------+
| logical_plan  | Dml: op=[Delete] table=[t]       |
|               |   Filter: t.column1 > Int64(1)   |
|               |     TableScan: t                 |
| physical_plan | CooperativeExec                  |
|               |   DmlResultExec: rows_affected=2 |
+---------------+----------------------------------+

> select * from t;
+---------+
| column1 |
+---------+
| 1       |
+---------+

Two causes combined. handle_explain() builds the physical plan in order to print the physical_plan section, and the planner awaits TableProvider::delete_from() and TableProvider::update() while it builds. MemTable did the whole row change inside those hooks: it took a write lock on each partition, overwrote it, cleared the declared sort order, and returned a constant DmlResultExec carrying a count it had already computed. The row count baked into the plan text was the tell.

EXPLAIN ANALYZE DELETE is expected to change the rows, because it runs the plan by design. It must apply the statement exactly once.

What changes are included in this PR?

Each hook is split into a planning half and an execution half.

Stays in the hook (planning)	Moves to execute()
Clone the batches and sort_order handles	Take the write lock on each partition
Build the physical predicates of the WHERE clause	Evaluate the mask per batch
Validate the SET column names, build the physical assignments	Rewrite or filter the batches
	Clear the declared sort order
	Count the affected rows and emit the count batch

delete_from() now returns MemDeleteExec and update() returns MemUpdateExec. Each node holds the shared partition handles plus the expressions the hook built, and applies the change on the first poll of the stream that execute() returns. DmlResultExec is removed, since the count is not known while the plan is built. Each run of the plan applies the statement once, as DataSinkExec does for an INSERT.

The row logic itself moved unchanged into delete_rows() and update_rows(), including the SQL three-valued logic for a NULL predicate and the evaluate_selection call that keeps an error such as a divide by zero away from the rows the statement does not touch.

Planning errors stay in the hooks, so EXPLAIN still reports an unknown SET column and a predicate that cannot be planned. apply_expressions() now visits the expressions the nodes hold; the constant node it replaces had none.

Both nodes are private to datafusion-catalog, so there is no public API change.

What is the testing strategy for this PR?

New sqllogictest cases in dml_delete.slt and dml_update.slt assert that EXPLAIN leaves the three rows alone and that EXPLAIN ANALYZE changes them. Against the unfixed provider both fail. The update case adds ten to each matching value rather than deleting, so a plan that ran twice would print 22 and 23 instead of 12 and 13; that is what pins the once-only guarantee, and it is what the old code produced.

The eight physical plan expectations in delete.slt and update.slt are regenerated. They can no longer carry rows_affected, and the new text names the predicates and the assignments instead:

02)--MemDeleteExec: predicate=[CAST(a@0 AS Int64) = 1, c@2 > CAST(3 AS Float64)]
02)--MemUpdateExec: set=[a=CAST(c@2 + CAST(1 AS Float64) AS Int32), b=CAST(a@0 AS Utf8View)]

cargo test --test sqllogictests, cargo test -p datafusion --lib, and cargo test -p datafusion --test core_integration show identical results before and after the change. Clippy passes with -D warnings on datafusion-catalog, datafusion, and datafusion-sqllogictest.

Are there any user-facing changes?

Yes, and all three are the point of the fix or follow from it.

EXPLAIN DELETE and EXPLAIN UPDATE on a MemTable no longer change data, and they no longer clear the table's declared sort order.

The physical plan text of a DELETE or an UPDATE on a MemTable changed. DmlResultExec: rows_affected=N becomes MemDeleteExec or MemUpdateExec, which name the predicates and the assignments in place of the count.

One behaviour change beyond the fix, noted for completeness: the early return for a MemTable with no partitions is gone, so an UPDATE naming an unknown column now raises its plan error rather than reporting zero rows. MemTable::try_new rejects an empty partition list, so this is unreachable through normal construction.

EXPLAIN INSERT still clears the declared sort order, because insert_into() does that inside the hook. That is a smaller instance of the same shape and is left for a follow-up.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 87.67361% with 71 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.72%. Comparing base (6ab4ce6) to head (dfdcd91).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/core/src/datasource/memory_test.rs	84.12%	7 Missing and 33 partials ⚠️
datafusion/catalog/src/memory/table.rs	90.43%	21 Missing and 10 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #25040    +/-   ##
========================================
  Coverage   81.72%   81.72%            
========================================
  Files        1127     1127            
  Lines      416237   416694   +457     
  Branches   416237   416694   +457     
========================================
+ Hits       340156   340561   +405     
- Misses      56092    56109    +17     
- Partials    19989    20024    +35     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[flyovers-bot]

Watch the flyover → · 12:29 · 6 chapters · 6 files

An animated walkthrough of this diff: context, the problem, the change, implementation care, code walkthrough, and impact.

_{dfdcd91 · updated automatically by flyovers}