Skip to content

docs: document DELETE and UPDATE for SQL users and table provider authors - #24567

Open
michaelsembwever wants to merge 5 commits into
apache:mainfrom
thelastpickle:mck/dml-delete-update-docs
Open

docs: document DELETE and UPDATE for SQL users and table provider authors#24567
michaelsembwever wants to merge 5 commits into
apache:mainfrom
thelastpickle:mck/dml-delete-update-docs

Conversation

@michaelsembwever

@michaelsembwever michaelsembwever commented Aug 21, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close relate to?

Rationale for this change

Since 52.0.0, DataFusion runs DELETE and UPDATE against a table whose provider implements TableProvider::delete_from() or TableProvider::update(), and the built-in in-memory table implements both. No page in the documentation says so.

A SQL user therefore cannot learn which tables accept the two statements, what a statement returns, or which forms fail. A provider author cannot learn what the planner passes to each hook, or what the hook must return.

Two current behaviours are surprising enough to warn about in the same pass:

  • A DELETE or an UPDATE whose WHERE clause holds an IN or an EXISTS subquery applies to all rows of the table. The optimizer rewrites the subquery into a LeftSemi Join, so extract_dml_filters() finds no predicate on the target table, and the provider reads the
    empty filter list as "no WHERE clause".

    > create table s1 as values (1), (2), (3);
    > create table s2 as values (2);
    > delete from s1 where column1 in (select column1 from s2);
    -- count 3; s1 is now empty
  • EXPLAIN DELETE and EXPLAIN UPDATE execute the statement on an in-memory table. MemTable changes the rows inside the hook, and the physical planner calls the hook while it builds the plan.

Both behaviours need code fixes, which this PR does not attempt. Until then a reader needs the warning.

What changes are included in this PR?

docs/source/user-guide/sql/dml.md:

  • A DELETE section and an UPDATE section: syntax, the count result, three-valued logic, and examples.
  • A "Table support for DELETE and UPDATE" section: which table kinds support the statements, and the exact error text for a table that does not.
  • A "Limitations" section: the two warnings above, the ignored LIMIT on DELETE, and UPDATE ... FROM.

docs/source/library-user-guide/custom-table-providers.md:

  • A "Row-Level DML: DELETE and UPDATE" section: what the planner passes to each hook (split AND conjunctions, stripped table qualifiers, target-table predicates only), the single-row count return contract, the two semantic rules a provider must follow, a compiling example,
    the clauses a hook never receives, and when the work happens.

No code changes.

Are these changes tested?

Yes.

  • cargo test --doc -p datafusion library_user_guide_custom_table_providers passes. The new example is a compiled doctest, not an ignore block.
  • ./ci/scripts/doc_prettier_check.sh passes.
  • Every behavioural statement in the new text was checked against main with temporary sqllogictest cases, rather than read from the code alone: the ignored LIMIT; the pre-statement values in SET a = b, b = a; the error text for an external table and for a view; the scalar
    subquery error; the IN and EXISTS all-rows result; and the EXPLAIN side effect. Those cases are not part of this PR, because the last two assert behaviour that should change.

Are there any user-facing changes?

Documentation only. No change to any API.

…hors

PR apache#19142 added `TableProvider::delete_from()` and `TableProvider::update()`,
and implemented both for `MemTable`, but added no documentation.

Add a `DELETE` section and an `UPDATE` section to the SQL user guide, with
the syntax, the result shape, which table kinds support the statements, and
the current limitations.

Add a "Row-Level DML" section to the custom table provider guide, covering
what the planner passes to each hook, the `count` result contract, the
semantic rules a provider must follow, and a compiling example.

Two behaviours found while verifying the documentation are recorded as
warnings, since users meet them today:

- An `IN` or an `EXISTS` subquery in the `WHERE` clause makes the statement
  apply to all rows, because the optimizer rewrites the subquery into a join
  and the predicate never reaches the provider.
- `EXPLAIN DELETE` and `EXPLAIN UPDATE` execute the statement on an
  in-memory table, because `MemTable` changes the rows inside the hook and
  the hook runs during physical planning.

Assisted-by: Claude Code:claude-opus-5
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 21, 2026
@codecov-commenter

codecov-commenter commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.71%. Comparing base (a6e2d3f) to head (989cb7c).
⚠️ Report is 200 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24567      +/-   ##
==========================================
+ Coverage   81.36%   81.71%   +0.34%     
==========================================
  Files        1117     1127      +10     
  Lines      397872   416072   +18200     
  Branches   397872   416072   +18200     
==========================================
+ Hits       323725   339984   +16259     
- Misses      55229    56099     +870     
- Partials    18918    19989    +1071     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@alamb

alamb commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Both behaviours need code fixes, which this PR does not attempt. Until then a reader needs the warning.

Are there tickets that cover these issues? I agree they sound serious

@alamb

alamb commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Specifically, I want to make sure the issues are tracked (ideally with a link in the docs as well) so that as we resolve them we also know to come and update the docs

@michaelsembwever

michaelsembwever commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@michaelsembwever

Copy link
Copy Markdown
Member Author

@alamb , are we good for merging this now ?

@martin-g martin-g left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this PR adds useful information about some current limitations in DataFusion.
It would be nice if every limitation is accompanied with text similar to "This limitation is tracked at issue XYZ"
Once the limitation is implemented it would be easier to detect that the documentation is obsolete and be updated too.
There are opened PRs for some of the limitations already.

Comment thread docs/source/user-guide/sql/dml.md Outdated
Comment thread docs/source/user-guide/sql/dml.md Outdated
Comment thread docs/source/user-guide/sql/dml.md Outdated
michaelsembwever and others added 4 commits September 7, 2026 14:23
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
…ed limitation

docs: document DELETE and UPDATE for SQL users and table provider authors

PR apache#19142 added `TableProvider::delete_from()` and `TableProvider::update()`, and implemented both for `MemTable`, but added no documentation.

Add a `DELETE` section and an `UPDATE` section to the SQL user guide, with the syntax, the result shape, which table kinds support the statements, and the current limitations.

Add a "Row-Level DML" section to the custom table provider guide, covering what the planner passes to each hook, the `count` result contract, the semantic rules a provider must follow, and a compiling example.

Two behaviours found while verifying the documentation are recorded as warnings, since users meet them today:

- An `IN` or an `EXISTS` subquery in the `WHERE` clause makes the statement apply to all rows, because the optimizer rewrites the subquery into a join and the predicate never reaches the provider.
- `EXPLAIN DELETE` and `EXPLAIN UPDATE` execute the statement on an in-memory table, because `MemTable` changes the rows inside the hook and the hook runs during physical planning.

Every documented limitation ends with the issue that tracks it, in one phrasing a contributor can grep for, so a merged fix makes the obsolete paragraph easy to find: apache#24654 for the subquery cases, apache#24656 for `EXPLAIN`, apache#24998 for the ignored `LIMIT` on a `DELETE`, and apache#19950 for `UPDATE ... FROM`. Which tables support the statements is a capability rather than a defect, so those two lines cite nothing.

Assisted-by: Claude Code:claude-opus-5
@michaelsembwever

Copy link
Copy Markdown
Member Author

ideally with a link in the docs as well

I think this PR adds useful information about some current limitations in DataFusion.
It would be nice if every limitation is accompanied with text similar to "This limitation is tracked at issue [XYZ (https://github.com/apache/datafusion/issues/XYZ)"
Once the limitation is implemented it would be easier to detect that the documentation is obsolete and be updated too.

Every limitation now ends with "This limitation is tracked at issue NNNNN": #24654 for the subquery cases, #24656 for EXPLAIN, #24998 for the ignored LIMIT, and #19950 for UPDATE ... FROM. Three of the four have an open fix (#24657, #24655, #25005), so whichever of those merges after this PR should drop the matching paragraph.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support delete_from and update in TableProvider

4 participants