docs: document DELETE and UPDATE for SQL users and table provider authors - #24567
docs: document DELETE and UPDATE for SQL users and table provider authors#24567michaelsembwever wants to merge 5 commits into
Conversation
…hors PR apache#19142 added `TableProvider::delete_from()` and `TableProvider::update()`, and implemented both for `MemTable`, but added no documentation. Add a `DELETE` section and an `UPDATE` section to the SQL user guide, with the syntax, the result shape, which table kinds support the statements, and the current limitations. Add a "Row-Level DML" section to the custom table provider guide, covering what the planner passes to each hook, the `count` result contract, the semantic rules a provider must follow, and a compiling example. Two behaviours found while verifying the documentation are recorded as warnings, since users meet them today: - An `IN` or an `EXISTS` subquery in the `WHERE` clause makes the statement apply to all rows, because the optimizer rewrites the subquery into a join and the predicate never reaches the provider. - `EXPLAIN DELETE` and `EXPLAIN UPDATE` execute the statement on an in-memory table, because `MemTable` changes the rows inside the hook and the hook runs during physical planning. Assisted-by: Claude Code:claude-opus-5
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24567 +/- ##
==========================================
+ Coverage 81.36% 81.71% +0.34%
==========================================
Files 1117 1127 +10
Lines 397872 416072 +18200
Branches 397872 416072 +18200
==========================================
+ Hits 323725 339984 +16259
- Misses 55229 56099 +870
- Partials 18918 19989 +1071 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Are there tickets that cover these issues? I agree they sound serious |
|
Specifically, I want to make sure the issues are tracked (ideally with a link in the docs as well) so that as we resolve them we also know to come and update the docs |
@alamb , i've created them
with PRs attached to each. |
|
@alamb , are we good for merging this now ? |
martin-g
left a comment
There was a problem hiding this comment.
I think this PR adds useful information about some current limitations in DataFusion.
It would be nice if every limitation is accompanied with text similar to "This limitation is tracked at issue XYZ"
Once the limitation is implemented it would be easier to detect that the documentation is obsolete and be updated too.
There are opened PRs for some of the limitations already.
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
…ed limitation docs: document DELETE and UPDATE for SQL users and table provider authors PR apache#19142 added `TableProvider::delete_from()` and `TableProvider::update()`, and implemented both for `MemTable`, but added no documentation. Add a `DELETE` section and an `UPDATE` section to the SQL user guide, with the syntax, the result shape, which table kinds support the statements, and the current limitations. Add a "Row-Level DML" section to the custom table provider guide, covering what the planner passes to each hook, the `count` result contract, the semantic rules a provider must follow, and a compiling example. Two behaviours found while verifying the documentation are recorded as warnings, since users meet them today: - An `IN` or an `EXISTS` subquery in the `WHERE` clause makes the statement apply to all rows, because the optimizer rewrites the subquery into a join and the predicate never reaches the provider. - `EXPLAIN DELETE` and `EXPLAIN UPDATE` execute the statement on an in-memory table, because `MemTable` changes the rows inside the hook and the hook runs during physical planning. Every documented limitation ends with the issue that tracks it, in one phrasing a contributor can grep for, so a merged fix makes the obsolete paragraph easy to find: apache#24654 for the subquery cases, apache#24656 for `EXPLAIN`, apache#24998 for the ignored `LIMIT` on a `DELETE`, and apache#19950 for `UPDATE ... FROM`. Which tables support the statements is a capability rather than a defect, so those two lines cite nothing. Assisted-by: Claude Code:claude-opus-5
Every limitation now ends with "This limitation is tracked at issue NNNNN": #24654 for the subquery cases, #24656 for EXPLAIN, #24998 for the ignored LIMIT, and #19950 for UPDATE ... FROM. Three of the four have an open fix (#24657, #24655, #25005), so whichever of those merges after this PR should drop the matching paragraph. |
Which issue does this PR
closerelate to?TableProviderrow-level DML hooks and closed Supportdelete_fromandupdateinTableProvider#16959. That PR shipped no documentation.UPDATE ...FROMbug #19950.Rationale for this change
Since 52.0.0, DataFusion runs
DELETEandUPDATEagainst a table whose provider implementsTableProvider::delete_from()orTableProvider::update(), and the built-in in-memory table implements both. No page in the documentation says so.A SQL user therefore cannot learn which tables accept the two statements, what a statement returns, or which forms fail. A provider author cannot learn what the planner passes to each hook, or what the hook must return.
Two current behaviours are surprising enough to warn about in the same pass:
A
DELETEor anUPDATEwhoseWHEREclause holds anINor anEXISTSsubquery applies to all rows of the table. The optimizer rewrites the subquery into aLeftSemi Join, soextract_dml_filters()finds no predicate on the target table, and the provider reads theempty filter list as "no
WHEREclause".EXPLAIN DELETEandEXPLAIN UPDATEexecute the statement on an in-memory table.MemTablechanges the rows inside the hook, and the physical planner calls the hook while it builds the plan.Both behaviours need code fixes, which this PR does not attempt. Until then a reader needs the warning.
What changes are included in this PR?
docs/source/user-guide/sql/dml.md:DELETEsection and anUPDATEsection: syntax, thecountresult, three-valued logic, and examples.LIMITonDELETE, andUPDATE ... FROM.docs/source/library-user-guide/custom-table-providers.md:ANDconjunctions, stripped table qualifiers, target-table predicates only), the single-rowcountreturn contract, the two semantic rules a provider must follow, a compiling example,the clauses a hook never receives, and when the work happens.
No code changes.
Are these changes tested?
Yes.
cargo test --doc -p datafusion library_user_guide_custom_table_providerspasses. The new example is a compiled doctest, not anignoreblock../ci/scripts/doc_prettier_check.shpasses.mainwith temporary sqllogictest cases, rather than read from the code alone: the ignoredLIMIT; the pre-statement values inSET a = b, b = a; the error text for an external table and for a view; the scalarsubquery error; the
INandEXISTSall-rows result; and theEXPLAINside effect. Those cases are not part of this PR, because the last two assert behaviour that should change.Are there any user-facing changes?
Documentation only. No change to any API.