fix: correct operator precedence for IS [NOT] DISTINCT FROM

[adriangb]

Which issue does this PR close?

• Closes type_coercion error: multi-condition IS NOT DISTINCT FROM in JOIN ON clause fails #23692.

Rationale for this change

Combining two or more IS NOT DISTINCT FROM conditions with AND fails to plan:

SELECT 1 IS NOT DISTINCT FROM 1 AND true;

Error during planning: Cannot infer common argument type for logical boolean operation Int64 AND Boolean

A single condition works, and adding parentheses around each condition works, so the failure looks like a type coercion problem. It isn't. sqlparser parses the right operand of IS [NOT] DISTINCT FROM with parse_expr() — that is, at the lowest possible precedence — instead of stopping at the first operator that binds less tightly than IS. Everything after the operator is swallowed into its right operand:

l.a IS NOT DISTINCT FROM r.a AND l.b IS NOT DISTINCT FROM r.b

  => IsNotDistinctFrom(l.a, BinaryOp { r.a AND IsNotDistinctFrom(l.b, r.b) })

The Int64 AND Boolean in the error is that inner r.a AND <bool>. PostgreSQL binds AND less tightly than IS, so the expected parse is (l.a IS NOT DISTINCT FROM r.a) AND (l.b IS NOT DISTINCT FROM r.b).

This affects every column type and every join type, and it also affects IS DISTINCT FROM and plain WHERE clauses. It blocks multi-column equality delete resolution in Apache Iceberg.

The same greedy parse_expr() call is still present on datafusion-sqlparser-rs main (src/parser/mod.rs:4074), so the fix is applied on the DataFusion side.

What changes are included in this PR?

In datafusion/sql/src/expr/mod.rs, before planning an expression the planner now restores the expected associativity:

1. has_greedy_distinct_from cheaply detects whether the mis-parse is present. When it isn't — the overwhelmingly common case — nothing else runs.
2. flatten_and_or flattens the AND / OR spine into its first operand plus the remaining (operator, operand) pairs, re-attaching each IS [NOT] DISTINCT FROM to only the first operand of its right hand side.
3. rebuild_and_or rebuilds the expression with AND binding more tightly than OR, both left associative.

Prefix NOT is handled the same way, since it also binds more tightly than AND / OR. Handling OR and NOT is required for correctness rather than completeness: a purely local rotation gets a IS NOT DISTINCT FROM 1 AND b IS NOT DISTINCT FROM 2 OR c IS NOT DISTINCT FROM 3 wrong, producing A AND (B OR C) and turning a planning error into a silently wrong result.

Operands are not descended into, so a parenthesised sub-expression keeps its explicit grouping and is handled when the planner recurses into it. The rewrite's output is a fixed point — it never leaves an IS [NOT] DISTINCT FROM whose right operand is an AND / OR — so re-entry cannot loop. Both recursive helpers carry the crate's usual recursive_protection attribute.

Known limitation left in place

The postfix IS family at the same precedence level (a IS NOT DISTINCT FROM b IS NULL) is still associated as sqlparser produces it. That behaviour is unchanged by this PR and belongs with the broader precedence discussion in #22461.

Are these changes tested?

Yes.

New planner tests in datafusion/sql/tests/sql_integration.rs covering join ON clauses, WHERE clauses, projections, IS DISTINCT FROM, AND / OR chains, and parenthesised forms as a control.

New end-to-end cases in datafusion/sqllogictest/test_files/join_is_not_distinct_from.slt, including the LEFT ANTI JOIN from the issue, three-condition chains, and the AND / OR and NOT precedence cases. The precedence cases are chosen so that incorrect grouping returns a different set of rows, not just a different plan — the plan display alone does not distinguish NOT (A AND B) from (NOT A) AND B.

Verification run:

• cargo test -p datafusion-sql — 565 + 87 + 12 doctests pass
• full sqllogictest suite — 501/501 files pass
• cargo test -p datafusion-optimizer -p datafusion-expr — passes
• cargo clippy -p datafusion-sql -p datafusion-optimizer --all-targets -- -D warnings — clean
• cargo fmt --all applied

Are there any user-facing changes?

Yes, and they are the point of the PR: IS [NOT] DISTINCT FROM combined with AND / OR / NOT without parentheses now parses the way PostgreSQL parses it, so queries that previously failed to plan now succeed.

Queries that were already parenthesised are unaffected. No public API changes.

---

Generated by Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.23%. Comparing base (fc846dd) to head (672dfc1).
⚠️ Report is 160 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24479      +/-   ##
==========================================
+ Coverage   81.05%   81.23%   +0.18%     
==========================================
  Files        1107     1113       +6     
  Lines      381574   392649   +11075     
  Branches   381574   392649   +11075     
==========================================
+ Hits       309281   318987    +9706     
- Misses      54034    54905     +871     
- Partials    18259    18757     +498     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[adriangb]

Closing this in favour of fixing the root cause upstream.

The mis-parse originates in sqlparser, which parses the right operand of IS [NOT] DISTINCT FROM with parse_expr() — the lowest precedence — so everything after the operator is swallowed into it. The one-line fix is to use parse_subexpr(precedence) instead, and it is already proposed upstream:

• Honour operator precedence in IS [NOT] DISTINCT FROM datafusion-sqlparser-rs#2436 — includes the fix plus a related correction to the default Precedence::PgOther placement
• Parser: fix IS [NOT] DISTINCT FROM right-operand precedence datafusion-sqlparser-rs#2443 — the same parser change

Fixing it there removes the need for the 271-line planner-side rewrite in this PR, which would become dead code as soon as DataFusion picks up the release containing it.

I verified the upstream fix end-to-end against this branch's base (main @ 9a96f67) by applying it to sqlparser v0.62.0 via [patch.crates-io]:

• every case from type_coercion error: multi-condition IS NOT DISTINCT FROM in JOIN ON clause fails #23692 that previously failed with Cannot infer common argument type for logical boolean operation Int64 AND Boolean now plans and returns the correct result, including the multi-column LEFT ANTI JOIN
• on the AND/OR and NOT cases where an incorrect grouping changes the result rather than raising an error, DataFusion now agrees with PostgreSQL 17
• no regressions: datafusion-sql 88 + 572 + 12 passed; sqllogictest 502/502 files; datafusion-optimizer + datafusion-expr 248 + 760 + 26 + 55 + 5 passed

#23692 should stay open until DataFusion bumps its sqlparser dependency past the release carrying the fix. The .slt and planner tests added here are still worth having and can be re-proposed alongside that bump.