Skip to content

fix: preserve overflow behavior and exhaustively destructure expression proto hooks - #25034

Open
peterxcli wants to merge 1 commit into
apache:mainfrom
peterxcli:fix/physical-expr-proto-overflow
Open

fix: preserve overflow behavior and exhaustively destructure expression proto hooks#25034
peterxcli wants to merge 1 commit into
apache:mainfrom
peterxcli:fix/physical-expr-proto-overflow

Conversation

@peterxcli

@peterxcli peterxcli commented Sep 7, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #24614.

Rationale for this change

This PR fixes serialization losing the setting that makes arithmetic fail on overflow. For example, checked Int32::MAX + 1 raises an error before serialization but returns Int32::MIN after decoding. Preserving this setting ensures that sending an expression through protobuf preserves its arithmetic behavior.

What changes are included in this PR?

The protobuf message now stores BinaryExpr::fail_on_overflow, and both supported decoding formats restore it. When the encoder combines nested expressions into a flat list of operands, it requires their operators and overflow settings to match. This preserves the behavior of expressions that mix checked and wrapping arithmetic. The generated Rust and JSON bindings include the new field.

All six encoding and decoding hooks for BinaryExpr, LikeExpr, and SqlSimilarToPattern explicitly list every field without a rest pattern. Adding a field to an expression or its protobuf payload will cause a compile error until the corresponding hook handles it.

The PR also changes the PostgreSQL SQLLogicTest decimal formatter to borrow its argument, resolving an existing Clippy error that blocked the required checks before committing.

What is the testing strategy for this PR?

The new tests serialize and decode nested additions, then check their evaluated results for all four combinations of checked and wrapping arithmetic. The regression test failed before the fix because an expression that should raise an overflow error returned Int32(-2147483648). Additional tests cover older messages that omit the new field and verify that JSON preserves the overflow setting.

All 17 focused expression tests passed. The extended workspace run passed 11,260 Rust tests, with 8 ignored, and completed all 511 SQLLogicTest files. Both conversion tests passed with the PostgreSQL feature enabled. Formatting, Clippy with all targets and features, and the complete ./dev/rust_lint.sh suite also passed.

Are there any user-facing changes?

Expressions configured to fail on arithmetic overflow now raise the expected error after serialization and decoding, including nested expressions with different overflow settings.

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) proto Related to proto crate labels Sep 7, 2026
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-physical-expr v55.0.0 (current)
       Built [  22.335s] (current)
     Parsing datafusion-physical-expr v55.0.0 (current)
      Parsed [   0.033s] (current)
    Building datafusion-physical-expr v55.0.0 (baseline)
       Built [  20.562s] (baseline)
     Parsing datafusion-physical-expr v55.0.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion-physical-expr v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.242s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  44.834s] datafusion-physical-expr
    Building datafusion-proto v55.0.0 (current)
       Built [  35.409s] (current)
     Parsing datafusion-proto v55.0.0 (current)
      Parsed [   0.013s] (current)
    Building datafusion-proto v55.0.0 (baseline)
       Built [  35.348s] (baseline)
     Parsing datafusion-proto v55.0.0 (baseline)
      Parsed [   0.013s] (baseline)
    Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.078s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  72.184s] datafusion-proto
    Building datafusion-proto-models v55.0.0 (current)
       Built [  16.135s] (current)
     Parsing datafusion-proto-models v55.0.0 (current)
      Parsed [   0.087s] (current)
    Building datafusion-proto-models v55.0.0 (baseline)
       Built [  15.668s] (baseline)
     Parsing datafusion-proto-models v55.0.0 (baseline)
      Parsed [   0.090s] (baseline)
    Checking datafusion-proto-models v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   1.112s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field PhysicalBinaryExprNode.fail_on_overflow in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1789
  field PhysicalBinaryExprNode.fail_on_overflow in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1789

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  34.127s] datafusion-proto-models
    Building datafusion-sqllogictest v55.0.0 (current)
       Built [  64.170s] (current)
     Parsing datafusion-sqllogictest v55.0.0 (current)
      Parsed [   0.016s] (current)
    Building datafusion-sqllogictest v55.0.0 (baseline)
       Built [  63.894s] (baseline)
     Parsing datafusion-sqllogictest v55.0.0 (baseline)
      Parsed [   0.017s] (baseline)
    Checking datafusion-sqllogictest v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.060s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 130.774s] datafusion-sqllogictest

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Sep 7, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.76471% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.74%. Comparing base (6ab4ce6) to head (6061043).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-expr/src/expressions/binary.rs 84.84% 1 Missing and 4 partials ⚠️
datafusion/proto-models/src/generated/pbjson.rs 69.23% 1 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25034      +/-   ##
==========================================
+ Coverage   81.72%   81.74%   +0.02%     
==========================================
  Files        1127     1127              
  Lines      416237   416271      +34     
  Branches   416237   416271      +34     
==========================================
+ Hits       340156   340276     +120     
+ Misses      56092    55967     -125     
- Partials    19989    20028      +39     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@peterxcli
peterxcli marked this pull request as draft September 7, 2026 22:53
@peterxcli peterxcli changed the title fix: preserve binary expression overflow policy in protobuf fix: preserve overflow behavior and exhaustively destructure expression proto hooks Sep 8, 2026
@peterxcli
peterxcli marked this pull request as ready for review September 8, 2026 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change physical-expr Changes to the physical-expr crates proto Related to proto crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Destructure proto hooks for binary and pattern physical expressions

2 participants