Skip to content

fix(benchmarks): validate predicate_eval results - #25037

Merged
adriangb merged 1 commit into
apache:mainfrom
pydantic:fix-predicate-eval-result-directive
Sep 8, 2026
Merged

fix(benchmarks): validate predicate_eval results#25037
adriangb merged 1 commit into
apache:mainfrom
pydantic:fix-predicate-eval-result-directive

Conversation

@adriangb

@adriangb adriangb commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

N/A - no issue; this is a standalone fix to the benchmark harness.

Rationale for this change

The predicate_eval SQL benchmark template
(benchmarks/sql_benchmarks/predicate_eval/predicate_eval.benchmark.template)
had no result directive. Every other suite that supports
--result-mode validate has one (e.g.
benchmarks/sql_benchmarks/clickbench/benchmarks/q00.benchmark,
benchmarks/sql_benchmarks/h2o/window_sorted.benchmark.template), but
predicate_eval was missing it entirely. As a result,
benchmark_runner predicate_eval --result-mode validate silently verified
nothing: SqlBenchmark::verify_results returns early when there are no
result_queries, so the run reports success without ever comparing query
output against a persisted baseline.

What changes are included in this PR?

Add a result directive to the predicate_eval template, after the run
line, pointing at a per-query CSV keyed by the benchmark's ${NAME}
parameter (each qNN.benchmark in the suite sets a unique NAME), mirroring
the convention used by the clickbench and h2o suites:

result sql_benchmarks/predicate_eval/results/${NAME}.csv

No expected-result CSVs are committed — like the other suites, these are
generated locally with --result-mode persist (sql_benchmarks/*/results/
is excluded from .gitignore's general results rule but no CSVs under it
are tracked in the repo today).

What is the testing strategy for this PR?

Built benchmark_runner in release mode and ran the predicate_eval
costsel subgroup with PRED_ROWS=100000 to keep it fast:

  1. --result-mode persist — writes one CSV per query under
    sql_benchmarks/predicate_eval/results/, e.g.
    costsel_q01_regexp_selective_last.csv containing count(*) / 51.
  2. --result-mode validate — passes cleanly against the persisted CSVs.
  3. Negative control: hand-edited one persisted CSV's value from 51 to
    999999 and re-ran --result-mode validate. It now fails loudly:
    Error: Execution error: Error in result on row 1, column 1 running query "":
    expected value "999999" but got value "51" in row: ["51"]
    
    (exit code 1). Restored the file afterward.
  4. Before-the-fix demonstration: reverted just the template change (via
    git stash) with no result CSVs present on disk, and reran
    --result-mode validate. It exited 0 and printed normal benchmark timing
    output with no validation performed at all — confirming the bug this PR
    fixes.

Are there any user-facing changes?

No public API changes. This only affects the benchmark harness
(benchmarks/), enabling predicate_eval to actually validate query
results when run with --result-mode validate, matching every other SQL
benchmark suite.

🤖 Generated with Claude Code

The predicate_eval benchmark template had no `result` directive, so
`benchmark_runner predicate_eval --result-mode validate` silently
verified nothing: it reported success without comparing any query
output. Add a `result` directive pointing at a per-query CSV keyed by
`${NAME}`, mirroring the convention used by clickbench and h2o.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@adriangb
adriangb requested a review from kosiew September 7, 2026 17:29
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.72%. Comparing base (262936e) to head (f13b780).
⚠️ Report is 20 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25037      +/-   ##
==========================================
+ Coverage   81.67%   81.72%   +0.04%     
==========================================
  Files        1126     1127       +1     
  Lines      414842   416273    +1431     
  Branches   414842   416273    +1431     
==========================================
+ Hits       338841   340196    +1355     
- Misses      56070    56087      +17     
- Partials    19931    19990      +59     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adriangb

@adriangb
adriangb added this pull request to the merge queue Sep 8, 2026
Merged via the queue into apache:main with commit 8f40f33 Sep 8, 2026
39 checks passed
@adriangb
adriangb deleted the fix-predicate-eval-result-directive branch September 8, 2026 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants