Skip to content

build: cut the gates job's run time - #1086

Merged
bpowers merged 2 commits into
mainfrom
build/gates-ignored-only
Oct 4, 2026
Merged

bpowers merged 2 commits into
mainfrom
build/gates-ignored-only

Conversation

@bpowers

@bpowers bpowers commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Why

The gates job is the longest job in CI: about 20 minutes on main (5.5 to 8 to compile, 10 to 13 to run on the last four runs), past the macOS jobs at about 15.

What

  • The budget property test draws 40 generated models a run, not 200. On a runner this one test was the run step: its cases run one after another on one thread, about 11.5 minutes, and for the last 6 of them it was the only test still running. Each run draws new shapes and replays every saved failing seed, so coverage accumulates across runs.
  • scripts/gates.sh with no arguments runs the ignored tests only (47 today), not the whole engine suite again. It still builds with the ext_data feature (the Excel data provider) and runs that feature's own tests by name (data_provider::, 43 tests, under a second): they are not ignored, and no other job compiles them. scripts/gates.sh <filter> is unchanged.

Evidence

Measured on this PR's own gates job:

Compile Run
main (last four runs) 5.5 to 8.3 min 7.3 to 12.7 min
ignored tests only 8.1 min 14.0 min
plus 40 cases 6.9 min 6.3 min

Running only the ignored tests saved nothing by itself: the already-run tests cost seconds once optimized. The property test was the cost. After the change the unit gates finish in 4.6 minutes and the integration gates in 1.7.

What this does not establish

  • The compile step, now the larger half, is unchanged.
  • 40 cases a run finds a rare overrun later than 200 would. The two this gate found this week were each hit within a few dozen cases.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LYR7N9WZwQuFnZFW8JDUgx

The gates job reran the whole engine suite optimized, about 7,600 tests
the Build job had already run, on top of the ignored tests it exists
for. scripts/gates.sh with no arguments now runs the ignored tests
only. It still builds with ext_data on and runs that feature's own
tests by name: they are not ignored, and no other job compiles them, so
without that they would run nowhere. A filter still runs what it names,
ignored or not.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-04T06:57:13.203397Z 22c2ae0 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@claude

claude Bot commented Oct 4, 2026

Copy link
Copy Markdown

Review

No blocking issues found. The script correctly:

  • Preserves the single-invocation path for user-supplied filters (--include-ignored "$@").
  • Runs ignored tests across the engine, then the data_provider:: tests separately, so a #[ignore]d test anywhere in the crate (including hypothetical future ignored data_provider tests) still runs via the first pass.
  • Keeps the second run going even if the first fails (|| status=$?), and exits non-zero if either failed.
  • Compiles both passes under one --no-run build, since the targets and features are identical.

Verified that only data_provider/ contains #[cfg(feature = "ext_data")]-gated code (the single hit in json.rs:4175 is a user-facing string label, not test code), so the data_provider:: filter catches every ext_data-only test.

Overall correctness verdict

Correct.

@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.90%. Comparing base (e707b69) to head (5f0d7a7).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1086   +/-   ##
=======================================
  Coverage   92.90%   92.90%           
=======================================
  Files         313      313           
  Lines      194144   194144           
=======================================
+ Hits       180363   180369    +6     
+ Misses      13781    13775    -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

On a CI runner the gates job's run step is one test: the property test
that sends every tool through generated models runs its 200 cases one
after another on one thread, about eleven minutes, and for the last six
of them it is the only test still running. Forty cases is about two
minutes. Each run draws new shapes and replays every saved failing
seed, so coverage accumulates across runs.
@bpowers bpowers changed the title build: run only the ignored tests in the gates job build: cut the gates job's run time Oct 4, 2026
@claude

claude Bot commented Oct 4, 2026

Copy link
Copy Markdown

Review

Reviewed the trim to scripts/gates.sh and the proptest cap drop. No blocking issues found.

What I verified:

  • data_provider:: as a libtest substring filter matches all ext_data-sensitive tests. The three files that reference feature = "ext_data" are src/data_provider/csv_provider.rs, src/data_provider/mod.rs, and tests/integration/mdl_vensim_truth.rs; the first two are covered by the filter and the third's only cfg!(feature = "ext_data") branch is inside the #[ignore]d corpus gate (the_mdl_reader_matches_vensim_over_the_corpus), which still runs via --ignored. The non-ignored companion each_class_is_told_from_the_others only classifies four paths, none in READS_A_SPREADSHEET, so it is insensitive to the feature flag either way.
  • Exit-code handling: run --ignored || status=$? followed by run "${FEATURE_TESTS[@]}" || status=$? and exit "$status" correctly surfaces a non-zero exit if either run failed, without one run masking the other, matching the stated intent.
  • Filter branch (scripts/gates.sh <filter>) still passes --include-ignored so users can name an ignored test by name — behavior preserved.
  • Proptest 200 → 40: the ignored-reason string is updated to match ("over 40 generated models"), and the comment's claim that per-run coverage grows across runs via the saved failing seeds matches how proptest handles the regressions file.

Overall correctness verdict: correct. Existing behavior is preserved for the filter and --no-run paths, no coverage is dropped relative to what any other CI lane compiles, and the proptest cap change is intentional with a clear rationale.

@bpowers
bpowers merged commit c9a8636 into main Oct 4, 2026
19 checks passed
@bpowers
bpowers deleted the build/gates-ignored-only branch October 4, 2026 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant