Skip to content

test: Make the mocked run status sequence independent of poll timing - #1028

Closed
vdusek wants to merge 1 commit into
masterfrom
test/deterministic-status-message-mock
Closed

test: Make the mocked run status sequence independent of poll timing#1028
vdusek wants to merge 1 commit into
masterfrom
test/deterministic-status-message-mock

Conversation

@vdusek

@vdusek vdusek commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

test_actor_call_redirect_logs_to_default_logger_sync and its three siblings flake in CI, dropping ('Status: RUNNING, Message: Initial message', 20) (example).

StatusResponseGenerator advanced the status every 5 requests, counted across every caller of the run endpoint — the two setup GETs, the watcher's 1-per-second polls, and wait_for_finish's 4-per-second polling loop. wait_for_finish burned the "Initial message" window in roughly half a second, so the watcher's first poll had a ~0.5s margin to land inside it.

Plain requests and waitForFinish requests are now answered separately: plain requests walk the status sequence, and waitForFinish reports the run finished only once they have walked all of it. The first status covers three plain requests, which is call's two setup GETs plus the watcher's own first poll in any order. With nothing watching there are no plain requests and the run reads as finished immediately, so call(logger=None) still returns at once.

Verified by injecting a delay before the watcher's first poll: on master the tests fail from 0.6s upward; with this change the two call()-based tests pass at 0.6s, 2s, 5s and 10s. (test_redirect_status_message_sync still fails past ~5s, but that is bounded by its own 1s sleep, not by the mock.)

✍️ Drafted by Claude Code

@vdusek vdusek added adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. labels Aug 21, 2026
@vdusek vdusek self-assigned this Aug 21, 2026
@github-actions github-actions Bot added this to the 147th sprint - Tooling team milestone Aug 21, 2026
@github-actions github-actions Bot added the tested Temporary label used only programatically for some analytics. label Aug 21, 2026
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.95%. Comparing base (df80da1) to head (45dd84c).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1028   +/-   ##
=======================================
  Coverage   94.95%   94.95%           
=======================================
  Files          58       58           
  Lines        5436     5436           
=======================================
  Hits         5162     5162           
  Misses        274      274           
Flag Coverage Δ
integration 91.72% <ø> (ø)
unit 85.81% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek

vdusek commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Closing - the flake this targeted was in the client, not the mock. In the CI run cited above, Status: RUNNING, Message: Initial message appears in captured stderr but not in caplog, so the watcher's poll did land inside the first-status window; the record was dropped by the logger rebuild that #1036 fixed.

The mock's timing coupling is real but latent, and the trade here isn't worth it: holding the run open until the plain sequence is walked means a stalled watcher hangs wait_for_finish indefinitely instead of failing fast, and _requests_for_first_status = 3 hardcodes how many setup GETs call() happens to make.

✍️ Drafted by Claude Code

@vdusek vdusek closed this Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants