Repository navigation
Fix services ending when replicas exit with 0 - #4482
Merged
Merged
Conversation
A service whose replicas exited with 0 either had them resubmitted by scaling or ended with ALL_JOBS_DONE, depending on whether all of them finished in the same tick. With `stop_criteria: master-done`, any replica reaching DONE could end the whole service, but only if the run pipeline saw the job after it left TERMINATING. - Services never end with ALL_JOBS_DONE. Replicas that exit with 0 are brought back by scaling, and an all-DONE tick moves the run to PENDING, so it is resubmitted with a backoff. - `master-done` applies within a replica: once the master job is DONE or finishing with DONE_BY_RUNNER, the replica counts as DONE. It has no effect on single-job service replicas. A worker failing in the same tick the master finishes no longer fails the run. Fixes: #4479 Fixes: #4481 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A service whose replicas exited with 0 either had them resubmitted by scaling or ended with ALL_JOBS_DONE, depending on whether all of them finished in the same tick. With
stop_criteria: master-done, any replica reaching DONE could end the whole service, but only if the run pipeline saw the job after it left TERMINATING.master-doneapplies within a replica: once the master job is DONE or finishing with DONE_BY_RUNNER, the replica counts as DONE. It has no effect on single-job service replicas. A worker failing in the same tick the master finishes no longer fails the run.Fixes: #4479
Fixes: #4481