Skip to content

Fix services ending when replicas exit with 0 - #4482

Merged
un-def merged 1 commit into
masterfrom
issue_4479_service_done_replicas
Nov 12, 2026
Merged

un-def merged 1 commit into
masterfrom
issue_4479_service_done_replicas

Conversation

@un-def

@un-def un-def commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

A service whose replicas exited with 0 either had them resubmitted by scaling or ended with ALL_JOBS_DONE, depending on whether all of them finished in the same tick. With stop_criteria: master-done, any replica reaching DONE could end the whole service, but only if the run pipeline saw the job after it left TERMINATING.

  • Services never end with ALL_JOBS_DONE. Replicas that exit with 0 are brought back by scaling, and an all-DONE tick moves the run to PENDING, so it is resubmitted with a backoff.
  • master-done applies within a replica: once the master job is DONE or finishing with DONE_BY_RUNNER, the replica counts as DONE. It has no effect on single-job service replicas. A worker failing in the same tick the master finishes no longer fails the run.

Fixes: #4479
Fixes: #4481

A service whose replicas exited with 0 either had them resubmitted by
scaling or ended with ALL_JOBS_DONE, depending on whether all of them
finished in the same tick. With `stop_criteria: master-done`, any
replica reaching DONE could end the whole service, but only if the run
pipeline saw the job after it left TERMINATING.

- Services never end with ALL_JOBS_DONE. Replicas that exit with 0 are
  brought back by scaling, and an all-DONE tick moves the run to
  PENDING, so it is resubmitted with a backoff.
- `master-done` applies within a replica: once the master job is DONE
  or finishing with DONE_BY_RUNNER, the replica counts as DONE. It has
  no effect on single-job service replicas. A worker failing in the
  same tick the master finishes no longer fails the run.

Fixes: #4479
Fixes: #4481
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@un-def
un-def merged commit 5e8977f into master Nov 12, 2026
101 of 104 checks passed
@un-def
un-def deleted the issue_4479_service_done_replicas branch November 12, 2026 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant