Skip to content

deploy: wait for the router rollout as long as the routers take - #666

Merged
aojea merged 1 commit into
google:mainfrom
aojea:deploy-router-rollout-wait
Oct 11, 2026
Merged

aojea merged 1 commit into
google:mainfrom
aojea:deploy-router-rollout-wait

Conversation

@aojea

@aojea aojea commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

The router StatefulSet rolls one pod at a time, each held 90 s after it is ready (minReadySeconds) so members re-attach before the next one goes: about two minutes a router. The wait in the deploy step was 600 s, sized for three routers (the comment said so). At ROUTER_REPLICAS=6 the healthy rollout of 4c8124b4 (run 38159658330, attempt 2) outran it by about a minute: applied 18:35:49, router-1 ready 18:44:40, router-0 still to go; at 18:45:57 the step's fallback ran kubectl rollout undo and the environment went back to 9125f667, without the #663 watermark and resource-manager changes.

The wait now follows the count: ROUTER_REPLICAS * 180 + 120 seconds (20 minutes at six).

The router StatefulSet rolls one pod at a time, each held 90 s after it
is ready (minReadySeconds) so members re-attach before the next one goes:
about two minutes a router. The wait was 600 s, sized for three routers,
and at six the healthy rollout of 4c8124b outran it by a minute; the
fallback then undid it and the environment went back to the previous
image. The wait is now 180 s a router plus two minutes.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

@aojea
aojea merged commit 9d42945 into google:main Oct 11, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant