Skip to content

node: concurrent router dials, ready on the first, redial a dropped router; pace router rollouts #613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aojea merged 5 commits into from
Oct 9, 2026
Merged

node: concurrent router dials, ready on the first, redial a dropped router; pace router rollouts #613

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7c8d723
node: dial the routers at once, be ready on the first, and redial one…
aojea Oct 9, 2026
6804eeb
deploy, chart: hold a router rollout 90s between pods
aojea Oct 9, 2026
da140a5
deploy: give the control plane and router rollouts the time they take
aojea Oct 9, 2026
68e8aa7
node: name the address behind which no router was found, log a redial…
aojea Oct 9, 2026
d612b64
go.mod: golang.org/x/net v0.60.0
aojea Oct 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/k8s/sam-router-template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,13 @@ metadata:
spec:
serviceName: "sam-router-${ENV_NAME}"
replicas: 3
# A router that is ready has not been re-handshaken by anyone yet: members
# redial a dropped router with a backoff that reaches a minute, and older
# members only on their monitor interval of one minute. A rollout that
# takes the next router sooner takes every session a member has before
# any is back. The first rollout under the soak fleet did exactly that in
# 45 seconds.
minReadySeconds: 90
selector:
matchLabels:
app: sam-router-${ENV_NAME}
Expand Down
9 changes: 7 additions & 2 deletions .github/workflows/deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -413,7 +413,10 @@ jobs:
echo "::warning::PodMonitoring CRD not found; managed collection is off on this cluster, metrics will not be scraped"
fi

kubectl rollout status deployment/sam-control-plane-${ENV_NAME} -n ${NAMESPACE} --timeout=120s || {
# The pod is Ready only once the load balancer's NEG controller has
# seen it healthy, which has taken anywhere from half a minute to
# several minutes; two replicas roll one after the other.
kubectl rollout status deployment/sam-control-plane-${ENV_NAME} -n ${NAMESPACE} --timeout=600s || {
echo "Control Plane Deployment failed!"
print_rollout_diagnostics "${NAMESPACE}" "deployment/sam-control-plane-${ENV_NAME}" "app=sam-control-plane-${ENV_NAME}"
exit 1
Expand Down Expand Up @@ -465,7 +468,9 @@ jobs:
exit 1
}

kubectl rollout status statefulset/sam-router-${ENV_NAME} -n ${NAMESPACE} --timeout=120s || {
# Three routers, each held 90s after it is ready (minReadySeconds)
# so members re-attach before the next one goes.
kubectl rollout status statefulset/sam-router-${ENV_NAME} -n ${NAMESPACE} --timeout=600s || {
echo "Router Deployment failed!"
print_rollout_diagnostics "${NAMESPACE}" "statefulset/sam-router-${ENV_NAME}" "app=sam-router-${ENV_NAME}"

Expand Down
1 change: 1 addition & 0 deletions charts/sam-mesh/templates/router-statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ metadata:
spec:
serviceName: {{ include "sam-mesh.fullname" . }}-router
replicas: {{ .Values.router.replicaCount }}
minReadySeconds: {{ .Values.router.minReadySeconds }}
selector:
matchLabels:
app: {{ include "sam-mesh.fullname" . }}-router
Expand Down
5 changes: 5 additions & 0 deletions charts/sam-mesh/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,11 @@ database:
router:
enabled: true
replicaCount: 1
# Seconds a new router must be ready before the rollout moves to the next
# one. Members redial a dropped router with a backoff that reaches a
# minute; a rollout faster than that takes every router a member has
# before any is back.
minReadySeconds: 90
podSecurityContext:
runAsNonRoot: true
# Dockerfiles say "USER nonroot" by name; runAsNonRoot needs the numeric uid.
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ require (
github.com/spf13/cobra v1.10.2
go.etcd.io/bbolt v1.5.0
go.uber.org/zap v1.28.0
golang.org/x/net v0.59.0
golang.org/x/net v0.60.0
golang.org/x/oauth2 v0.37.0
golang.org/x/sys v0.48.0
golang.org/x/time v0.16.0
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -361,8 +361,8 @@ golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues=
golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/oauth2 v0.37.0 h1:JUlcxA8oAtauLfiH8FX2/FkAWHAdi0QtGCGc+hofE98=
golang.org/x/oauth2 v0.37.0/go.mod h1:IxwZNxUULJmpBFf9K/9NTMSIfZZuvuTy1gGxhigP/58=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
Expand Down
Loading
Loading