router: graceful shutdown through the lease ttl, relay budget sized to the mesh, capacity metrics; node --reachability by aojea · Pull Request #615 · google/agentmesh

aojea · 2026-10-09T18:42:22Z

Stacked follow-up to #612 and #613, from the first 500-member ladder step on bananas. One commit per feature.

router: announce the time away on shutdown through the lease ttl

A router that stopped stayed listed on /info until its lease expired, up to 15 minutes, so joiners were handed a dead address first. RouterLeaseRequest gains an optional ttl: a router may shorten its lease, never lengthen it, and the control plane lists it for exactly that long (zero withdraws it with that request). On SIGTERM the router sends one last lease with --shutdown-lease-ttl, the time it expects to be away, the way BGP graceful shutdown announces a maintenance window. Default 30s, about a pod restarting in place; 0 sends nothing. A renewal racing the shutdown sends nothing. sam-one --router-shutdown-lease-ttl.

Tests: TestRouterLeaseTTL (shorten honoured, lengthen capped, zero leaves the active set, negative is 400), TestRouterCloseWithdrawsLease, integration TestRouterGracefulShutdownWithdrawsLease (three routers; the one with ttl 1s is gone within the poll, the one with 0 stays).

router: size the relay budget to the mesh

libp2p's relay defaults assume a public DHT: 128 reservations, 32 per ASN, 8 per address, 16 circuits per peer. On the testnet the 33rd member in one region was unreachable and a service called by hundreds of members at once ran out of circuits (RESOURCE_LIMIT_EXCEEDED). The budget now follows the connection budget: --relay-max-reservations = --high-watermark, per address = --conns-per-source-ip, no per-ASN cap, --relay-max-circuits = 1024.

router: export capacity, draining and the relay budget on /metrics

sam_router_draining, sam_router_connections{direction}, sam_router_connection_watermark{level}, sam_router_relay_limit{limit}, and the relay service's own libp2p_relaysvc_* counters. Free capacity is inbound connections against the high watermark; the relay's is open reservations against the limit. Both are inputs for an autoscaler; the reference page has the table.

node: `--reachability private|auto`

Private stays the default (always hold a relay reservation; works behind any NAT). auto lets AutoNAT v2 decide from the routers' dial-backs, so a node with a public address is dialled directly. Auto is only right where the routers are on the public side of every NAT the node is behind, which is why it is opt-in.

scale: member ladder

tests/scale/member-ladder.sh climbs a testnet in steps of GCE minions (about 500 resident members each) and reads the control plane and every router per pod before and after each step; member-ladder-summary.py merges the per-member distributions.

Validation

make lint 0 issues, verify-generated.sh and verify-sdk-generated.sh pass, go test ./... green, every commit builds and vets alone.

A router that stopped stayed listed on /info until its lease expired, up to 15 minutes, so joiners were handed a dead address first. The lease request now carries an optional ttl: a router may shorten its lease, never lengthen it, and the control plane lists it for exactly that long. On SIGTERM the router sends one last lease with --shutdown-lease-ttl (default 30s, about the time a pod takes to come back in place; 0 sends nothing), the way BGP graceful shutdown announces a maintenance window, and a renewal racing the shutdown sends nothing. sam-one exposes it as --router-shutdown-lease-ttl.

libp2p's relay defaults assume a public DHT: 128 reservations in all, 32 per autonomous system, 8 per address and 16 circuits per peer. A reservation is what makes a member reachable through the router, so on the testnet the 33rd member in one cloud region was unreachable, and a service called by hundreds of members at once ran out of circuits (RESOURCE_LIMIT_EXCEEDED) after 16. The budget now follows the connection budget: --relay-max-reservations defaults to --high-watermark, the per-address cap is --conns-per-source-ip, the per-ASN cap is not applied, and --relay-max-circuits defaults to 1024.

Telling whether a router fleet is full, or which router is on its way out, needed the logs. The router now exports its open connections by direction, the watermarks and relay limits in force, and a draining gauge set from the moment it starts stopping, and wires the relay service's own counters (reservations and circuits opened and closed, requests by status, bytes) into the same registry. Free capacity is the gap between inbound connections and the high watermark; open reservations against the reservation limit is the relay's. Both are inputs for an autoscaler.

Every node forced its reachability to private, so it always held a relay reservation and was dialled through a router even when it had a public address: every call between two members in a cloud region went through the relay. The new --reachability keeps private as the default, which works behind any NAT, and adds auto, which lets AutoNAT v2 decide from the routers' dial-backs: a node they reach on a public address advertises it and holds no reservation. Auto is only right where the routers sit on the public side of every NAT the node is behind; a router in the same cluster can reach a node the rest of the mesh cannot, which is why private stays the default.

member-journey.sh asks what the Nth member experiences from one host. The ladder climbs: GCE minions join the mesh in steps, every fleet stays resident, and each step lands on a mesh already carrying the ones before it. One minion carries one address and, by the router's per-address cap, about 500 members, so the ladder is in minions. The control plane and every router are read per pod before and after each step; the summary merges the per-member distributions across minions and prints the deltas and the hold table.

gemini-code-assist

Code Review

This pull request introduces a graceful shutdown mechanism for routers, allowing them to advertise a temporary TTL to the control plane when stopping, and adds a reachability configuration option for nodes. It also introduces scale testing scripts (member-ladder.sh and member-ladder-summary.py) and integration tests to verify these changes. The review feedback focuses on improving the robustness of the scale testing scripts, specifically by ensuring proper directory permissions on minion VMs, dynamically setting file descriptor limits to avoid permission errors, and using Python best practices (such as with statements and explicit UTF-8 encoding) when opening files.

gemini-code-assist · 2026-10-09T18:44:24Z

+for m in "${ALL_MINIONS[@]}"; do
+  rssh "$m" "test -x /usr/local/bin/sam-bench && test -x /usr/local/bin/sam-node" || fail "$m has no binaries yet (startup script still running?)"
+  rscp_to "$OUT/bootstrap-token" "$m" /tmp/bootstrap-token
+  rssh "$m" "sudo install -m 0644 /tmp/bootstrap-token ${REMOTE_TOKEN_PATH} && rm -f /tmp/bootstrap-token" || fail "could not place the token on $m"


The /var/lib/ladder and /var/log/ladder directories might not exist on a clean minion VM, or if created by sudo install they will be owned by root, which prevents the non-root sam-bench process from writing to them. Creating these directories beforehand and changing their ownership to the current user avoids permission issues and allows running install without sudo.

Suggested change

rssh "$m" "sudo install -m 0644 /tmp/bootstrap-token ${REMOTE_TOKEN_PATH} && rm -f /tmp/bootstrap-token" || fail "could not place the token on $m"

rssh "$m" "sudo mkdir -p /var/lib/ladder /var/log/ladder && sudo chown -R \$(whoami) /var/lib/ladder /var/log/ladder && install -m 0644 /tmp/bootstrap-token ${REMOTE_TOKEN_PATH} && rm -f /tmp/bootstrap-token" || fail "could not place the token on $m"

gemini-code-assist · 2026-10-09T18:44:25Z

+  event "step${step_no}-start"
+  for m in "${minions[@]}"; do
+    # The run detaches from the ssh session; its stderr is the progress log.
+    rssh "$m" "ulimit -n 1048576; rm -rf /var/lib/ladder/${m} /var/log/ladder/${m}.json /var/log/ladder/resident; nohup sam-bench join --node-bin /usr/local/bin/sam-node --control-plane ${CONTROL_PLANE} --bootstrap-token-path ${REMOTE_TOKEN_PATH} --service ${SERVICE} --count ${PER_MINION} --ramp ${RAMP} --hold ${HOLD} --dir /var/lib/ladder/${m} --label minion=${m} --label step=${step_no} --label resident_before=${resident} --resident-marker /var/log/ladder/resident --out /var/log/ladder/${m}.json > /var/log/ladder/${m}.log 2>&1 < /dev/null &" || fail "could not start the run on $m"


Setting a hardcoded ulimit -n 1048576 can fail with 'Operation not permitted' if the system's hard limit is lower than that value, which would prevent the bench run from starting. Setting the limit to the maximum allowed hard limit using ulimit -n $(ulimit -Hn) is much more robust.

Suggested change

rssh "$m" "ulimit -n 1048576; rm -rf /var/lib/ladder/${m} /var/log/ladder/${m}.json /var/log/ladder/resident; nohup sam-bench join --node-bin /usr/local/bin/sam-node --control-plane ${CONTROL_PLANE} --bootstrap-token-path ${REMOTE_TOKEN_PATH} --service ${SERVICE} --count ${PER_MINION} --ramp ${RAMP} --hold ${HOLD} --dir /var/lib/ladder/${m} --label minion=${m} --label step=${step_no} --label resident_before=${resident} --resident-marker /var/log/ladder/resident --out /var/log/ladder/${m}.json > /var/log/ladder/${m}.log 2>&1 < /dev/null &" || fail "could not start the run on $m"

rssh "$m" "ulimit -n \$(ulimit -Hn) 2>/dev/null || true; rm -rf /var/lib/ladder/${m} /var/log/ladder/${m}.json /var/log/ladder/resident; nohup sam-bench join --node-bin /usr/local/bin/sam-node --control-plane ${CONTROL_PLANE} --bootstrap-token-path ${REMOTE_TOKEN_PATH} --service ${SERVICE} --count ${PER_MINION} --ramp ${RAMP} --hold ${HOLD} --dir /var/lib/ladder/${m} --label minion=${m} --label step=${step_no} --label resident_before=${resident} --resident-marker /var/log/ladder/resident --out /var/log/ladder/${m}.json > /var/log/ladder/${m}.log 2>&1 < /dev/null &" || fail "could not start the run on $m"

gemini-code-assist · 2026-10-09T18:44:25Z

+        for raw in open(path):
+            m = line.match(raw)
+            if not m:
+                continue
+            labels = tuple(sorted(re.findall(r'(\w+)="((?:[^"\\]|\\.)*)"', m.group(2) or ""))) + (("pod", pod),)
+            try:
+                series[(m.group(1), labels)] = float(m.group(3))
+            except ValueError:
+                pass


It is recommended to use a with statement when opening files to ensure they are properly closed, and to explicitly specify the encoding (e.g., utf-8) to prevent platform-dependent decoding issues.

Suggested change

for raw in open(path):

m = line.match(raw)

if not m:

continue

labels = tuple(sorted(re.findall(r'(\w+)="((?:[^"\\]|\\.)*)"', m.group(2) or ""))) + (("pod", pod),)

try:

series[(m.group(1), labels)] = float(m.group(3))

except ValueError:

pass

with open(path, encoding="utf-8") as f:

for raw in f:

m = line.match(raw)

if not m:

continue

labels = tuple(sorted(re.findall(r'(\w+)="((?:[^"\\]|\\.)*)"', m.group(2) or ""))) + (("pod", pod),)

try:

series[(m.group(1), labels)] = float(m.group(3))

except ValueError:

pass

gemini-code-assist · 2026-10-09T18:44:25Z

+    for m in minions:
+        path = os.path.join(out, f"step{i}-{m}.json")
+        if os.path.exists(path):
+            reports.append(json.load(open(path))["join"])


It is recommended to use a with statement when opening files to ensure they are properly closed, and to explicitly specify the encoding (e.g., utf-8) to prevent platform-dependent decoding issues.

Suggested change

reports.append(json.load(open(path))["join"])

with open(path, encoding="utf-8") as f:

reports.append(json.load(f)["join"])

gemini-code-assist · 2026-10-09T18:44:25Z

+# Hold reports, when the fleets were interrupted.
+holds = []
+for path in sorted(glob.glob(os.path.join(out, "sam-ladder-*.json"))):
+    j = json.load(open(path))["join"]


It is recommended to use a with statement when opening files to ensure they are properly closed, and to explicitly specify the encoding (e.g., utf-8) to prevent platform-dependent decoding issues.

Suggested change

j = json.load(open(path))["join"]

with open(path, encoding="utf-8") as f:

j = json.load(f)["join"]

aojea added 5 commits October 9, 2026 18:40

gemini-code-assist Bot reviewed Oct 9, 2026

View reviewed changes

aojea merged commit 9b7652d into google:main Oct 9, 2026
22 checks passed

This was referenced Oct 10, 2026

Ladder findings: retry enrollment and admission, size the router as a hub, 502 for an unreachable provider, no CPU limits #638

Merged

Feature Request: Hierarchical Network Topology #193

Closed

Scalability testing #277

Open

Navigation Menu

Repository navigation

router: graceful shutdown through the lease ttl, relay budget sized to the mesh, capacity metrics; node --reachability - #615

router: graceful shutdown through the lease ttl, relay budget sized to the mesh, capacity metrics; node --reachability#615
aojea merged 5 commits into
google:mainfrom
aojea:router-graceful

aojea commented Oct 9, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Oct 9, 2026

Uh oh!

gemini-code-assist Bot Oct 9, 2026

Uh oh!

gemini-code-assist Bot Oct 9, 2026

Uh oh!

gemini-code-assist Bot Oct 9, 2026

Uh oh!

gemini-code-assist Bot Oct 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	rssh "$m" "sudo install -m 0644 /tmp/bootstrap-token ${REMOTE_TOKEN_PATH} && rm -f /tmp/bootstrap-token" \|\| fail "could not place the token on $m"
	rssh "$m" "sudo mkdir -p /var/lib/ladder /var/log/ladder && sudo chown -R \$(whoami) /var/lib/ladder /var/log/ladder && install -m 0644 /tmp/bootstrap-token ${REMOTE_TOKEN_PATH} && rm -f /tmp/bootstrap-token" \|\| fail "could not place the token on $m"

	reports.append(json.load(open(path))["join"])
	with open(path, encoding="utf-8") as f:
	reports.append(json.load(f)["join"])

	j = json.load(open(path))["join"]
	with open(path, encoding="utf-8") as f:
	j = json.load(f)["join"]

Conversation

aojea commented Oct 9, 2026

router: announce the time away on shutdown through the lease ttl

router: size the relay budget to the mesh

router: export capacity, draining and the relay budget on /metrics

node: --reachability private|auto

scale: member ladder

Validation

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

node: `--reachability private|auto`