Skip to content

Budget the DHT and gossip streams per member; find members through the router they reserved with - #669

Merged
aojea merged 3 commits into
google:mainfrom
aojea:router-protocol-limits
Oct 11, 2026
Merged

aojea merged 3 commits into
google:mainfrom
aojea:router-protocol-limits

Conversation

@aojea

@aojea aojea commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Three fixes, one per commit, for what the 10k ladder and the cold-path probe showed on bananas today. Each commit builds and tests alone.

1. router, node: budget the DHT and gossip streams per member, count refused streams by protocol

hubLimits named identify, the auth handshake and the relay protocols; /mesh/kad/1.0.0 and GossipSub fell to libp2p's default for an unnamed protocol, which scales from host memory and was 768 inbound streams on the testnet's routers. A member holds one DHT request stream and one gossip stream to its router for as long as it is connected, so past 768 members a router refused every further one. kad-dht runs a FIND_NODE check before adding a peer to a routing table (logged at debug level only); the refused streams failed it, the router never entered those members' tables and fell out of the ones it was in, and the routers' own tables went to 0–4 of 6 (three at 0). Members found no providers, FindPeer answered nothing, and callers fell back to circuits through their own routers, which refused a destination they did not hold.

On the live routers during ladder 10: /mesh/kad/1.0.0 inbound streams 760, /meshsub/1.3.0 760, libp2p_rcmgr_blocked_resources{scope="protocol"} at 250k–267k per router, routing tables 0. This is the dominant cause of the 1–4 % data plane in ladder runs 7–9, with the per-IP cap (#663) second.

  • Router and node budget the DHT protocol, the GossipSub protocols and (node) MCP like the auth handshake, from the connection budget.
  • p2plimits.StreamRefusals, an rcmgr MetricsReporter: every refused stream counted by scope and protocol in agentmesh_router_streams_refused_total / agentmesh_node_streams_refused_total, logged once a minute per protocol. libp2p's gauge only says "protocol".
  • api.DHTProtocolPrefix / api.DHTProtocolID replace the two hard-coded prefixes; a test builds a DHT server the router's way and checks it registers the ID the limits budget for.

2. router: record where a member that reserved with it is reached

A member tells its other routers about its reservations through identify, but not the router it reserved with: identify ran there before the reservation, and an SDK member listens on nothing else. On bananas the JS agent's own reservation router held no address for it; only the other router did. With two of seven routers held, a lookup found the agent only when it happened to reach that other router — the probe's SDK stage failed about half the time.

When the relay ACL grants a reservation the router adds <advertised>/p2p/<router>/p2p-circuit for the member to its peerstore, for the reservation's lifetime (identify rewrites a peer's connected-TTL addresses whenever the peer announces new ones; this TTL it leaves alone), and forgetPeer drops it on disconnect, ban or expiry — those paths now all go through forgetPeer. The advertised address set is shared with the lease, and a malformed --external-addr is a start-up error rather than an address silently dropped.

3. node: take the address a failed lookup learned, clear the backoff its dial left

kad-dht's FindPeer dials the peer during the lookup and answers "not found" when it cannot reach it, whatever the lookup learned on the way. A peer reached only through a router this node has not handshaken with is exactly that case: the router refuses the circuit to a caller it has not admitted. The address the lookup learned sat in the peerstore — and in the swarm's dial backoff.

preparePeerAddrs reads the peerstore after a failed lookup and goes on with it, so selectTransit handshakes with the router the address names, then clears the swarm's backoff for the peer. Lookups are counted in agentmesh_node_peer_lookups_total{found,not_found,failed}; a miss is logged at Info with the routing-table size, a failed lookup is a warning. The test fails without the fallback (0 handshakes on the router the lookup named).

Validation

make lint 0 issues; make test green; router/caller integration tests green. Docs: scalability guide (new rows, stream budget paragraph), router and node references, networking concept.

After deploy, the things to watch on bananas: agentmesh_router_dht_routing_table_size back to 6 on every router, agentmesh_router_streams_refused_total flat, agentmesh_node_peer_lookups_total{result="not_found"} on the probe, and router RSS over a full ladder hold (the recorded circuit addresses are one peerstore entry per reservation, dropped with the member).

aojea added 3 commits October 11, 2026 21:03
…used streams by protocol

A member holds one DHT request stream and one GossipSub stream open to its
router for as long as it is connected. hubLimits named only identify, the
auth handshake and the relay protocols; the DHT and GossipSub fell to
libp2p's default for an unnamed protocol, which scales from host memory and
was 768 inbound streams on the testnet's routers. Past 768 members a router
refused every further DHT and gossip stream: kad-dht then failed the
FIND_NODE check it runs before adding a peer to a routing table, so the
router never entered those members' tables and fell out of the ones it was
in, and the routers' own tables went to 0–4 of 6. Members found no
providers, FindPeer answered nothing and the caller fell back to circuits
through its own routers, which refused a destination they did not hold.
The testnet showed it as libp2p_rcmgr_blocked_resources{scope="protocol"}
at 53k–71k per router with no protocol named, and the cold-path probe's
SDK stage failing about half the time once members held two of seven
routers.

The router and the node now budget the DHT protocol, the GossipSub
protocols and (at the node) the MCP protocol like the auth handshake, one
stream per member of the connection budget, and the resource manager is
built with a MetricsReporter that counts every refused stream by scope and
protocol in agentmesh_router_streams_refused_total and
agentmesh_node_streams_refused_total, logged once a minute per protocol.
The DHT protocol prefix and ID move to api so the limits and the DHT name
the same protocol; a test builds a DHT server the way the router does and
checks it registers the ID the limits budget for.
A member tells its other routers about its relay reservations through
identify, but not the router it reserved with: identify ran there before
the reservation, and an SDK member listens on nothing else, so that
router's peerstore held no address for it. A DHT lookup for the member
that reached its own reservation router learned nothing, and since the
member held two of seven routers the caller found it only when the
lookup happened to reach the other one. On the testnet the cold-path
probe's SDK stage failed about half the time this way.

When the relay ACL grants a reservation the router now adds the member's
address through itself (each advertised address with
/p2p/<router>/p2p-circuit) to its peerstore, for the reservation's
lifetime so that identify, which rewrites a peer's connected-TTL
addresses whenever the peer announces new ones, leaves it alone, and
forgetPeer drops it with the admission on disconnect, ban or expiry. The
advertised address set is shared with the lease, and a malformed
--external-addr is a start-up error rather than an address dropped from
the lease. A test reserves from a member that listens on nothing and
checks a third peer's lookup through the router leaves the circuit
address in its peerstore.
… dial left

kad-dht's FindPeer dials the peer during the lookup and answers "not
found" when it cannot reach it, whatever the lookup learned on the way.
A peer reached only through a router this node has not handshaken with
is exactly that: the router refuses the circuit to a caller it has not
admitted. The node then fell back to circuits through its own routers,
which reach the peer only when it holds one of them, while the address
the lookup had learned sat in the peerstore, and in the swarm's dial
backoff from the refused dial.

preparePeerAddrs now reads the peerstore after a failed lookup and goes
on with what is there, so selectTransit handshakes with the router the
address names; once the addresses are set it clears the swarm's backoff
for the peer, which the handshake has outdated. Every lookup is counted
in agentmesh_node_peer_lookups_total{result} and a miss is logged at Info
with the routing table size, where it was a debug line; a failed lookup
is a warning. A test puts a peer's circuit in a DHT server the node
holds, with the relay being a router it does not, and checks the node
handshakes with that router, keeps the circuit and leaves no backoff.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances DHT lookup handling and resource management by introducing protocol-specific stream tracking and metrics for stream refusals. It ensures peer addresses are correctly recorded during relay reservations and updates resource limits to account for protocol-specific stream usage. The code review identified a potential nil pointer dereference in the sendLease function, which requires a nil check for r.Host to prevent runtime panics.

Comment thread internal/router/router.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant