Repository navigation
Budget the DHT and gossip streams per member; find members through the router they reserved with - #669
Merged
Conversation
…used streams by protocol
A member holds one DHT request stream and one GossipSub stream open to its
router for as long as it is connected. hubLimits named only identify, the
auth handshake and the relay protocols; the DHT and GossipSub fell to
libp2p's default for an unnamed protocol, which scales from host memory and
was 768 inbound streams on the testnet's routers. Past 768 members a router
refused every further DHT and gossip stream: kad-dht then failed the
FIND_NODE check it runs before adding a peer to a routing table, so the
router never entered those members' tables and fell out of the ones it was
in, and the routers' own tables went to 0–4 of 6. Members found no
providers, FindPeer answered nothing and the caller fell back to circuits
through its own routers, which refused a destination they did not hold.
The testnet showed it as libp2p_rcmgr_blocked_resources{scope="protocol"}
at 53k–71k per router with no protocol named, and the cold-path probe's
SDK stage failing about half the time once members held two of seven
routers.
The router and the node now budget the DHT protocol, the GossipSub
protocols and (at the node) the MCP protocol like the auth handshake, one
stream per member of the connection budget, and the resource manager is
built with a MetricsReporter that counts every refused stream by scope and
protocol in agentmesh_router_streams_refused_total and
agentmesh_node_streams_refused_total, logged once a minute per protocol.
The DHT protocol prefix and ID move to api so the limits and the DHT name
the same protocol; a test builds a DHT server the way the router does and
checks it registers the ID the limits budget for.
A member tells its other routers about its relay reservations through identify, but not the router it reserved with: identify ran there before the reservation, and an SDK member listens on nothing else, so that router's peerstore held no address for it. A DHT lookup for the member that reached its own reservation router learned nothing, and since the member held two of seven routers the caller found it only when the lookup happened to reach the other one. On the testnet the cold-path probe's SDK stage failed about half the time this way. When the relay ACL grants a reservation the router now adds the member's address through itself (each advertised address with /p2p/<router>/p2p-circuit) to its peerstore, for the reservation's lifetime so that identify, which rewrites a peer's connected-TTL addresses whenever the peer announces new ones, leaves it alone, and forgetPeer drops it with the admission on disconnect, ban or expiry. The advertised address set is shared with the lease, and a malformed --external-addr is a start-up error rather than an address dropped from the lease. A test reserves from a member that listens on nothing and checks a third peer's lookup through the router leaves the circuit address in its peerstore.
… dial left
kad-dht's FindPeer dials the peer during the lookup and answers "not
found" when it cannot reach it, whatever the lookup learned on the way.
A peer reached only through a router this node has not handshaken with
is exactly that: the router refuses the circuit to a caller it has not
admitted. The node then fell back to circuits through its own routers,
which reach the peer only when it holds one of them, while the address
the lookup had learned sat in the peerstore, and in the swarm's dial
backoff from the refused dial.
preparePeerAddrs now reads the peerstore after a failed lookup and goes
on with what is there, so selectTransit handshakes with the router the
address names; once the addresses are set it clears the swarm's backoff
for the peer, which the handshake has outdated. Every lookup is counted
in agentmesh_node_peer_lookups_total{result} and a miss is logged at Info
with the routing table size, where it was a debug line; a failed lookup
is a warning. A test puts a peer's circuit in a DHT server the node
holds, with the relay being a router it does not, and checks the node
handshakes with that router, keeps the circuit and leaves no backoff.
Contributor
There was a problem hiding this comment.
Code Review
This pull request enhances DHT lookup handling and resource management by introducing protocol-specific stream tracking and metrics for stream refusals. It ensures peer addresses are correctly recorded during relay reservations and updates resource limits to account for protocol-specific stream usage. The code review identified a potential nil pointer dereference in the sendLease function, which requires a nil check for r.Host to prevent runtime panics.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three fixes, one per commit, for what the 10k ladder and the cold-path probe showed on bananas today. Each commit builds and tests alone.
1. router, node: budget the DHT and gossip streams per member, count refused streams by protocol
hubLimitsnamed identify, the auth handshake and the relay protocols;/mesh/kad/1.0.0and GossipSub fell to libp2p's default for an unnamed protocol, which scales from host memory and was 768 inbound streams on the testnet's routers. A member holds one DHT request stream and one gossip stream to its router for as long as it is connected, so past 768 members a router refused every further one. kad-dht runs a FIND_NODE check before adding a peer to a routing table (logged at debug level only); the refused streams failed it, the router never entered those members' tables and fell out of the ones it was in, and the routers' own tables went to 0–4 of 6 (three at 0). Members found no providers,FindPeeranswered nothing, and callers fell back to circuits through their own routers, which refused a destination they did not hold.On the live routers during ladder 10:
/mesh/kad/1.0.0inbound streams 760,/meshsub/1.3.0760,libp2p_rcmgr_blocked_resources{scope="protocol"}at 250k–267k per router, routing tables 0. This is the dominant cause of the 1–4 % data plane in ladder runs 7–9, with the per-IP cap (#663) second.p2plimits.StreamRefusals, an rcmgrMetricsReporter: every refused stream counted by scope and protocol inagentmesh_router_streams_refused_total/agentmesh_node_streams_refused_total, logged once a minute per protocol. libp2p's gauge only says "protocol".api.DHTProtocolPrefix/api.DHTProtocolIDreplace the two hard-coded prefixes; a test builds a DHT server the router's way and checks it registers the ID the limits budget for.2. router: record where a member that reserved with it is reached
A member tells its other routers about its reservations through identify, but not the router it reserved with: identify ran there before the reservation, and an SDK member listens on nothing else. On bananas the JS agent's own reservation router held no address for it; only the other router did. With two of seven routers held, a lookup found the agent only when it happened to reach that other router — the probe's SDK stage failed about half the time.
When the relay ACL grants a reservation the router adds
<advertised>/p2p/<router>/p2p-circuitfor the member to its peerstore, for the reservation's lifetime (identify rewrites a peer's connected-TTL addresses whenever the peer announces new ones; this TTL it leaves alone), andforgetPeerdrops it on disconnect, ban or expiry — those paths now all go throughforgetPeer. The advertised address set is shared with the lease, and a malformed--external-addris a start-up error rather than an address silently dropped.3. node: take the address a failed lookup learned, clear the backoff its dial left
kad-dht's
FindPeerdials the peer during the lookup and answers "not found" when it cannot reach it, whatever the lookup learned on the way. A peer reached only through a router this node has not handshaken with is exactly that case: the router refuses the circuit to a caller it has not admitted. The address the lookup learned sat in the peerstore — and in the swarm's dial backoff.preparePeerAddrsreads the peerstore after a failed lookup and goes on with it, soselectTransithandshakes with the router the address names, then clears the swarm's backoff for the peer. Lookups are counted inagentmesh_node_peer_lookups_total{found,not_found,failed}; a miss is logged at Info with the routing-table size, a failed lookup is a warning. The test fails without the fallback (0 handshakes on the router the lookup named).Validation
make lint0 issues;make testgreen; router/caller integration tests green. Docs: scalability guide (new rows, stream budget paragraph), router and node references, networking concept.After deploy, the things to watch on bananas:
agentmesh_router_dht_routing_table_sizeback to 6 on every router,agentmesh_router_streams_refused_totalflat,agentmesh_node_peer_lookups_total{result="not_found"}on the probe, and router RSS over a full ladder hold (the recorded circuit addresses are one peerstore entry per reservation, dropped with the member).