node: size inbound limits from the connection budget and count refusals

[aojea]

Every ladder run on bananas (runs 7–9, up to 10,000 resident members) had the data plane at a few percent: a provider called by a fleet answered the first callers and then nothing, and callers saw CONNECTION_REFUSED (remote) over QUIC or connection reset by peer before TLS.

Root cause. The node ran libp2p's default resource manager: eight inbound connections per source address, the rest scaled from the memory the process sees (126 inbound on the canary pod). The per-address refusal happens before any trace event, so libp2p_rcmgr_blocked_resources stayed at zero, and the QUIC transport takes the decision in its ConnContext hook before the handshake, which is the CONNECTION_REFUSED the callers reported. Loopback is exempt, so no local run showed it. Reproduced on a non-loopback address: 60 callers from one host reached a provider 15 times.

Fix. The node builds its resource manager the way the router does, moved to internal/p2plimits for both: limits sized from the node's connection budget (400 inbound, its connection manager's high watermark), a per-source cap of a quarter of it, every refusal counted in agentmesh_node_inbound_connections_refused_total{reason} and logged once per source per minute, and the cap exported as agentmesh_node_conns_per_source_ip_limit. The same 300 callers from one host now reach the provider up to the cap, and the rest show in the counter (per_ip_limit 575, rate_limit 4034).

Router. Its limits followed the default watermark rather than the configured one, and its connection cap sat at the watermark, so a router refused connections before it counted itself full enough to shed: the rollout of the 10k mesh read 92,702 scope_limit refusals and no go-away. The cap is now twice the configured watermark.

Deploy. Routers get --high-watermark from ROUTER_HIGH_WATERMARK, 6000 by default: with two routers per member, six routers hold 10,000 members with one router out for a rollout. The router pod requests the memory a router at that watermark uses (1 GiB, 500m CPU).

Tests: p2plimits (refusal classification, log rate limiting, a live per-IP cap over the resource manager), TestNodeLimitsFollowTheConnectionBudget, TestHubLimitsFollowTheConnectionBudget updated for the configured watermark and headroom, node metrics test asserts the new series. Lint, unit, integration and generated-file checks pass.

[gemini-code-assist]

Code Review

This pull request introduces a shared p2plimits package to centralize libp2p resource manager limits, per-IP connection caps, and refusal logging for both nodes and routers. It configures the node to use this resource manager, updates Prometheus metrics to track inbound connection refusals, and updates documentation and tests accordingly. The review feedback focuses on correctness and defensive programming, recommending safe defaults for ConnsPerSourceIP in the router configuration to prevent breaking functionality when unconfigured, as well as adding nil checks for the warnf logging function and the endpoint multiaddr in the p2plimits package to prevent potential nil pointer dereference panics.