Rust crates for SQD Network nodes: the services that collect pings and logs from workers, gateways, and portals.
SQD is an open data platform for Web3. SQD Network is a decentralized data lake and query engine run by worker, gateway, and scheduler nodes. This repository holds the shared and standalone Rust components used by those nodes. For the network protocol and node implementations, see subsquid/sqd-network.
This is a Cargo workspace. The crates under crates/ are:
| Crate | Type | Description |
|---|---|---|
logs-collector |
binary | Collects query-execution logs from workers and gateways over the network transport and stores them in ClickHouse. |
portal-logs-collector |
binary | Collects logs from SQD Portal instances and stores them in ClickHouse. |
pings-collector |
binary | Collects pings from workers and stores them in ClickHouse. |
collector-utils |
library | Shared ClickHouse storage types and helpers used by the collector binaries. |
The collector binaries connect to the network over sqd-network-transport (libp2p / QUIC) and persist to ClickHouse.
The archive router (router, router-controller) is on the arrowsquid branch.
Requires the Rust toolchain pinned in rust-toolchain (channel 1.83) and protobuf-compiler for building the protocol message dependencies.
# Build the whole workspace
cargo build --release --workspace
# Build a single crate, for example the pings collector
cargo build --release -p pings-collectorBinaries are written to target/release/.
The Dockerfile defines a build target for each collector:
docker build --target logs-collector -t logs-collector .
docker build --target pings-collector -t pings-collector .
docker build --target portal-logs-collector -t portal-logs-collector .Each binary is configured with command-line flags and environment variables (run with --help for the full list). The collector binaries read their P2P listen address from P2P_LISTEN_ADDRS and their ClickHouse connection details from flags and environment variables.
The scripts/ directory contains Python helpers used during development and operations (worker reachability and status checks, chunk summaries, scheduling simulation) and a traffic generator under scripts/traffic_generator.
- SQD documentation: https://docs.sqd.dev
- SQD Network: https://docs.sqd.dev/en/network
- SQD: https://sqd.dev
This project is licensed under the AGPL v3.0 license. See LICENSE.txt for details.