docs(samples): add multi-server/multi-cluster OpenTelemetry sample - #27
Open
gabriele-wolfox wants to merge 5 commits into
Open
docs(samples): add multi-server/multi-cluster OpenTelemetry sample#27gabriele-wolfox wants to merge 5 commits into
gabriele-wolfox wants to merge 5 commits into
Conversation
gabriele-wolfox
force-pushed
the
dev/8879
branch
4 times, most recently
from
August 5, 2026 11:30
d3704cd to
9c43f63
Compare
jbattiato
force-pushed
the
dev/8879
branch
2 times, most recently
from
August 7, 2026 13:46
eb67985 to
082506e
Compare
gabriele-wolfox
force-pushed
the
dev/8879
branch
from
August 10, 2026 13:27
089330a to
5f68849
Compare
Adds operator/config/samples/opentelemetry-multi, deploying two Klio servers and three CNPG clusters (one in a different namespace, backed cross-namespace by one of the servers) against the same OTel/Prometheus/ Grafana stack the single-server sample uses. The existing sample only ever exercises one server and one cluster, so it can't validate the Grafana dashboard's namespace/server/cluster template variables or its per-tier and per-cluster aggregations. Validated end to end on Kind: all three clusters reconcile, stream WAL and complete backups, including the cross-namespace client whose certificates are bridged via copy-cross-namespace-secrets.sh (cert-manager Issuers are namespace scoped, so that certificate has to be generated in the server's namespace and copied over rather than issued locally). Closes #26 Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
Adds a fourth deployment to opentelemetry-multi: team-e runs its own Klio server also named "klio-a" (see ../klio_server_a.yaml in "default"). Since a Server's StatefulSet pod name is derived from the Server's own name alone, both servers' pods are named "klio-a-klio-0", giving them an identical host_name label -- a case the dashboard's $server variable cannot disambiguate on its own. bootstrap-remote-server.sh bootstraps this scenario: it copies the shared CA's public certificate (so team-e's independently self-signed server can validate clients signed by the same klio-server-ca) and the OTel collector trust anchor, plus one cluster's client certificate generated in "default" (cert-manager Issuers are namespace-scoped). Confirmed live on Kind: both servers run independently and cluster-e backs up successfully. Querying Prometheus directly shows two distinct series for host_name=klio-a-klio-0 (uptime ~9778s in default vs ~299s in team-e, disambiguated only by k8s_namespace_name); the dashboard's actual "Server uptime" panel query, with $namespace left at its default "All" and $server narrowed to klio-a-klio-0, collapses both into a single value and silently picks one. Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
Merges operator/config/samples/opentelemetry and opentelemetry-multi into one tree (base/single/multi) so the shared OTel/Jaeger/CA infrastructure is defined once instead of duplicated byte-for-byte across both directories, which had already drifted (e.g. mismatched otel-collector-contrib versions). multi's per-instance servers and clusters now reuse single's Server/Cluster definitions via kustomize resource+patch composition instead of duplicating their spec, and team-e is renamed to team-d for consistency with the other team-* namespaces. The klio-server-ca Issuer becomes a ClusterIssuer so cluster-c's and cluster-d's client-auth certificates can be requested directly in their own namespace instead of being generated in "default" and copied over by script; the two remaining scripts now only copy secrets that are genuinely pinned by exact bytes rather than CA-validated. Also fixes two related sample bugs: a missing cluster-d Backup in multi/backups-example.yaml, and the complete absence of a backup example for the "single" scenario. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Jonathan Battiato <jonathan.battiato@enterprisedb.com>
…ples Each cluster now gets its own cnpg.io/cluster-scoped PodMonitor, bundled into single/cluster (the reusable cluster unit multi/team-a through team-d already build on) and patched per instance like the rest of that unit's resources, following the same pattern used in the sibling klio-benchmark repo. Without it, Prometheus never sees CloudNativePG's own cnpg_pg_stat_replication_* metrics, which the Grafana dashboard's WAL Replication Lag row depends on. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Jonathan Battiato <jonathan.battiato@enterprisedb.com>
…icit setup step The opentelemetry sample's klio-server-ca ClusterIssuer only resolves its CA secret from cert-manager's --cluster-resource-namespace, which this sample assumes is "default". The only concrete setup instructions the README gives (task integration:deploy-to-kind) install cert-manager without that flag, defaulting to the "cert-manager" namespace instead; the override lived only as a prose aside under Prerequisites, so a reader following the documented flow ends up with a permanently unresolvable ClusterIssuer. Confirmed live: on a cluster set up via task integration:deploy-to-kind, klio-server-ca's ClusterIssuer reports "secrets \"klio-server-ca\" not found", every client certificate in the sample (single and multi alike) gets stuck at Ready: False, and every dependent cluster never leaves "Setting up primary". Re-running the corrected helm upgrade command below flips the ClusterIssuer to Ready and the stuck certificates issue immediately. Moves the fix into an explicit step right after the deploy-to-kind command instead of a Prerequisites aside, and corrects that aside's claim that the namespace mismatch "matters only for cluster-c's and cluster-d's": ClusterIssuer resolution is a single, fixed lookup for the whole cert-manager installation, not scoped per requesting namespace, so it affects cluster-a's and cluster-b's certificates identically. Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
gabriele-wolfox
force-pushed
the
dev/8879
branch
from
August 10, 2026 13:27
5f68849 to
d1de89a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
operator/config/samples/opentelemetry-multi, deploying two Klio servers and three CNPG clusters (one in a different namespace, backed cross-namespace by one of the servers) against the same OTel/Prometheus/Grafana stack the single-server sample uses.operator/config/samples/opentelemetrysample only ever exercises one server and one cluster, so it can't validate the Grafana dashboard's$namespace/$server/$clustertemplate variables or its per-tier and per-cluster aggregationscopy-cross-namespace-secrets.sh, which bridges the cross-namespace client's certificates (cert-managerIssuers are namespace scoped, so that certificate has to be generated in the server's namespace and copied over rather than issued locally), andbackups-example.yamlto trigger a base backup on all three clusters.team-e, exercising a second Klio server that reuses the nameklio-a. Since a Server's StatefulSet pod name is derived from the Server's own name alone, both servers' pods share the identical host_nameklio-a-klio-0, a case the dashboard's$servervariable can't disambiguate on its own.bootstrap-remote-server.shbootstraps this scenario (copies the shared CA's public certificate and the OTel collector trust anchor so the independently self-signed server can participate).Closes #26
Test plan
pg-operator-e2e-v1-35-1):kubectl apply -k operator/config/samples/opentelemetry-multi, then./copy-cross-namespace-secrets.sh, thenkubectl apply -k operator/config/samples/opentelemetry-multi/team-ccluster-a,cluster-b,cluster-c) reachedCluster in healthy stateklio-breceiving WAL taggedclusterName: cluster-cfrom theteam-cnamespace)backups-example.yaml)team-e:./bootstrap-remote-server.sh team-e cluster-e-klio-user, thenkubectl apply -k operator/config/samples/opentelemetry-multi/team-eklio-aservers (defaultandteam-e) run independently;cluster-ecompleted a base backuphost_name=klio-a-klio-0, distinguished only byk8s_namespace_name