Sharp increase in S3 egress fees with OpenTelemetry metrics enabled

[bendilley]

Overview

Since upgrading to PGO 6.0 and pg18, I've had to switch to OpenTelemetry metrics because the old Exporter is no longer available. It works fine except that I've seen a very steep increase in AWS S3 data egress fees from the pgbackrest buckets I'm using for backup. It's not proportionate to the size of the dataset being backed-up: a very small database is costing around as much in egress as a much larger one. Removing the instrumentation: {} spec stops metrics and the egress fees drop back down to the expected near-zero level.

It's too expensive to sustain, so I've had to disable metrics on the pg18 clusters until this problem can be addressed.

Perhaps the instrumentation: {} config in a PostgresCluster spec could be expanded to allow the user to set a level of backup monitoring which works with the constraints of their egress budget?

Environment

Please provide the following details:

• Platform: Kubernetes
• Platform Version: 1.35.3
• PGO Image Tag: ubi9-6.0.2-0
• Postgres Version: 18
• Storage: AWS S3

Steps to Reproduce

REPRO

Provide steps to get to the error condition:

1. Add instrumentation: {} to a PostgresCluster spec which uses postgres 18 and a pgbackrest backup to an AWS S3 bucket.
2. Ensure the PGO has the OpenTelemetryMetrics=true feature flag set.
3. Monitor AWS egress cost.

EXPECTED

1. For data egress and request fees to run at near-zero cost.

ACTUAL

1. Egress fees from buckets run at several dollars per bucket, per day, which adds up to a significant monthly bill over multiple buckets.

Logs

The jump in July corresponds with rollout to a subset of our staging pg clusters:

Note the lack of variation in the cost of storage over the same period.

Additional Information

I can provide any more detail you require on request in a non-public channel.