feat(spanner): support user-provided OpenTelemetry for built-in client metrics and endpoint attribute for location-aware routing

java-spanner/.readme-partials.yaml

@@ -53,29 +53,70 @@ custom_content: |
   ## Metrics
 
   Cloud Spanner client supports [client-side metrics](https://cloud.google.com/spanner/docs/view-manage-client-side-metrics) that you can use along with server-side metrics to optimize performance and troubleshoot performance issues if they occur.
-  
-  Client-side metrics are measured from the time a request leaves your application to the time your application receives the response. 
+
+  Client-side metrics are measured from the time a request leaves your application to the time your application receives the response.
   In contrast, server-side metrics are measured from the time Spanner receives a request until the last byte of data is sent to the client.
-  
-  These metrics are enabled by default. You can opt out of using client-side metrics with the following code:
-  
+
+  The default Cloud Monitoring export for these metrics is enabled by default. You can opt out of the default Cloud Monitoring export with the following code:
+
   ```
   SpannerOptions options = SpannerOptions.newBuilder()
     .setBuiltInMetricsEnabled(false)
     .build();
   ```
-
-  You can also disable these metrics by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`.
-
-  > Note: Client-side metrics needs `monitoring.timeSeries.create` IAM permission to export metrics data. Ask your administrator to grant your service account the [Monitoring Metric Writer](https://cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.metricWriter) (roles/monitoring.metricWriter) IAM role on the project.
+
+  You can also disable the default Cloud Monitoring export by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`. These controls affect only the Cloud Monitoring export. They do not affect a caller-owned client-metrics export configured with `CustomOpenTelemetryMetricsProvider`.
+
+  > Note: Client-side metrics needs `monitoring.timeSeries.create` IAM permission to export metrics data to Cloud Monitoring. Ask your administrator to grant your service account the [Monitoring Metric Writer](https://cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.metricWriter) (roles/monitoring.metricWriter) IAM role on the project.
+
+  #### Exporting client metrics to OpenTelemetry
+
+  Client metrics export to a caller-owned OpenTelemetry destination is controlled by a `MetricsProvider`,
+  set with `SpannerOptions.Builder.setClientMetricsProvider(MetricsProvider)`. The available
+  providers are:
+
+  * `DefaultMetricsProvider` (the default): no caller-owned client-metrics destination is configured.
+    The built-in Cloud Monitoring export follows `setBuiltInMetricsEnabled` and the
+    `SPANNER_DISABLE_BUILTIN_METRICS` environment variable. On Spanner Omni, where the Cloud Monitoring
+    export is not available, the default provider results in no client-metrics export.
+  * `NoopMetricsProvider`: caller-owned client metrics are explicitly disabled. The Cloud Monitoring
+    export is controlled separately.
+  * `CustomOpenTelemetryMetricsProvider`: the same Spanner client instruments are additionally recorded
+    on an `OpenTelemetry` instance that you provide. You own the metrics pipeline (readers, exporters
+    and resource). This custom destination is independent of the built-in Cloud Monitoring export on all
+    instance types, including Spanner Omni (`InstanceType.OMNI`), for which Cloud Monitoring export is
+    not available.
+
+  Client metrics are not recorded when the client runs against the Spanner emulator, regardless of the
+  configured `MetricsProvider`. gRPC-layer metrics are recorded on a custom destination only when the
+  provided `OpenTelemetry` instance is an `OpenTelemetrySdk`.
+
+  When using `CustomOpenTelemetryMetricsProvider`, it is recommended to register the Spanner
+  client-metrics views on a dedicated `SdkMeterProviderBuilder` with
+  `SpannerMetrics.configureMeterProviderBuilder(SdkMeterProviderBuilder)` before creating the
+  `OpenTelemetry` instance. The views rename the raw instruments, apply the Spanner latency
+  histogram buckets and restrict the recorded attributes to the supported client-metric labels. Attempt
+  metrics include the routed `endpoint` label when Spanner Omni location-aware routing supplies it.
+
+  ```java
+  SdkMeterProviderBuilder meterProviderBuilder =
+      SdkMeterProvider.builder().registerMetricReader(PeriodicMetricReader.create(myExporter));
+  SpannerMetrics.configureMeterProviderBuilder(meterProviderBuilder);
+  OpenTelemetry openTelemetry =
+      OpenTelemetrySdk.builder().setMeterProvider(meterProviderBuilder.build()).build();
+  SpannerOptions options =
+      SpannerOptions.newBuilder()
+          .setClientMetricsProvider(CustomOpenTelemetryMetricsProvider.create(openTelemetry))
+          .build();
+  ```
 
   ## Traces
-  Cloud Spanner client supports OpenTelemetry Traces, which gives insight into the client internals and aids in debugging/troubleshooting production issues. 
+  Cloud Spanner client supports OpenTelemetry Traces, which gives insight into the client internals and aids in debugging/troubleshooting production issues.
 
   By default, the functionality is disabled. You need to add OpenTelemetry dependencies, enable OpenTelemetry traces and must configure the OpenTelemetry with appropriate exporters at the startup of your application.
 
   See [Configure client-side tracing](https://cloud.google.com/spanner/docs/set-up-tracing#configure-client-side-tracing) for more details on configuring traces.
-  
+
   #### OpenTelemetry Dependencies
 
   If you are using Maven, add this to your pom.xml file
@@ -129,7 +170,7 @@ custom_content: |
 
   Spanner spanner = options.getService();
   ```
-  
+
   #### OpenTelemetry SQL Statement Tracing
   The OpenTelemetry traces that are generated by the Java client include any request and transaction
   tags that have been set. The traces can also include the SQL statements that are executed and the
@@ -149,42 +190,42 @@ custom_content: |
   #### OpenTelemetry API Tracing
   You can enable tracing of each API call that the Spanner client executes with the `enableApiTracing`
   option. These traces also include any retry attempts for an API call:
-    
+
   ```
   SpannerOptions options = SpannerOptions.newBuilder()
   .setOpenTelemetry(openTelemetry)
   .setEnableApiTracing(true)
   .build();
   ```
-    
+
   This option can also be enabled by setting the environment variable
   `SPANNER_ENABLE_API_TRACING=true`.
 
   > Note: The attribute keys that are used for additional information about retry attempts and the number of requests might change in a future release.
-  
-  #### End-to-end Tracing 
-              
+
+  #### End-to-end Tracing
+
   In addition to client-side tracing, you can opt in for [end-to-end tracing](https://cloud.google.com/spanner/docs/tracing-overview#end-to-end-side-tracing). End-to-end tracing helps you understand and debug latency issues that are specific to Spanner such as the following:
   * Identify whether the latency is due to network latency between your application and Spanner, or if the latency is occurring within Spanner.
   * Identify the Google Cloud regions that your application requests are being routed through and if there is a cross-region request. A cross-region request usually means higher latencies between your application and Spanner.
-  
+
   ```
   SpannerOptions options = SpannerOptions.newBuilder()
   .setOpenTelemetry(openTelemetry)
   .setEnableEndToEndTracing(true)
   .build();
   ```
-  
+
   Refer to [Configure end-to-end tracing](https://cloud.google.com/spanner/docs/set-up-tracing#configure-end-to-end-tracing) to configure end-to-end tracing and to understand its attributes.
-  
+
   > Note: End-to-end traces can only be exported to [Cloud Trace](https://cloud.google.com/trace/docs).
-  
-  
+
+
   ## Instrument with OpenCensus
 
   > Note: OpenCensus project is deprecated. See [Sunsetting OpenCensus](https://opentelemetry.io/blog/2023/sunsetting-opencensus/).
   We recommend migrating to OpenTelemetry, the successor project.
-  
+
   ## Migrate from OpenCensus to OpenTelemetry
 
   > Using the [OpenTelemetry OpenCensus Bridge](https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-opencensus-shim), you can immediately begin exporting your metrics and traces with OpenTelemetry.
@@ -214,4 +255,4 @@ custom_content: |
 
   Update your dashboards and alerts to reflect below changes
   * **Metrics name** : `cloud.google.com/java` prefix has been removed from OpenTelemery metrics and instead has been added as Instrumenation Scope.
-  * **Metrics namespace** : OpenTelmetry exporters uses `workload.googleapis.com` namespace opposed to `custom.googleapis.com` with OpenCensus. 
+  * **Metrics namespace** : OpenTelmetry exporters uses `workload.googleapis.com` namespace opposed to `custom.googleapis.com` with OpenCensus.

java-spanner/README.md

@@ -155,17 +155,58 @@ Cloud Spanner client supports [client-side metrics](https://cloud.google.com/spa
 Client-side metrics are measured from the time a request leaves your application to the time your application receives the response. 
 In contrast, server-side metrics are measured from the time Spanner receives a request until the last byte of data is sent to the client.
 
-These metrics are enabled by default. You can opt out of using client-side metrics with the following code:
+The default Cloud Monitoring export for these metrics is enabled by default. You can opt out of the default Cloud Monitoring export with the following code:
 
 ```
 SpannerOptions options = SpannerOptions.newBuilder()
   .setBuiltInMetricsEnabled(false)
   .build();
 ```
 
-You can also disable these metrics by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`.
+You can also disable the default Cloud Monitoring export by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`. These controls affect only the Cloud Monitoring export. They do not affect a caller-owned client-metrics export configured with `CustomOpenTelemetryMetricsProvider`.
 
-> Note: Client-side metrics needs `monitoring.timeSeries.create` IAM permission to export metrics data. Ask your administrator to grant your service account the [Monitoring Metric Writer](https://cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.metricWriter) (roles/monitoring.metricWriter) IAM role on the project.
+> Note: Client-side metrics needs `monitoring.timeSeries.create` IAM permission to export metrics data to Cloud Monitoring. Ask your administrator to grant your service account the [Monitoring Metric Writer](https://cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.metricWriter) (roles/monitoring.metricWriter) IAM role on the project.
+
+#### Exporting client metrics to OpenTelemetry
+
+Client metrics export to a caller-owned OpenTelemetry destination is controlled by a `MetricsProvider`,
+set with `SpannerOptions.Builder.setClientMetricsProvider(MetricsProvider)`. The available
+providers are:
+
+* `DefaultMetricsProvider` (the default): no caller-owned client-metrics destination is configured.
+  The built-in Cloud Monitoring export follows `setBuiltInMetricsEnabled` and the
+  `SPANNER_DISABLE_BUILTIN_METRICS` environment variable. On Spanner Omni, where the Cloud Monitoring
+  export is not available, the default provider results in no client-metrics export.
+* `NoopMetricsProvider`: caller-owned client metrics are explicitly disabled. The Cloud Monitoring
+  export is controlled separately.
+* `CustomOpenTelemetryMetricsProvider`: the same Spanner client instruments are additionally recorded
+  on an `OpenTelemetry` instance that you provide. You own the metrics pipeline (readers, exporters
+  and resource). This custom destination is independent of the built-in Cloud Monitoring export on all
+  instance types, including Spanner Omni (`InstanceType.OMNI`), for which Cloud Monitoring export is
+  not available.
+
+Client metrics are not recorded when the client runs against the Spanner emulator, regardless of the
+configured `MetricsProvider`. gRPC-layer metrics are recorded on a custom destination only when the
+provided `OpenTelemetry` instance is an `OpenTelemetrySdk`.
+
+When using `CustomOpenTelemetryMetricsProvider`, it is recommended to register the Spanner
+client-metrics views on a dedicated `SdkMeterProviderBuilder` with
+`SpannerMetrics.configureMeterProviderBuilder(SdkMeterProviderBuilder)` before creating the
+`OpenTelemetry` instance. The views rename the raw instruments, apply the Spanner latency
+histogram buckets and restrict the recorded attributes to the supported client-metric labels. Attempt
+metrics include the routed `endpoint` label when Spanner Omni location-aware routing supplies it.
+
+```java
+SdkMeterProviderBuilder meterProviderBuilder =
+    SdkMeterProvider.builder().registerMetricReader(PeriodicMetricReader.create(myExporter));
+SpannerMetrics.configureMeterProviderBuilder(meterProviderBuilder);
+OpenTelemetry openTelemetry =
+    OpenTelemetrySdk.builder().setMeterProvider(meterProviderBuilder.build()).build();
+SpannerOptions options =
+    SpannerOptions.newBuilder()
+        .setClientMetricsProvider(CustomOpenTelemetryMetricsProvider.create(openTelemetry))
+        .build();
+```
 
 ## Traces
 Cloud Spanner client supports OpenTelemetry Traces, which gives insight into the client internals and aids in debugging/troubleshooting production issues.