feat(secretmanager): add OpenTelemetry distributed tracing samples

[chalmerlowe]

Note

WORK IN PROGRESS. NOT READY FOR REVIEW

Description

This pull request adds canonical code samples and companion unit test suites demonstrating native OpenTelemetry distributed tracing in the Secret Manager Python client library (SecretManagerServiceClient and SecretManagerServiceAsyncClient).

Samples Added

1. Quickstart Console Tracing (secretmanager_trace.py):
   • Region tag: secretmanager_enable_tracing
   • Standard OTel initialization with ConsoleSpanExporter and global activation via GOOGLE_SDK_PYTHON_TRACING_ENABLED=true.
2. Google Cloud Trace Exporter (secretmanager_trace_cloud_trace.py):
   • Region tag: secretmanager_trace_cloud_trace
   • Production export to Google Cloud Trace using CloudTraceSpanExporter and BatchSpanProcessor.
3. Programmatic Configuration (secretmanager_trace_client_options.py):
   • Region tag: secretmanager_trace_client_options
   • Passing an independent TracerProvider via ClientOptions(tracer_provider=...) for fine-grained or multi-tenant control without global environment variables.
4. Asynchronous Client Tracing (secretmanager_trace_async.py):
   • Region tag: secretmanager_enable_tracing_async
   • Async tracing using SecretManagerServiceAsyncClient within an asyncio event loop.

Testing

• Isolated unit test suites using in-memory mock transports and servers.
• Added conftest.py fixture to ensure clean OpenTelemetry global state between test runs.
• Formatted and linted with black and flake8.

[gemini-code-assist]

Code Review

This pull request introduces several code snippets and unit tests demonstrating OpenTelemetry distributed tracing with the Google Cloud Secret Manager client library, covering synchronous, asynchronous, ClientOptions, and Google Cloud Trace configurations. The review feedback highlights a critical architectural anti-pattern across multiple snippets where the global TracerProvider is initialized and shut down inside reusable utility functions. To prevent global side effects and premature shutdowns, the reviewer recommends moving the TracerProvider configuration and shutdown logic to the application entry points (main blocks) and updating the unit tests accordingly.

[gemini-code-assist]

Initializing and shutting down the global TracerProvider inside a reusable utility function like access_secret_version_with_tracing is a major anti-pattern and can cause bugs in production.

1. Global Side Effects: Calling trace.set_tracer_provider(provider) inside the function will log warnings or fail on subsequent calls because the global tracer provider can only be set once.
2. Premature Shutdown: Calling provider.shutdown() in the finally block will shut down the global tracer provider. Any subsequent client calls or other parts of the application trying to use the global tracer provider will fail to emit spans.

Recommendation:
Move the OpenTelemetry configuration (TracerProvider initialization, global registration, and shutdown) to the application's entry point (the __main__ block). This keeps the utility function clean, reusable, and free of global side effects.

Suggested change

	# 1. Initialize the OpenTelemetry TracerProvider with a Console exporter
	provider = TracerProvider()
	provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
	trace.set_tracer_provider(provider)
	# 2. Enable Google Cloud client library tracing
	os.environ["GOOGLE_SDK_PYTHON_TRACING_ENABLED"] = "true"
	# 3. Instantiate the client (automatically discovers the active TracerProvider)
	client = secretmanager.SecretManagerServiceClient()
	# 4. Construct the resource name and invoke the RPC
	name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
	try:
	response = client.access_secret_version(request={"name": name})
	payload = response.payload.data.decode("UTF-8")
	return payload
	finally:
	# Flush buffered spans before program completion
	provider.shutdown()
	# Enable Google Cloud client library tracing
	os.environ["GOOGLE_SDK_PYTHON_TRACING_ENABLED"] = "true"
	# Instantiate the client (automatically discovers the active TracerProvider)
	client = secretmanager.SecretManagerServiceClient()
	# Construct the resource name and invoke the RPC
	name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
	response = client.access_secret_version(request={"name": name})
	payload = response.payload.data.decode("UTF-8")
	return payload

[gemini-code-assist]

Update the __main__ block to initialize the global TracerProvider and ensure it is properly shut down when the program exits. Note that the --project-id argument should be optional, as the project can be inferred from Application Default Credentials (ADC) if not provided.

if __name__ == "__main__":
    import argparse

    parser = argparse.ArgumentParser(
        description="Access a Secret Manager secret version with OpenTelemetry tracing."
    )
    parser.add_argument("--project-id", help="Google Cloud Project ID")
    parser.add_argument("--secret-id", required=True, help="Secret ID")
    parser.add_argument("--version-id", default="latest", help="Secret version ID")

    args = parser.parse_args()

    # Initialize the OpenTelemetry TracerProvider with a Console exporter
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
    trace.set_tracer_provider(provider)

    try:
        secret_data = access_secret_version_with_tracing(
            project_id=args.project_id,
            secret_id=args.secret_id,
            version_id=args.version_id,
        )
        print(f"Retrieved secret payload: {secret_data}")
    finally:
        # Flush buffered spans before program completion
        provider.shutdown()

References

1. When using argparse for a command-line script, the --project_id argument can be optional, as the project can be inferred from Application Default Credentials (ADC) if not provided.

[gemini-code-assist]

Similar to the synchronous sample, initializing and shutting down the global TracerProvider inside access_secret_version_with_tracing_async introduces global side effects and prematurely shuts down tracing for the rest of the application.

Recommendation:
Move the OpenTelemetry configuration and shutdown to the __main__ block.

Suggested change

	# 1. Initialize the OpenTelemetry TracerProvider with a Console exporter
	provider = TracerProvider()
	provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
	trace.set_tracer_provider(provider)
	# 2. Enable Google Cloud client library tracing
	os.environ["GOOGLE_SDK_PYTHON_TRACING_ENABLED"] = "true"
	# 3. Instantiate the async client
	client = secretmanager_v1.SecretManagerServiceAsyncClient()
	# 4. Construct resource name and invoke the async RPC
	name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
	try:
	response = await client.access_secret_version(request={"name": name})
	payload = response.payload.data.decode("UTF-8")
	return payload
	finally:
	# Flush buffered spans before program completion
	provider.shutdown()
	# Enable Google Cloud client library tracing
	os.environ["GOOGLE_SDK_PYTHON_TRACING_ENABLED"] = "true"
	# Instantiate the async client
	client = secretmanager_v1.SecretManagerServiceAsyncClient()
	# Construct resource name and invoke the async RPC
	name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
	response = await client.access_secret_version(request={"name": name})
	payload = response.payload.data.decode("UTF-8")
	return payload

[gemini-code-assist]

Update the __main__ block to initialize the global TracerProvider and ensure it is properly shut down when the program exits. Note that the --project-id argument should be optional, as the project can be inferred from Application Default Credentials (ADC) if not provided.

if __name__ == "__main__":
    import argparse

    parser = argparse.ArgumentParser(
        description="Access a Secret Manager secret version asynchronously with OpenTelemetry tracing."
    )
    parser.add_argument("--project-id", help="Google Cloud Project ID")
    parser.add_argument("--secret-id", required=True, help="Secret ID")
    parser.add_argument("--version-id", default="latest", help="Secret version ID")

    args = parser.parse_args()

    # Initialize the OpenTelemetry TracerProvider with a Console exporter
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
    trace.set_tracer_provider(provider)

    try:
        secret_data = asyncio.run(
            access_secret_version_with_tracing_async(
                project_id=args.project_id,
                secret_id=args.secret_id,
                version_id=args.version_id,
            )
        )
        print(f"Retrieved secret payload asynchronously: {secret_data}")
    finally:
        # Flush buffered spans before program completion
        provider.shutdown()

References

1. When using argparse for a command-line script, the --project_id argument can be optional, as the project can be inferred from Application Default Credentials (ADC) if not provided.

[gemini-code-assist]

Similar to the other samples, initializing and shutting down the global TracerProvider inside access_secret_version_cloud_trace introduces global side effects and prematurely shuts down tracing for the rest of the application.

Recommendation:
Move the OpenTelemetry configuration and shutdown to the __main__ block.

Suggested change

	# 1. Initialize OpenTelemetry TracerProvider with CloudTraceSpanExporter and BatchSpanProcessor
	provider = TracerProvider()
	cloud_exporter = CloudTraceSpanExporter()
	provider.add_span_processor(BatchSpanProcessor(cloud_exporter))
	trace.set_tracer_provider(provider)
	# 2. Enable Google Cloud client library tracing
	os.environ["GOOGLE_SDK_PYTHON_TRACING_ENABLED"] = "true"
	# 3. Instantiate the client (automatically discovers the active TracerProvider)
	client = secretmanager.SecretManagerServiceClient()
	# 4. Construct resource name and invoke RPC
	name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
	try:
	response = client.access_secret_version(request={"name": name})
	payload = response.payload.data.decode("UTF-8")
	return payload
	finally:
	# Flush buffered spans to Cloud Trace before exiting
	provider.shutdown()
	# Enable Google Cloud client library tracing
	os.environ["GOOGLE_SDK_PYTHON_TRACING_ENABLED"] = "true"
	# Instantiate the client (automatically discovers the active TracerProvider)
	client = secretmanager.SecretManagerServiceClient()
	# Construct resource name and invoke RPC
	name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
	response = client.access_secret_version(request={"name": name})
	payload = response.payload.data.decode("UTF-8")
	return payload

[gemini-code-assist]

Update the __main__ block to initialize the global TracerProvider with the Cloud Trace exporter and ensure it is properly shut down when the program exits. Note that the --project-id argument should be optional, as the project can be inferred from Application Default Credentials (ADC) if not provided.

if __name__ == "__main__":
    import argparse

    parser = argparse.ArgumentParser(
        description="Access a Secret Manager secret version and export traces to Google Cloud Trace."
    )
    parser.add_argument("--project-id", help="Google Cloud Project ID")
    parser.add_argument("--secret-id", required=True, help="Secret ID")
    parser.add_argument("--version-id", default="latest", help="Secret version ID")

    args = parser.parse_args()

    # Initialize OpenTelemetry TracerProvider with CloudTraceSpanExporter and BatchSpanProcessor
    provider = TracerProvider()
    cloud_exporter = CloudTraceSpanExporter()
    provider.add_span_processor(BatchSpanProcessor(cloud_exporter))
    trace.set_tracer_provider(provider)

    try:
        secret_data = access_secret_version_cloud_trace(
            project_id=args.project_id,
            secret_id=args.secret_id,
            version_id=args.version_id,
        )
        print(f"Retrieved secret payload: {secret_data}")
    finally:
        # Flush buffered spans to Cloud Trace before exiting
        provider.shutdown()

References

1. When using argparse for a command-line script, the --project_id argument can be optional, as the project can be inferred from Application Default Credentials (ADC) if not provided.

[gemini-code-assist]

Since we removed the TracerProvider instantiation from the utility function, we no longer need to mock it in the test. Instead, we can set the global tracer provider directly in the test setup.

Suggested change

	with mock.patch("secretmanager_trace.TracerProvider", return_value=provider):
	with mock.patch.object(
	SecretManagerServiceGrpcTransport, "__init__", patched_init
	):
	payload = access_secret_version_with_tracing(
	project_id="test-project",
	secret_id="my-secret",
	version_id="1",
	)
	trace.set_tracer_provider(provider)
	with mock.patch.object(
	SecretManagerServiceGrpcTransport, "__init__", patched_init
	):
	payload = access_secret_version_with_tracing(
	project_id="test-project",
	secret_id="my-secret",
	version_id="1",
	)

[gemini-code-assist]

Since we removed the TracerProvider instantiation from the async utility function, we no longer need to mock it in the test. Instead, we can set the global tracer provider directly in the test setup.

Suggested change

	with mock.patch("secretmanager_trace_async.TracerProvider", return_value=provider):
	with mock.patch.object(
	SecretManagerServiceGrpcAsyncIOTransport, "__init__", patched_init
	):
	payload = await access_secret_version_with_tracing_async(
	project_id="test-project",
	secret_id="my-secret",
	version_id="1",
	)
	trace.set_tracer_provider(provider)
	with mock.patch.object(
	SecretManagerServiceGrpcAsyncIOTransport, "__init__", patched_init
	):
	payload = await access_secret_version_with_tracing_async(
	project_id="test-project",
	secret_id="my-secret",
	version_id="1",
	)

[chalmerlowe]

Closing in favor of #14647, which is opened directly from the upstream branch.