Skip to content

feat: add o11y gRPC tracing - #9267

Draft
shivanee-p wants to merge 7 commits into
shivaneep-o11y-tracer-helper-updatesfrom
shivaneep-o11y-grpc-tracing
Draft

feat: add o11y gRPC tracing#9267
shivanee-p wants to merge 7 commits into
shivaneep-o11y-tracer-helper-updatesfrom
shivaneep-o11y-grpc-tracing

Conversation

@shivanee-p

Copy link
Copy Markdown
Contributor

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request integrates OpenTelemetry tracing into API calls within google-gax. It updates createApiCall to wrap invocations with traceAttempt when telemetry is enabled, captures static and dynamic trace contexts, and propagates the internal method name via constructSettings. Additionally, traceAttempt is refactored to synchronously handle both synchronous results and Promises. The feedback highlights two important issues: first, checking result instanceof Promise is fragile and fails to correctly track the lifetime of streaming calls (which return streams rather than Promises); second, the unit tests use a synchronous callback mock that masks this premature span termination. It is recommended to check for thenables and streams to properly manage span lifetimes, and to update the tests to invoke callbacks asynchronously.

Comment on lines 117 to 130
if (result instanceof Promise) {
result
.then(() => {
span.end();
return null;
})
.catch(err => {
recordError(err);
span.end();
});
return result;
}
span.end();
return result;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using result instanceof Promise to detect asynchronous operations is fragile because custom promise implementations (such as CancellablePromise in google-gax) might not inherit directly from the global Promise class.

Additionally, for streaming calls, invokeCall returns a stream/EventEmitter rather than a Promise. Since a stream is not an instance of Promise, the span is ended synchronously and immediately, which prevents capturing the actual duration of the stream or any asynchronous errors.

To resolve both issues, check if the result is a thenable (has a .then method) or a stream (has an .on method) to correctly manage the span lifetime.

      if (result && typeof (result as any).then === 'function') {
        (result as any)
          .then(() => {
            span.end();
            return null;
          })
          .catch((err: unknown) => {
            recordError(err);
            span.end();
          });
        return result;
      }
      if (result && typeof (result as any).on === 'function') {
        const stream = result as any;
        let ended = false;
        const endSpan = () => {
          if (!ended) {
            ended = true;
            span.end();
          }
        };
        stream.on('end', endSpan);
        stream.on('close', endSpan);
        stream.on('error', (err: any) => {
          recordError(err);
          endSpan();
        });
        return result;
      }
      span.end();
      return result;

Comment on lines +589 to +600
function failingFunc(
argument: {},
metadata: {},
options: {},
callback: (err: GoogleError | null, resp?: unknown) => void,
) {
const error = new GoogleError('RPC test failure');
callback(error);
return {
cancel: () => {},
};
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The mock failingFunc invokes the callback synchronously. This masks a bug in traceAttempt where real asynchronous callback-based or streaming calls (which return control objects synchronously and invoke callbacks/emit events asynchronously) will have their spans ended prematurely before the asynchronous error or completion occurs.

To make the test more realistic and catch these issues, the mock should invoke the callback asynchronously (e.g., using setImmediate or process.nextTick).

Suggested change
function failingFunc(
argument: {},
metadata: {},
options: {},
callback: (err: GoogleError | null, resp?: unknown) => void,
) {
const error = new GoogleError('RPC test failure');
callback(error);
return {
cancel: () => {},
};
}
function failingFunc(
argument: {},
metadata: {},
options: {},
callback: (err: GoogleError | null, resp?: unknown) => void,
) {
const error = new GoogleError('RPC test failure');
setImmediate(() => {
callback(error);
});
return {
cancel: () => {},
};
}

@shivanee-p
shivanee-p force-pushed the shivaneep-o11y-grpc-tracing branch 2 times, most recently from fa84a3f to 0706fc7 Compare September 8, 2026 23:31
@shivanee-p
shivanee-p changed the base branch from main to shivaneep-o11y-tracer-helper-updates September 8, 2026 23:33
@shivanee-p shivanee-p changed the title Shivaneep o11y grpc tracing feat: add o11y gRPC tracing Sep 8, 2026
@shivanee-p
shivanee-p force-pushed the shivaneep-o11y-grpc-tracing branch 2 times, most recently from d231ce3 to 6ae7e02 Compare September 8, 2026 23:53
@shivanee-p
shivanee-p force-pushed the shivaneep-o11y-grpc-tracing branch from 6ae7e02 to 57d1e9a Compare September 9, 2026 00:03
@shivanee-p
shivanee-p force-pushed the shivaneep-o11y-grpc-tracing branch 3 times, most recently from 6bc70b1 to 6ed7d57 Compare September 9, 2026 00:36
@shivanee-p
shivanee-p force-pushed the shivaneep-o11y-grpc-tracing branch from 6ed7d57 to 98c43f2 Compare September 9, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant