feat: add o11y gRPC tracing - #9267
Conversation
There was a problem hiding this comment.
Code Review
This pull request integrates OpenTelemetry tracing into API calls within google-gax. It updates createApiCall to wrap invocations with traceAttempt when telemetry is enabled, captures static and dynamic trace contexts, and propagates the internal method name via constructSettings. Additionally, traceAttempt is refactored to synchronously handle both synchronous results and Promises. The feedback highlights two important issues: first, checking result instanceof Promise is fragile and fails to correctly track the lifetime of streaming calls (which return streams rather than Promises); second, the unit tests use a synchronous callback mock that masks this premature span termination. It is recommended to check for thenables and streams to properly manage span lifetimes, and to update the tests to invoke callbacks asynchronously.
| if (result instanceof Promise) { | ||
| result | ||
| .then(() => { | ||
| span.end(); | ||
| return null; | ||
| }) | ||
| .catch(err => { | ||
| recordError(err); | ||
| span.end(); | ||
| }); | ||
| return result; | ||
| } | ||
| span.end(); | ||
| return result; |
There was a problem hiding this comment.
Using result instanceof Promise to detect asynchronous operations is fragile because custom promise implementations (such as CancellablePromise in google-gax) might not inherit directly from the global Promise class.
Additionally, for streaming calls, invokeCall returns a stream/EventEmitter rather than a Promise. Since a stream is not an instance of Promise, the span is ended synchronously and immediately, which prevents capturing the actual duration of the stream or any asynchronous errors.
To resolve both issues, check if the result is a thenable (has a .then method) or a stream (has an .on method) to correctly manage the span lifetime.
if (result && typeof (result as any).then === 'function') {
(result as any)
.then(() => {
span.end();
return null;
})
.catch((err: unknown) => {
recordError(err);
span.end();
});
return result;
}
if (result && typeof (result as any).on === 'function') {
const stream = result as any;
let ended = false;
const endSpan = () => {
if (!ended) {
ended = true;
span.end();
}
};
stream.on('end', endSpan);
stream.on('close', endSpan);
stream.on('error', (err: any) => {
recordError(err);
endSpan();
});
return result;
}
span.end();
return result;| function failingFunc( | ||
| argument: {}, | ||
| metadata: {}, | ||
| options: {}, | ||
| callback: (err: GoogleError | null, resp?: unknown) => void, | ||
| ) { | ||
| const error = new GoogleError('RPC test failure'); | ||
| callback(error); | ||
| return { | ||
| cancel: () => {}, | ||
| }; | ||
| } |
There was a problem hiding this comment.
The mock failingFunc invokes the callback synchronously. This masks a bug in traceAttempt where real asynchronous callback-based or streaming calls (which return control objects synchronously and invoke callbacks/emit events asynchronously) will have their spans ended prematurely before the asynchronous error or completion occurs.
To make the test more realistic and catch these issues, the mock should invoke the callback asynchronously (e.g., using setImmediate or process.nextTick).
| function failingFunc( | |
| argument: {}, | |
| metadata: {}, | |
| options: {}, | |
| callback: (err: GoogleError | null, resp?: unknown) => void, | |
| ) { | |
| const error = new GoogleError('RPC test failure'); | |
| callback(error); | |
| return { | |
| cancel: () => {}, | |
| }; | |
| } | |
| function failingFunc( | |
| argument: {}, | |
| metadata: {}, | |
| options: {}, | |
| callback: (err: GoogleError | null, resp?: unknown) => void, | |
| ) { | |
| const error = new GoogleError('RPC test failure'); | |
| setImmediate(() => { | |
| callback(error); | |
| }); | |
| return { | |
| cancel: () => {}, | |
| }; | |
| } |
fa84a3f to
0706fc7
Compare
d231ce3 to
6ae7e02
Compare
6ae7e02 to
57d1e9a
Compare
6bc70b1 to
6ed7d57
Compare
…remature span closure tests
6ed7d57 to
98c43f2
Compare
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes #<issue_number_goes_here> 🦕