Repository navigation
fix(bigtable): copy Metadata per GetClientConfiguration call - #14950
Conversation
There was a problem hiding this comment.
Code Review
This pull request improves the ClientConfigurationManager by preventing metadata accumulation across polls, avoiding retry amplification during background polls when falling back to the default configuration, and introducing a mockable Clock for better testability. It also adds comprehensive unit tests to verify these behaviors. The reviewer's feedback highlights an issue in the logging implementation where concatenating a Throwable into a log message discards the stack trace and suggests a Java 8-compatible logging pattern to preserve the stack trace and avoid unnecessary string concatenation.
| if (willRetry) { | ||
| logger.log( | ||
| Level.FINE, | ||
| () -> | ||
| "Failed to fetch client config (attempt " | ||
| + (attempt + 1) | ||
| + "), retrying: " | ||
| + throwable); | ||
| } else if (!isInitialPoll && !shouldReset) { |
There was a problem hiding this comment.
Concatenating the throwable into the log message string only prints its toString() representation, which discards the stack trace. For Level.FINE logs, having the full stack trace of the failed RPC is extremely valuable for debugging network or gRPC issues. Additionally, the lambda-based logging overload log(Level, Throwable, Supplier) is only available in Java 9+. If this library targets Java 8 compatibility, using standard log(Level, String, Throwable) wrapped in logger.isLoggable(Level.FINE) is safer and avoids string concatenation overhead when FINE logging is disabled.
| if (willRetry) { | |
| logger.log( | |
| Level.FINE, | |
| () -> | |
| "Failed to fetch client config (attempt " | |
| + (attempt + 1) | |
| + "), retrying: " | |
| + throwable); | |
| } else if (!isInitialPoll && !shouldReset) { | |
| if (willRetry) { | |
| if (logger.isLoggable(Level.FINE)) { | |
| logger.log( | |
| Level.FINE, | |
| "Failed to fetch client config (attempt " + (attempt + 1) + "), retrying", | |
| throwable); | |
| } | |
| } else if (!isInitialPoll && !shouldReset) { |
ClientConfigurationManager previously passed the same mutable Metadata instance to ClientCall.start() on every GetClientConfiguration poll and retry. Because ClientCall.start() takes ownership of the Metadata passed to it and channel ClientInterceptors may mutate it in place, per-RPC headers attached by interceptors accumulated across polls until outbound header encoding exceeded HTTP/2 SETTINGS_MAX_HEADER_LIST_SIZE. Copy the base Metadata into a fresh Metadata instance per ClientCall in sendRequest(), matching GrpcClientCalls, BigtableChannelPrimer, and SessionPoolImpl.
8dfb57d to
2456fe2
Compare
Problem
ClientConfigurationManagerconstructs a singleio.grpc.Metadatainstance in its constructor and passes that same mutable reference toClientCall.start(listener, metadata)on everyGetClientConfigurationpoll and retry. Ingrpc-java,ClientCall.start()takes ownership of theMetadatainstance, and channelClientInterceptors wraprealChannelaboveRetriableStreamand mutateheadersin place before any defensive copy is made. Whenever a channelClientInterceptor(for exampleMetadataUtils.newAttachHeadersInterceptoror a tracing interceptor callingheaders.put(...)instart()) appends per-RPC headers withoutdiscardAll, duplicate header entries accumulate onClientConfigurationManager.metadataevery 5 minutes until outbound header encoding exceeds the server's HTTP/2SETTINGS_MAX_HEADER_LIST_SIZE(64 KiB), causing Netty'sHpackEncoderto fail every subsequent poll locally withStatus.INTERNAL: http2 exception(HeaderListSizeException) — survivingreplaceChannel()until JVM restart.Fix
this.metadatainto a freshMetadatainstance perClientCallinClientConfigurationManager.sendRequest(), matchinggax-grpc'sGrpcClientCalls,BigtableChannelPrimer, andSessionPoolImpl.Testing
metadataDoesNotAccumulateAcrossPollsWithHeaderAttachingInterceptorverifying 26 consecutive polls against a server advertisingmaxInboundMetadataSize(1024)with a header-appendingClientInterceptorsucceed without header accumulation (1copy of each header per poll on the wire).