Skip to content

fix(bigtable): copy Metadata per GetClientConfiguration call - #14950

Merged
sushanb merged 1 commit into
googleapis:mainfrom
sushanb:fix/bigtable-config-manager-metadata-copy
Nov 9, 2026
Merged

sushanb merged 1 commit into
googleapis:mainfrom
sushanb:fix/bigtable-config-manager-metadata-copy

Conversation

@sushanb

@sushanb sushanb commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

ClientConfigurationManager constructs a single io.grpc.Metadata instance in its constructor and passes that same mutable reference to ClientCall.start(listener, metadata) on every GetClientConfiguration poll and retry. In grpc-java, ClientCall.start() takes ownership of the Metadata instance, and channel ClientInterceptors wrap realChannel above RetriableStream and mutate headers in place before any defensive copy is made. Whenever a channel ClientInterceptor (for example MetadataUtils.newAttachHeadersInterceptor or a tracing interceptor calling headers.put(...) in start()) appends per-RPC headers without discardAll, duplicate header entries accumulate on ClientConfigurationManager.metadata every 5 minutes until outbound header encoding exceeds the server's HTTP/2 SETTINGS_MAX_HEADER_LIST_SIZE (64 KiB), causing Netty's HpackEncoder to fail every subsequent poll locally with Status.INTERNAL: http2 exception (HeaderListSizeException) — surviving replaceChannel() until JVM restart.

Fix

  • Copy this.metadata into a fresh Metadata instance per ClientCall in ClientConfigurationManager.sendRequest(), matching gax-grpc's GrpcClientCalls, BigtableChannelPrimer, and SessionPoolImpl.

Testing

  • Added metadataDoesNotAccumulateAcrossPollsWithHeaderAttachingInterceptor verifying 26 consecutive polls against a server advertising maxInboundMetadataSize(1024) with a header-appending ClientInterceptor succeed without header accumulation (1 copy of each header per poll on the wire).

@sushanb
sushanb requested review from a team as code owners November 9, 2026 17:02

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves the ClientConfigurationManager by preventing metadata accumulation across polls, avoiding retry amplification during background polls when falling back to the default configuration, and introducing a mockable Clock for better testability. It also adds comprehensive unit tests to verify these behaviors. The reviewer's feedback highlights an issue in the logging implementation where concatenating a Throwable into a log message discards the stack trace and suggests a Java 8-compatible logging pattern to preserve the stack trace and avoid unnecessary string concatenation.

Comment on lines +616 to +624
if (willRetry) {
logger.log(
Level.FINE,
() ->
"Failed to fetch client config (attempt "
+ (attempt + 1)
+ "), retrying: "
+ throwable);
} else if (!isInitialPoll && !shouldReset) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Concatenating the throwable into the log message string only prints its toString() representation, which discards the stack trace. For Level.FINE logs, having the full stack trace of the failed RPC is extremely valuable for debugging network or gRPC issues. Additionally, the lambda-based logging overload log(Level, Throwable, Supplier) is only available in Java 9+. If this library targets Java 8 compatibility, using standard log(Level, String, Throwable) wrapped in logger.isLoggable(Level.FINE) is safer and avoids string concatenation overhead when FINE logging is disabled.

Suggested change
if (willRetry) {
logger.log(
Level.FINE,
() ->
"Failed to fetch client config (attempt "
+ (attempt + 1)
+ "), retrying: "
+ throwable);
} else if (!isInitialPoll && !shouldReset) {
if (willRetry) {
if (logger.isLoggable(Level.FINE)) {
logger.log(
Level.FINE,
"Failed to fetch client config (attempt " + (attempt + 1) + "), retrying",
throwable);
}
} else if (!isInitialPoll && !shouldReset) {

ClientConfigurationManager previously passed the same mutable Metadata instance to ClientCall.start() on every GetClientConfiguration poll and retry. Because ClientCall.start() takes ownership of the Metadata passed to it and channel ClientInterceptors may mutate it in place, per-RPC headers attached by interceptors accumulated across polls until outbound header encoding exceeded HTTP/2 SETTINGS_MAX_HEADER_LIST_SIZE.

Copy the base Metadata into a fresh Metadata instance per ClientCall in sendRequest(), matching GrpcClientCalls, BigtableChannelPrimer, and SessionPoolImpl.
@sushanb
sushanb force-pushed the fix/bigtable-config-manager-metadata-copy branch from 8dfb57d to 2456fe2 Compare November 9, 2026 19:44
@sushanb sushanb changed the title fix(bigtable): copy Metadata per GetClientConfiguration call and avoid background retry amplification fix(bigtable): copy Metadata per GetClientConfiguration call Nov 9, 2026
@sushanb
sushanb merged commit 278fe2f into googleapis:main Nov 9, 2026
208 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants