Skip to content

perf(bigquery): pre-size InsertAll batches and response maps - #15011

Draft
lqiu96 wants to merge 1 commit into
googleapis:mainfrom
lqiu96:perf/bq-t1-10-insertall-pre-sizing
Draft

lqiu96 wants to merge 1 commit into
googleapis:mainfrom
lqiu96:perf/bq-t1-10-insertall-pre-sizing

Conversation

@lqiu96

@lqiu96 lqiu96 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

Reduces intermediate allocations and collection resizing in InsertAllRequest, BigQueryImpl.insertAll, and InsertAllResponse without changing lazy BigQueryError behavior:

  • Flat ArrayList staging (InsertAllRequest.Builder.setRows): Stages rows in an ArrayList instead of a LinkedList (which allocates an internal Node object per row before build() copies them into an ImmutableList).
  • Pre-sized insertAll loop (BigQueryImpl.insertAll): Replaces FluentIterable.from(rows).transform(...).toList() and its boolean[] wrapper with a plain for loop into new ArrayList<>(rows.size()). Because .toList() already materialized everything immediately while hiding .size(), pre-sizing avoids repeated array resizing on large batches.
  • Pre-sized InsertAllResponse builders (InsertAllResponse.toPb / fromPb): Replaces ImmutableList.copyOf(Iterables.transform(...)) in toPb() with ImmutableList.builderWithExpectedSize(insertErrors.size()), and builds the ImmutableMap directly via ImmutableMap.builderWithExpectedSize(errorsPb.size()).buildKeepingLast() in fromPb() instead of copying through a temporary HashMap, while keeping Lists.transform on each row's BigQueryError list.

Verification

  • mvn test -pl java-bigquery/google-cloud-bigquery -Dtest=InsertAllRequestTest,BigQueryImplTest -Dclirr.skip=true -Denforcer.skip=true -Dflatten.skip=true (InsertAllRequestTest: 21 passed, BigQueryImplTest: 133 passed)
  • mvn test -pl java-bigquery/google-cloud-bigquery -Dtest=InsertAllResponseTest -Dclirr.skip=true -Denforcer.skip=true -Dflatten.skip=true (4 passed)

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several performance and memory optimizations across the BigQuery client library. Key improvements include replacing lazy Guava transformations with eager materialization using pre-sized collections, converting a LinkedList to an ArrayList in InsertAllRequest to reduce per-row memory overhead, and simplifying a mutable single-element boolean array to a standard boolean variable in BigQueryImpl. I have no feedback to provide as there are no review comments.

@lqiu96
lqiu96 force-pushed the perf/bq-t1-10-insertall-pre-sizing branch from a551158 to 89636a1 Compare November 14, 2026 01:04
@lqiu96 lqiu96 changed the title perf(bigquery): pre-size InsertAll batches and eagerly materialize error lists perf(bigquery): pre-size InsertAll batches and response maps Nov 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant