normalize BigQuery CDC

[aIbrahiim]

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[github-actions]

Assigning reviewers:

R: @shunping for label python.
R: @chamikaramj for label yaml.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[aIbrahiim]

Hi @Abacn, I added an extended IT for WriteToBigQueryCDC under extended_tests/databases/bigquery_cdc.yaml but on Yaml_Xlang_Direct PostCommit (databases) it fails with:

PERMISSION_DENIED: Permission TABLES_UPDATE_DATA denied on the temp yaml_bq_it_* table

As normal WriteToBigQuery inserts work fine with this SA but CDC UPSERT via Storage Write API needs tables.updateData and CDC is already covered in xlang_bigqueryio_it_test and the Java SchemaTransform unit tests

Could you please advise how we should handle this?

[Abacn]

Did some search, it appears we can explicitly Include the Service Account in Dataset Access on Creation (Programmatic Fix):

dataset = bigquery.Dataset(dataset_ref)
entries = list(client.get_dataset(dataset_ref).access_entries)
entries.append(
    bigquery.AccessEntry(
        role="roles/bigquery.dataEditor",
        entity_type="userByEmail",
        entity_id="<SERVICE_ACCOUNT_EMAIL>",
    )
)
dataset.access_entries = entries
client.create_dataset(dataset)

to

beam/sdks/python/apache_beam/io/gcp/bigquery_tools.py

Line 850 in 28279d2

def get_or_create_dataset(

we may want to make permission settings configurable and pass a parameter here

beam/sdks/python/apache_beam/yaml/integration_tests.py

Line 246 in 9863d92

bigquery_client.get_or_create_dataset(project, dataset_id)

[aIbrahiim]

@Abacn, I implemented the dataset ACL approach and it passed from temp_bigquery_table, cdc IT appends roles/bigquery.dataEditor for beam-github-actions@apache-beam-testing.iam.gserviceaccount.com after create

but Yaml_Xlang_Direct databases still fails with TABLES_UPDATE_DATA on storage write cdc and plain WriteToBigQuery is fine

Should I look into the Yaml Direct SA / project IAM next, or drop the CDC YAML IT and rely on xlang_bigqueryio_it_test and the Java unit tests for coverage?

[Abacn]

Still strange, the service account already has BigQuery Data Editor role, which has bigquery.tables.updateData permission. YAML pipeline just expands to a xlang Beam Python pipeline, should be indifferent with xlang_bigqueryio_it_test. Can we make the yaml test have the same setting as xlang_bigqueryio_it_test?