HIVE-29777: Iceberg: Add GCS, ADLS, and OSS Hadoop mapping for REST vended credentials - #6655
Open
difin wants to merge 1 commit into
Open
HIVE-29777: Iceberg: Add GCS, ADLS, and OSS Hadoop mapping for REST vended credentials#6655difin wants to merge 1 commit into
difin wants to merge 1 commit into
Conversation
…ended credentials
difin
force-pushed
the
gcs_adls_oss_mappings_for_vended_creds
branch
from
July 28, 2026 20:51
166f643 to
254c757
Compare
difin
changed the title
asf-ci-hive
added
tests failed
tests pending
and removed
tests pending
tests failed
labels
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR extends Iceberg REST vended credential propagation beyond S3 to GCS, ADLS, and OSS. It refactors
IcebergVendedCredentialUtilinto provider-specific Hadoop mappers that translate IcebergStorageCredentialconfig keys into the correctfs.gs.*,fs.azure.*, andfs.oss.*properties. New GCS/ADLS/OSS secret keys are added tohive.conf.hidden.listso they travel via the secure Credentials channel rather than plain job properties. Unit tests cover the new mappings and round-trip behavior. Integration tests are left for follow-up work.Why are the changes needed?
The original vended-credential work only propagated Hadoop mappings for S3. Iceberg REST catalogs can vend credentials for GCS, ADLS, and OSS as well, but without equivalent mapping those jobs cannot configure the corresponding Hadoop FileSystem connectors on Tez/LLAP execution paths. This change closes that gap so multi-cloud Iceberg deployments can use vended credentials consistently across providers.
Does this PR introduce any user-facing change?
Yes, for deployments using Iceberg REST catalogs with vended credentials on GCS, ADLS, or OSS. Those workloads can now propagate storage credentials to Hadoop FileSystem paths on Tez/LLAP, which was not supported before. Users on S3-only setups or without vended credentials should see no change in behavior.
How was this patch tested?
Unit tests in
TestIcebergVendedCredentialUtilwere expanded to cover GCS, ADLS, and OSS Hadoop property mapping, secret routing intojobSecrets, mapper selection by URI scheme and config-key fallback, and Credentials-channel round-trip restore. Existing S3 coverage was kept and refactored to use the new mapper layout. No integration coverage was added for GCS/ADLS/OSS in this PR; that is planned as follow-up work.