[feature](s3) Support S3 Express One Zone#65504
Open
0AyanamiRei wants to merge 25 commits into
Open
Conversation
…nt-MD5 for S3 Express
### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: apache#63409\n\nProblem Summary: The initial S3 Express implementation detected directory buckets and endpoints with broad substring matches in two separate locations. This could classify ordinary bucket names or endpoint paths as S3 Express and allowed the client configuration and upload checksum behavior to diverge. Centralize the decision in one helper, require the documented --x-s3 bucket suffix or an s3express endpoint host label, and pass the resulting context into the object storage client. Add unit coverage for valid and invalid names.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: Unit Test (S3UTILTest.is_s3_express_context; ASAN UT rebuild started)\n- Behavior changed: No, except avoiding false-positive S3 Express detection\n- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
0AyanamiRei
marked this pull request as ready for review
July 13, 2026 01:38
0AyanamiRei
requested review from
Gabriel39,
gavinchou,
liaoxin01 and
morningman
as code owners
July 13, 2026 01:38
### What problem does this PR solve? Issue Number: N/A Related PR: apache#63409 Problem Summary: Document a production-ready design for S3 Express One Zone on Doris. The design audits the current PR, defines service-aware bucket capabilities, preserves ordinary S3 endpoint behavior, completes checksum and multipart flows across BE and FE, specifies unordered listing compatibility, rejects unsupported Cloud and Hadoop surfaces before I/O or persistence, and provides test, rollout, and data rollback gates. ### Release note None ### Check List (For Author) - Test: No need to test (documentation-only change; validated with git diff --cached --check and source-path consistency checks) - Behavior changed: No - Does this need documentation: No (this commit is the implementation design)
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: Add end-to-end S3 Express One Zone Directory Bucket support across native BE and FE S3 paths. Use SDK-managed session authentication and zonal endpoint routing, CRC32C multipart propagation, Directory Bucket listing semantics, active multipart cleanup, and fail-fast gates for unsupported Cloud Storage Vault and Hadoop/S3A surfaces while preserving ordinary S3-compatible behavior. ### Release note Add native S3 Express One Zone Directory Bucket support for Doris S3 file access. Cloud Storage Vault, Hadoop/S3A-backed external surfaces, and presigned URLs remain unsupported and fail fast. ### Check List (For Author) - Test: Test code added but not run per request - Behavior changed: Yes. Official AWS Directory Buckets use SDK-managed Express session authentication, CRC32C, and Directory Bucket listing semantics; unsupported surfaces fail fast. - Does this need documentation: Yes, goal.md
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: Document how the S3 Express One Zone design maps to the implemented BE, FE, Hive, Cloud boundary, checksum, multipart, listing, and fail-fast changes. Record compatibility behavior, unexecuted tests, and remaining validation gaps so the implementation status is not confused with GA acceptance. ### Release note None ### Check List (For Author) - Test: No need to test (documentation-only change) - Behavior changed: No - Does this need documentation: Yes, this commit adds the implementation summary
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: The previous S3 Express implementation expanded Directory Bucket awareness across listing, deletion, presigning, connectors, Cloud storage, checksums, and documentation. This change narrows support to official AWS endpoints and valid --x-s3 bucket names on Doris native known-object paths. AWS SDK endpoint rules provide the zonal endpoint and CreateSession authentication, Express clients use HTTPS and virtual-hosted addressing, Head/Get/range/Put and multipart create/upload/complete/abort are supported, exact S3 TVF and S3 Load paths use HeadObject, and Express uploads no longer send Content-MD5. General-purpose S3 and S3-compatible endpoints retain their existing client behavior. ### Release note Support native known-object reads and writes for Amazon S3 Express One Zone Directory Buckets. ### Check List (For Author) - Test: Not run (per user request) - Behavior changed: Yes. Official AWS Directory Buckets use SDK S3 Express routing and session authentication for native known-object operations. - Does this need documentation: Yes. The user documentation will be handled separately.
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: Align S3 Express support with the documented known-object contract. Only valid AWS Directory Buckets using a standard HTTPS regional endpoint, a matching region, and virtual-hosted-style access enable SDK endpoint rules and CreateSession. Exact S3 TVF and Broker Load paths use HeadObject and reject glob, range, query, empty-key, and directory-prefix inputs. Third-party S3-compatible endpoints retain conventional S3 authentication and checksum behavior even when a bucket name ends in --x-s3. BE known-object uploads omit Content-MD5 for Directory Buckets. Remove the prior generic Java operation and BE abort expansions. The existing SDK versions already provide the required endpoint and session APIs, so no SDK upgrade is included. ### Release note Support known-object access to Amazon S3 Express One Zone through native S3 paths. ### Check List (For Author) - Test: Not run (per user request) - Behavior changed: Yes. Valid AWS Directory Buckets use S3 Express session authentication for known-object access; unsupported listing and glob semantics are not enabled. - Does this need documentation: Yes. The user documentation is handled separately.
### What problem does this PR solve? Issue Number: None Related PR: apache#65504 Problem Summary: Synchronize the S3 Express branch with the latest master so the import-layer refactor and its unit tests use the current S3 filesystem interfaces. ### Release note None ### Check List (For Author) - Test: No need to test (base branch synchronization only) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#65504 Problem Summary: Amazon S3 Express One Zone directory buckets require directory-bucket listing rules and SDK-managed session authentication, while the existing S3 import paths used regular S3 clients. Scope the capability to one-shot INSERT SELECT from the S3 table-valued source and Broker Load WITH S3, validate native AWS directory-bucket context, select Express clients for discovery and reads, and preserve ordinary, third-party, legacy, and non-target import behavior. ### Release note Support importing from Amazon S3 Express One Zone directory buckets through one-shot INSERT SELECT S3 sources and Broker Load WITH S3. ### Check List (For Author) - Test: Unit Test - FE core targeted unit tests - FE filesystem targeted unit tests - BE targeted unit tests - Behavior changed: Yes, S3 Express reads are enabled only for the two scoped S3 import entry points. - Does this need documentation: Yes (follow-up documentation PR pending)
0AyanamiRei
requested review from
924060929,
englefly,
morrySnow and
starocean999
as code owners
July 18, 2026 12:51
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: The previous S3 Express implementation inferred directory-bucket behavior from user endpoints and changed generic S3/import abstractions beyond the intended feature. This refactor limits S3 Express selection to trusted one-shot INSERT SELECT and Broker Load read paths with an explicit AWS provider and a complete directory bucket name. Doris ignores the user endpoint for these scoped reads, derives known Regions from the bucket Zone ID, falls back to an unchecked user Region for future Zone prefixes, and lets the AWS SDK resolve the zonal endpoint and manage S3 Express session credentials. Ordinary S3 and non-target import, catalog, resource, storage, and write paths retain their existing behavior. ### Release note Add S3 Express One Zone read support for one-shot INSERT SELECT from S3 and Broker Load with S3. Provide the complete directory bucket name and AWS provider; endpoint is ignored, while region is used only as an unchecked fallback for an unknown Zone prefix. ### Check List (For Author) - Test: Unit Test - BE S3ClientFactoryTest: 12 tests passed - FE targeted S3 Express and import tests: 196 tests passed - Behavior changed: Yes. Only the scoped S3 import reads select SDK-managed S3 Express endpoint and session authentication. - Does this need documentation: Yes. The PR description documents the supported entry points and configuration contract.
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: The S3 Express statement-scope gate only admitted INSERT INTO SELECT and excluded a direct SELECT from the same S3 TVF, even though both statements share the same FE listing and BE read path. The gate also called Command.getType() for a real InsertIntoTableCommand, but Command does not implement that method. Recognize only the exact one-shot InsertIntoTableCommand and the parser's UnboundResultSink for a direct SELECT. This keeps CTAS, INSERT OVERWRITE, OUTFILE, COPY, FILE TVF, Streaming, internal commands, and specialized INSERT subclasses outside the trusted S3 Express marker boundary. ### Release note Support reading an Amazon S3 Express One Zone directory bucket with a direct SELECT from S3(...) in addition to INSERT INTO ... SELECT ... FROM S3(...) and Broker Load WITH S3. ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.common.util.S3URITest,org.apache.doris.datasource.property.storage.S3PropertiesTest,org.apache.doris.fs.FileSystemFactoryS3ExpressScopeTest,org.apache.doris.load.loadv2.BrokerLoadJobTest,org.apache.doris.nereids.trees.plans.commands.LoadCommandTest,org.apache.doris.tablefunction.ExternalFileTableValuedFunctionTest,org.apache.doris.qe.S3ExpressImportScopeTest (58 tests passed) - Behavior changed: Yes. Direct one-shot SELECT from the S3 TVF can select the scoped S3 Express read client; non-target entry points remain excluded. - Does this need documentation: Yes. The PR description documents the supported entry points and configuration contract.
### What problem does this PR solve? Issue Number: None Related PR: apache#65504 Problem Summary: S3 Express import requests whose bucket clearly uses the --x-s3 suffix could silently fall back to generic S3-compatible provider detection when the statement scope, provider, or directory bucket format was invalid. This produced misleading MinIO or ordinary S3 errors. FE list failures also lost AWS service details and incorrectly implied a specific CreateSession or ListObjectsV2 stage. Recognize S3 Express intent before fallback, validate only the supported import entry points, and preserve HTTP status, AWS error code, message, and request ID with joint operation context. ### Release note S3 Express import requests now report clear scope, provider, bucket-format, and FE service errors instead of silently falling back to ordinary S3-compatible handling. ### Check List (For Author) - Test: Unit Test - FE unit tests for S3 Express intent validation, TVF and Broker Load scope, and S3 filesystem error reporting - Behavior changed: Yes. Invalid S3 Express import requests now fail with feature-specific diagnostics. - Does this need documentation: No
Contributor
Author
|
run buildall |
### What problem does this PR solve? Issue Number: None Related PR: apache#65504 Problem Summary: FileSystemFactoryS3ExpressScopeTest used nonstandard line wrapping in the ASF license header. FE Checkstyle rejected the header and blocked the Compile, FE UT, and Performance pipelines before fe-core compilation. Align the file with the repository standard ASF header. ### Release note None ### Check List (For Author) - Test: Unit Test - Debug FE build: ./build.sh --fe -j48 - FE UT: ./run-fe-ut.sh --run org.apache.doris.fs.FileSystemFactoryS3ExpressScopeTest - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 29372 ms |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29891 ms |
Contributor
TPC-DS: Total hot run time: 177477 ms |
Contributor
ClickBench: Total hot run time: 25.06 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29541 ms |
Contributor
TPC-DS: Total hot run time: 177002 ms |
Contributor
ClickBench: Total hot run time: 24.83 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
1 similar comment
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve? Issue Number: None Related PR: apache#65504 Problem Summary: The S3 Express One Zone import change had unit coverage but no end-to-end regression suite for a real AWS directory bucket. Add an explicitly gated manual suite covering canonical Object URLs, s3:// URIs, S3 TVF, INSERT INTO SELECT, Broker Load, CSV, Parquet, ORC, multi-path and glob discovery, and ListObjectsV2 continuation over 1001 objects. Add a fixture preparation script that only uploads to a new prefix in an existing bucket and never creates or deletes storage. ### Release note None ### Check List (For Author) - Test: Manual test preparation - Groovy suite compiled with the regression framework Groovy compiler - bash -n and a stubbed fixture-script smoke test passed - Live AWS regression is pending and must generate the .out file with -genOut - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#65504 Problem Summary: Directory bucket ListObjectsV2 responses are not lexicographically ordered and do not support key-based StartAfter pagination. Doris consumed these pages as though they had general-purpose bucket ordering, which could break deterministic glob results and streaming cursors. Broker Load also created a job before exercising the s3express:CreateSession path. Exhaust directory-bucket pages, sort raw keys globally in UTF-8 binary order, and apply cursors and limits on the client. Validate directory-list request constraints, perform a bounded Express read check before job submission, reuse one filesystem across pending paths, and consolidate duplicated bucket, provider, prefix, and client-selection logic. ### Release note S3 Express directory-bucket imports now return deterministic file ordering and validate read access before Broker Load job creation. ### Check List (For Author) - Test: Unit Test (coverage added and updated; execution intentionally skipped at the user request) - Behavior changed: Yes (directory listings are globally ordered and Broker Load validates S3 Express read access before submission) - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: Doris needs to import existing objects from S3 Express directory buckets without extending support to uploads. The previous implementation added a separate access preflight and globally sorted unordered directory-bucket listings, while client selection was not sufficiently isolated from non-read paths. This refactor confines SDK-managed S3 Express clients to file discovery and reader paths, preserves service list order with opaque continuation tokens, rejects key-based cursors and FE mutations, and keeps ordinary S3 and S3-compatible providers on standard clients. ### Release note S3 TVF, INSERT SELECT, and Broker Load can import existing objects from AWS S3 Express One Zone directory buckets. Directory-bucket listings preserve service order; key-based incremental cursors, listing limits, and write operations are unsupported. ### Check List (For Author) - Test: No need to test (per request; implementation received static FE/BE review) - Behavior changed: Yes (S3 Express is restricted to import reads and directory-bucket list order is unspecified) - Does this need documentation: Yes (follow-up documentation should cover URI forms, ReadOnly IAM SessionMode, and unordered listing semantics)
0AyanamiRei
changed the title
### What problem does this PR solve? Issue Number: None Related PR: apache#65504 Problem Summary: The S3 Express import path accumulated redundant static and instance classification APIs and nested configuration wrappers. Consolidate provider gating, directory-bucket parsing, Object URL validation, Region derivation, and regional endpoint normalization into one internal path while preserving S3 TVF and Broker Load behavior. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.datasource.property.storage.S3PropertiesTest,org.apache.doris.nereids.trees.plans.commands.LoadCommandTest - ./run-fe-ut.sh --run org.apache.doris.datasource.property.storage.S3PropertiesTest - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 29718 ms |
Contributor
TPC-DS: Total hot run time: 177879 ms |
Contributor
ClickBench: Total hot run time: 25.12 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #63409
Problem Summary:
Doris import paths need to read existing Amazon S3 Express One Zone directory buckets with AWS SDK-managed session authentication and zonal endpoint resolution.
This PR supports S3 Express through the existing Doris S3 import entry points:
SELECT ... FROM S3(...).INSERT INTO ... SELECT ... FROM S3(...).WITH S3.No new storage object or user-managed short-term credential is introduced. Users keep configuring the long-term AWS credential path already supported by Doris, and the AWS SDK creates and refreshes the S3 Express session credentials.
Supported URI forms
S3 Express is selected when
s3.provider(orprovider) is explicitlyAWSand the request uses a complete directory-bucket name,<bucket-base>--<zone-id>--x-s3.The supported input forms are:
A canonical S3 Express Object URL:
For S3 TVF, Doris derives
s3.regionfrom this URL when it is omitted. Doris normalizes the URL tos3://<directory-bucket>/<key>before file discovery and execution.The existing S3 URI form:
This form requires
s3.regionas before.Broker Load still requires a shared
s3.regioninWITH S3, because itsBrokerDescis created before theDATA INFILEpaths and can be shared by multiple paths. Multiple Object URLs in the same Region are supported; a URL whose Region conflicts with the shared configuration is rejected.For a complete Object URL, Doris validates HTTPS, the canonical virtual-hosted endpoint form, the directory-bucket name, matching bucket/endpoint Zone IDs, matching URL/configured Regions, and
use_path_style=false. Canonical non-dualstack Object URLs are supported; dualstack URLs are rejected because the dualstack hostname choice is not propagated after URI normalization.s3.endpointmay be omitted. The Express clients ignore endpoint overrides and let the AWS SDK derive the zonal data endpoint from the complete directory-bucket name, so an endpoint retained for ordinary S3 configuration does not conflict with Express imports.Implementation
S3_EXPRESS_IMPORT_READproperty. S3 TVFSELECTandINSERT INTO SELECTuse the same construction path.ListObjectsV2,HeadObject, andGetObject) select the SDK-managed Express client for AWS directory buckets.StartAfterrequests.BrokerDescand replay path; complete Object URLs are normalized through the same typed S3 property path.Release note
Support importing existing Amazon S3 Express One Zone objects through S3 TVF queries,
INSERT INTO ... SELECT ... FROM S3(...), and Broker LoadWITH S3, using either a canonical S3 Express Object URL or a directory-buckets3://URI.Check List (For Author)
Test
aws_s3_express_p0/test_s3_express_import, gated byenableS3ExpressOneZoneTest, for canonical Object URLs,s3://URIs, S3 TVF,INSERT INTO SELECT, Broker Load, CSV, Parquet, ORC, multi-path/glob discovery, and 1001-object pagination..outmust be generated on a real AWS directory bucket with-genOut, then verified by a second run without-genOut../run-fe-ut.sh --run org.apache.doris.datasource.property.storage.S3PropertiesTest,org.apache.doris.tablefunction.ExternalFileTableValuedFunctionTest,org.apache.doris.nereids.trees.plans.commands.LoadCommandTest(37 tests passed)./run-fe-ut.sh --run org.apache.doris.tablefunction.ExternalFileTableValuedFunctionTest(4 tests passed after the final file-list path assertion)CreateSession -> List/Getintegration test has not run yet; Regression and Manual remain unchecked until the AWS run passes and the generated.outis committed.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)