feat(core): Support schema management over REST in RESTCatalog - #9673
Open
baiyangtx wants to merge 2 commits into
Open
feat(core): Support schema management over REST in RESTCatalog#9673baiyangtx wants to merge 2 commits into
baiyangtx wants to merge 2 commits into
Conversation
Add a Catalog-level API for listing schemas and expose it over the REST
protocol so that RESTCatalog-backed tables can read historical schemas
without direct filesystem access.
* Introduce `Catalog#supportsSchemaManagement` and
`Catalog#listSchemas(Identifier, SchemaFilter)` with a
`SchemaFilter` value object (all / latest / earliest / by id / by
range).
* Add `GET /v1/{prefix}/databases/{db}/tables/{obj}/schemas` with a
`ListSchemaResponse` payload; encode `SchemaFilter` as query
parameters (`latest`, `earliest`, `schemaId`, `maxSchemaId`,
`minSchemaId`).
* Implement `RESTCatalog#supportsSchemaManagement`/`listSchemas` and
map REST errors to catalog exceptions.
* Add `CatalogSchemaManager`, a `SchemaManager` that delegates to the
owning `Catalog` (analogous to `CatalogBranchManager`); reads go
through `listSchemas`, writes reuse existing
`Catalog#createTable`/`alterTable`/`rollbackSchema`.
* Wire `AbstractFileStoreTable#schemaManager` to prefer
`CatalogSchemaManager` whenever `supportsSchemaManagement()` is true.
* Extend the REST mock server and add tests covering the new filter
variants and the catalog-backed schema manager.
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Contributor
Author
|
Hi @JingsongLi , this PR adds schema management support to RESTCatalog. Could you please take a look when you have time? Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Follow-up to #9174 promised in review. This PR lets RESTCatalog-backed tables serve historical schemas over HTTP so that catalog-owning services (metastore, REST server) can hide the raw schema/ directory and enforce authorization, while filesystem-backed catalogs remain untouched.
Linked issues
Tests
New tests in MockRESTCatalogTest cover:
mvn spotless:apply applied.
API
Catalog:
Filter is a single value object with mutually exclusive factories:
Write side reuses existing catalog APIs (createTable, alterTable, rollbackSchema); no new HTTP endpoints are added for writes.
REST protocol
Design notes
Follows the pattern of CatalogBranchManager: no RestSchemaManager, only CatalogSchemaManager (a SchemaManager that delegates to the owning Catalog via CatalogLoader). Reads flow through Catalog#listSchemas; writes flow through the existing Catalog#createTable/alterTable/rollbackSchema.
Filesystem-only operations (schemaDirectory, commit(TableSchema), deleteSchema, toSchemaPath, schemaPaths) intentionally throw UnsupportedOperationException.
New capability flag supportsSchemaManagement (distinct from supportsVersionManagement) is threaded through CatalogEnvironment and consulted in AbstractFileStoreTable#schemaManager() — when the environment has a CatalogLoader and the catalog reports true, tables build a
CatalogSchemaManager; otherwise they continue to build a FileSystemSchemaManager, so on-disk semantics are preserved.
Read side deliberately collapsed to a single listSchemas(Identifier, SchemaFilter); the server holds all interpretation logic. This avoids fanning out N REST endpoints for latest/earliest/by-id/range/list-all.
RESTCatalog#listSchemas maps REST errors to TableNotExistException / TableNoPermissionException, matching the exception contract on the rest of the catalog surface.
Public API changes