Skip to content

feat(core): Support schema management over REST in RESTCatalog - #9673

Open
baiyangtx wants to merge 2 commits into
apache:masterfrom
baiyangtx:zhangyongxiang.alpha/rest-catalog-schema-api
Open

feat(core): Support schema management over REST in RESTCatalog#9673
baiyangtx wants to merge 2 commits into
apache:masterfrom
baiyangtx:zhangyongxiang.alpha/rest-catalog-schema-api

Conversation

@baiyangtx

Copy link
Copy Markdown
Contributor

Purpose

Follow-up to #9174 promised in review. This PR lets RESTCatalog-backed tables serve historical schemas over HTTP so that catalog-owning services (metastore, REST server) can hide the raw schema/ directory and enforce authorization, while filesystem-backed catalogs remain untouched.

Linked issues

Tests

  • New tests in MockRESTCatalogTest cover:

    • testSupportsSchemaManagement – flag is exposed correctly.
    • testListSchemasAll – returns every historical schema.
    • testListSchemasLatestAndEarliest – latest() / earliest() shortcuts.
    • testListSchemasById – single lookup + miss returns empty.
    • testListSchemasByRange – [minSchemaId, maxSchemaId] inclusive range.
    • testListSchemasTableNotExist – 404 mapped to TableNotExistException.
    • testCatalogSchemaManagerBackedTableUsesRest – AbstractFileStoreTable#schemaManager picks CatalogSchemaManager and rollback flows through the REST server.
  • mvn spotless:apply applied.

API

Catalog:

  default boolean supportsSchemaManagement() { return false; }
  default List<TableSchema> listSchemas(Identifier identifier, SchemaFilter filter)
          throws TableNotExistException {
      throw new UnsupportedOperationException();
  }

Filter is a single value object with mutually exclusive factories:

  SchemaFilter.all();
  SchemaFilter.latest();
  SchemaFilter.earliest();
  SchemaFilter.withId(long id);
  SchemaFilter.range(Long maxSchemaId /* inclusive */, Long minSchemaId /* inclusive */);

Write side reuses existing catalog APIs (createTable, alterTable, rollbackSchema); no new HTTP endpoints are added for writes.

REST protocol

  GET /v1/{prefix}/databases/{database}/tables/{table}/schemas
      ?latest=true
      | ?earliest=true
      | ?schemaId=<long>
      | ?maxSchemaId=<long>&minSchemaId=<long>

  ListSchemaResponse:

  {
    "schemas": [
      { "schemaId": 3, "schema": { ... }, "createdAt": 1700000000000 }
    ]
  }

  ForbiddenException -> TableNoPermissionException; NoSuchResourceException -> TableNotExistException.

Design notes

  • Follows the pattern of CatalogBranchManager: no RestSchemaManager, only CatalogSchemaManager (a SchemaManager that delegates to the owning Catalog via CatalogLoader). Reads flow through Catalog#listSchemas; writes flow through the existing Catalog#createTable/alterTable/rollbackSchema.
    Filesystem-only operations (schemaDirectory, commit(TableSchema), deleteSchema, toSchemaPath, schemaPaths) intentionally throw UnsupportedOperationException.

  • New capability flag supportsSchemaManagement (distinct from supportsVersionManagement) is threaded through CatalogEnvironment and consulted in AbstractFileStoreTable#schemaManager() — when the environment has a CatalogLoader and the catalog reports true, tables build a
    CatalogSchemaManager; otherwise they continue to build a FileSystemSchemaManager, so on-disk semantics are preserved.

  • Read side deliberately collapsed to a single listSchemas(Identifier, SchemaFilter); the server holds all interpretation logic. This avoids fanning out N REST endpoints for latest/earliest/by-id/range/list-all.

  • RESTCatalog#listSchemas maps REST errors to TableNotExistException / TableNoPermissionException, matching the exception contract on the rest of the catalog surface.

Public API changes

  • Added default methods to Catalog — no source break; both default to unsupported behavior.
  • Added a public value type org.apache.paimon.schema.SchemaFilter.
  • Added a public response type org.apache.paimon.rest.responses.ListSchemaResponse.
  • New REST route (versioned under /v1); unaffected servers stay compatible by returning 501/404 as today

zhangyongxiang.alpha and others added 2 commits September 7, 2026 15:51
Add a Catalog-level API for listing schemas and expose it over the REST
protocol so that RESTCatalog-backed tables can read historical schemas
without direct filesystem access.

* Introduce `Catalog#supportsSchemaManagement` and
  `Catalog#listSchemas(Identifier, SchemaFilter)` with a
  `SchemaFilter` value object (all / latest / earliest / by id / by
  range).
* Add `GET /v1/{prefix}/databases/{db}/tables/{obj}/schemas` with a
  `ListSchemaResponse` payload; encode `SchemaFilter` as query
  parameters (`latest`, `earliest`, `schemaId`, `maxSchemaId`,
  `minSchemaId`).
* Implement `RESTCatalog#supportsSchemaManagement`/`listSchemas` and
  map REST errors to catalog exceptions.
* Add `CatalogSchemaManager`, a `SchemaManager` that delegates to the
  owning `Catalog` (analogous to `CatalogBranchManager`); reads go
  through `listSchemas`, writes reuse existing
  `Catalog#createTable`/`alterTable`/`rollbackSchema`.
* Wire `AbstractFileStoreTable#schemaManager` to prefer
  `CatalogSchemaManager` whenever `supportsSchemaManagement()` is true.
* Extend the REST mock server and add tests covering the new filter
  variants and the catalog-backed schema manager.

Co-authored-by: TRAE CLI <traecli@bytedance.com>
@baiyangtx

Copy link
Copy Markdown
Contributor Author

Hi @JingsongLi , this PR adds schema management support to RESTCatalog. Could you please take a look when you have time? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant