Skip to content

feat(collections): create from and read raw schema JSON - #598

Open
dudanogueira wants to merge 5 commits into
weaviate:mainfrom
dudanogueira:feat/collections-raw-json
Open

dudanogueira wants to merge 5 commits into
weaviate:mainfrom
dudanogueira:feat/collections-raw-json

Conversation

@dudanogueira

@dudanogueira dudanogueira commented Aug 7, 2026 •

Copy link
Copy Markdown

What

Raw-JSON variants of the collection schema calls, on the sync and async clients:

client.collections.createFromJson(json);   // POST /schema, body sent as-is
client.collections.getConfigAsJson(name);  // GET /schema/{name} -> Optional<String>
client.collections.listAsJson();           // GET /schema -> String

Changes

  • CreateCollectionRequest, GetConfigRequest and ListCollectionRequest are parameterized by payload type; a String payload/response is passed through unparsed (SimpleEndpoint.deserializeClass), so typed and raw calls share one request class.
  • The create endpoint returns Void instead of parsing the echoed config, which a raw document may not fit into CollectionConfig.
  • createFromJson reads the "class" key to return a handle; a document without one fails with IllegalArgumentException before sending (a failed future on the async client).

Breaking changes

The public _ENDPOINT fields on CreateCollectionRequest, GetConfigRequest and ListCollectionRequest are replaced by endpoint(...) factory methods.

Add a raw-JSON escape hatch to the collections API, in both directions,
on the sync and async clients:

  client.collections.createFromJson(json)     // POST /schema
  client.collections.getConfigAsJson(name)    // GET  /schema/{name}
  client.collections.listAsJson()             // GET  /schema

Neither direction maps the document onto CollectionConfig. On write the
body is forwarded byte-for-byte, so any option the server accepts works,
including ones this client version does not model yet. On read the
response body is returned untouched.

The read side matters because CollectionConfig deserialization is lossy
today. Most visibly, VectorConfig.CustomTypeAdapterFactory looks for the
bq/pq/sq/rq keys at the top of vectorIndexConfig, which is correct for
hnsw and flat but not for dynamic, where they sit one level deeper under
hnsw/flat. A collection with a dynamic index and RQ enabled reports
quantization = null. Property-level moduleConfig, moduleConfig entries
that are neither reranker-* nor generative-*, and the read-only
shardingConfig fields are dropped as well. getConfigAsJson gives callers
that need the complete picture -- rendering or diffing a schema -- a way
to get it without waiting for the mapping to be fixed.

createFromJson reads exactly one key, "class", which it needs to return
a collection handle. A document that is not a JSON object, or that has
no non-empty string "class", raises IllegalArgumentException before the
request is sent. getConfigAsJson validates nothing and maps 404 to
Optional.empty(), matching the typed getConfig.

@orca-security-eu orca-security-eu Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@bevzzz

bevzzz commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

I suggest a different approach, probably one that will result in better code-reuse / less duplication.

The only difference between create(CollectionConfig) and createFromJson(String) is the payload. To add "from/to JSON" to the current client it should be enough to allow GetConfigRequest, CreateCollectionRequest, and ListCollectionsRequest to accept a generic parameter saying what the request/response payload should be:

public record GetConfigRequest(String collectionName) {
  public static final <T> Endpoint<GetConfigRequest, Optional<T>> endpoint(Class<T> cls) {
    return OptionalEndpoint
        .<GetConfigRequest, T>noBodyOptional(
            request -> "GET",
            request -> "/schema/" + request.collectionName,
            request -> Collections.emptyMap(),
            (statusCode, response) -> JSON.deserialize(response, cls));
  }
}

This way the same GetConfigRequest can be used to return:

  • CollectionConfig, if used like so GetConfigRequest.endpoint(CollectionConfig.class)
  • Raw JSON string, if used like so GetConfigRequest.endoint(String.class)

Same for list and create endpoints.

Would you like to take a stab at that? I'm happy to take this over, this seems like a small enough change.


Wrt to the "Not addressed here" section: if you find any bugs while working on a PR, please open a ticket for those. A PR description is not the right place to document bugs.

dudanogueira and others added 2 commits August 10, 2026 16:24
Address review feedback on weaviate#598: the raw-JSON escape hatch no longer
needs request classes of its own. The only thing that differed between
create(CollectionConfig) and createFromJson(String) was the payload, so
the existing request records now take the type as a parameter:

  GetConfigRequest.endpoint(CollectionConfig.class | String.class)
  ListCollectionRequest.endpoint(ListCollectionResponse.class | String.class)
  CreateCollectionRequest<CollectionConfig | String>.endpoint()

CreateCollectionFromJsonRequest, GetConfigJsonRequest and
ListCollectionJsonRequest are removed; the public client API is
unchanged.

SimpleEndpoint.deserializeClass special-cases String.class by returning
the response body unparsed, so any endpoint parameterized by response
type gets the raw option for free. On the write side, a String payload
is forwarded byte-for-byte and anything else is serialized first.

The create endpoint now returns Void instead of parsing the echoed
configuration into CollectionConfig: both clients discarded it, and a
raw payload may describe a collection CollectionConfig cannot represent,
which would have made that parse throw.

The "class" lookup moved to CreateCollectionRequest.collectionNameFromJson,
which both clients call before sending, so an unusable document still
fails before the request leaves the process.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJ9gPAiDmKMN15VyRbw5UR
The server refuses a "dynamic" vector index unless it was started with
ASYNC_INDEXING=true, so testCreateFromJsonAndGetConfigAsJson failed with
HTTP 422 on every version in CI.

Give that test its own container with async indexing enabled rather than
flipping the flag on the shared one: async indexing makes the vectors of
freshly inserted objects searchable only eventually, and most suites rely
on them being searchable at once.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WmY5dAGWCccWDoqkKNC2JU
@dudanogueira

Copy link
Copy Markdown
Author

Thanks — took a stab at it, done in 94de2ef.

GetConfigRequest, ListCollectionRequest and CreateCollectionRequest now take the payload type as a parameter, and CreateCollectionFromJsonRequest, GetConfigJsonRequest and ListCollectionJsonRequest are gone. The public client API is unchanged.

GetConfigRequest.endpoint(CollectionConfig.class | String.class)
ListCollectionRequest.endpoint(ListCollectionResponse.class | String.class)
CreateCollectionRequest.<CollectionConfig | String>endpoint()

Rather than a per-request deserializer, SimpleEndpoint.deserializeClass special-cases String.class and returns the body unparsed, so any endpoint parameterized by response type gets the raw option for free. On the write side a String payload is forwarded byte-for-byte and anything else is serialized first.

Two things I decided along the way, shout if you'd rather have them differently:

  • The create endpoint now returns Void instead of parsing the echoed configuration into CollectionConfig. Both clients were discarding it, and a raw payload may describe a collection CollectionConfig cannot represent — that parse would have thrown.
  • The "class" lookup moved to CreateCollectionRequest.collectionNameFromJson, which both clients call before sending, so an unusable document still fails before the request leaves the process.

On the "Not addressed here" section — fair point, moved out to #606 and #607 and trimmed the description down to a reference.

@dudanogueira

Copy link
Copy Markdown
Author

@bevzzz gentle nudge on this one — the refactor you asked for went in as 94de2ef a couple of weeks ago and the two decisions I flagged are still open for you to overrule (the create endpoint returning Void rather than parsing the echo, and the "class" lookup moving into CreateCollectionRequest.collectionNameFromJson). CI is green.

No rush on the rest of my open PRs — if it helps, this is the one worth looking at first. The other six are independent single-defect fixes, but this one is the shape decision, and I would rather build on your answer here than guess.

One thing I would like your view on before proposing it separately: the same raw-JSON escape hatch is missing on the update side. WeaviateConfigClient.update only takes the typed builder, so there is still no way to send a definition the client cannot model, which is the half that runs on every configuration change rather than once at creation.

It looks like a close parallel to createFromJson under the refactored shape — UpdateCollectionRequest<T>(T updated, String collectionName), with the name coming from the caller rather than from the document. It would also remove code rather than add a parallel path: the typed request currently re-serializes the config and then patches skipDefaultQuantization back out of the original, which exists only because the typed round trip does not preserve what the server sent. A document forwarded byte-for-byte needs none of that.

Happy to add it to this PR or send it as a follow-up once this lands — whichever you prefer.

…Json; drop comments

The async client threw IllegalArgumentException synchronously; it now
returns a failed future, like the rest of the async API. Also removes
explanatory comments from the diff and trims public Javadoc.

Claude-Session: https://claude.ai/code/session_01VCkN66JyKNd2RQpFE46nCE

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Outstanding review nits concern the misleading asyncClient name for a synchronous client and the unnecessary public exposure of a package-internal parsing helper.

2 open issues
What changed in this PR

Adds raw JSON schema operations to the synchronous and asynchronous collection clients, allowing callers to use schema options not represented by CollectionConfig.

Changes:

  • Adds createFromJson, getConfigAsJson, and listAsJson, with collection-name validation before creation.
  • Shares typed and raw endpoint implementations through factories, preserving raw strings and ignoring create-response bodies.
  • Adds validation and round-trip tests, plus async-indexing support for integration-test containers.
File Description
src/​test/​java/​io/​weaviate/​client6/​v1/​api/​collections/​GetConfigRequestTest.java Tests typed/raw reads and missing-collection handling.
src/​test/​java/​io/​weaviate/​client6/​v1/​api/​collections/​CreateCollectionRequestTest.java Tests payload serialization, validation, and failed futures.
src/​main/​java/​io/​weaviate/​client6/​v1/​internal/​rest/​SimpleEndpoint.java Returns string responses without parsing.
src/​main/​java/​io/​weaviate/​client6/​v1/​api/​collections/​WeaviateCollectionsClientAsync.java Adds asynchronous raw JSON operations.
src/​main/​java/​io/​weaviate/​client6/​v1/​api/​collections/​WeaviateCollectionsClient.java Adds synchronous raw JSON operations and documentation.
src/​main/​java/​io/​weaviate/​client6/​v1/​api/​collections/​ListCollectionRequest.java Adds an endpoint factory for typed or raw list responses.
src/​main/​java/​io/​weaviate/​client6/​v1/​api/​collections/​GetConfigRequest.java Adds an endpoint factory for typed or raw optional responses.
src/​main/​java/​io/​weaviate/​client6/​v1/​api/​collections/​CreateCollectionRequest.java Supports typed/raw payloads, validates names, and ignores response bodies.
src/​it/​java/​io/​weaviate/​integration/​CollectionsITest.java Tests raw schema round trips and missing-name rejection.
src/​it/​java/​io/​weaviate/​containers/​Weaviate.java Adds an async-indexing container setting.

Review effort: Max

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants