Repository navigation
feat(collections): create from and read raw schema JSON - #598
dudanogueira wants to merge 5 commits into
Conversation
Add a raw-JSON escape hatch to the collections API, in both directions,
on the sync and async clients:
client.collections.createFromJson(json) // POST /schema
client.collections.getConfigAsJson(name) // GET /schema/{name}
client.collections.listAsJson() // GET /schema
Neither direction maps the document onto CollectionConfig. On write the
body is forwarded byte-for-byte, so any option the server accepts works,
including ones this client version does not model yet. On read the
response body is returned untouched.
The read side matters because CollectionConfig deserialization is lossy
today. Most visibly, VectorConfig.CustomTypeAdapterFactory looks for the
bq/pq/sq/rq keys at the top of vectorIndexConfig, which is correct for
hnsw and flat but not for dynamic, where they sit one level deeper under
hnsw/flat. A collection with a dynamic index and RQ enabled reports
quantization = null. Property-level moduleConfig, moduleConfig entries
that are neither reranker-* nor generative-*, and the read-only
shardingConfig fields are dropped as well. getConfigAsJson gives callers
that need the complete picture -- rendering or diffing a schema -- a way
to get it without waiting for the mapping to be fixed.
createFromJson reads exactly one key, "class", which it needs to return
a collection handle. A document that is not a JSON object, or that has
no non-empty string "class", raises IllegalArgumentException before the
request is sent. getConfigAsJson validates nothing and maps 404 to
Optional.empty(), matching the typed getConfig.
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Infrastructure as Code | View in Orca | ||
| SAST | View in Orca | ||
| Secrets | View in Orca | ||
| Vulnerabilities | View in Orca |
|
I suggest a different approach, probably one that will result in better code-reuse / less duplication. The only difference between public record GetConfigRequest(String collectionName) {
public static final <T> Endpoint<GetConfigRequest, Optional<T>> endpoint(Class<T> cls) {
return OptionalEndpoint
.<GetConfigRequest, T>noBodyOptional(
request -> "GET",
request -> "/schema/" + request.collectionName,
request -> Collections.emptyMap(),
(statusCode, response) -> JSON.deserialize(response, cls));
}
}This way the same GetConfigRequest can be used to return:
Same for list and create endpoints. Would you like to take a stab at that? I'm happy to take this over, this seems like a small enough change. Wrt to the "Not addressed here" section: if you find any bugs while working on a PR, please open a ticket for those. A PR description is not the right place to document bugs. |
Address review feedback on weaviate#598: the raw-JSON escape hatch no longer needs request classes of its own. The only thing that differed between create(CollectionConfig) and createFromJson(String) was the payload, so the existing request records now take the type as a parameter: GetConfigRequest.endpoint(CollectionConfig.class | String.class) ListCollectionRequest.endpoint(ListCollectionResponse.class | String.class) CreateCollectionRequest<CollectionConfig | String>.endpoint() CreateCollectionFromJsonRequest, GetConfigJsonRequest and ListCollectionJsonRequest are removed; the public client API is unchanged. SimpleEndpoint.deserializeClass special-cases String.class by returning the response body unparsed, so any endpoint parameterized by response type gets the raw option for free. On the write side, a String payload is forwarded byte-for-byte and anything else is serialized first. The create endpoint now returns Void instead of parsing the echoed configuration into CollectionConfig: both clients discarded it, and a raw payload may describe a collection CollectionConfig cannot represent, which would have made that parse throw. The "class" lookup moved to CreateCollectionRequest.collectionNameFromJson, which both clients call before sending, so an unusable document still fails before the request leaves the process. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJ9gPAiDmKMN15VyRbw5UR
The server refuses a "dynamic" vector index unless it was started with ASYNC_INDEXING=true, so testCreateFromJsonAndGetConfigAsJson failed with HTTP 422 on every version in CI. Give that test its own container with async indexing enabled rather than flipping the flag on the shared one: async indexing makes the vectors of freshly inserted objects searchable only eventually, and most suites rely on them being searchable at once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WmY5dAGWCccWDoqkKNC2JU
|
Thanks — took a stab at it, done in 94de2ef.
GetConfigRequest.endpoint(CollectionConfig.class | String.class)
ListCollectionRequest.endpoint(ListCollectionResponse.class | String.class)
CreateCollectionRequest.<CollectionConfig | String>endpoint()Rather than a per-request deserializer, Two things I decided along the way, shout if you'd rather have them differently:
On the "Not addressed here" section — fair point, moved out to #606 and #607 and trimmed the description down to a reference. |
|
@bevzzz gentle nudge on this one — the refactor you asked for went in as No rush on the rest of my open PRs — if it helps, this is the one worth looking at first. The other six are independent single-defect fixes, but this one is the shape decision, and I would rather build on your answer here than guess. One thing I would like your view on before proposing it separately: the same raw-JSON escape hatch is missing on the update side. It looks like a close parallel to Happy to add it to this PR or send it as a follow-up once this lands — whichever you prefer. |
…Json; drop comments The async client threw IllegalArgumentException synchronously; it now returns a failed future, like the rest of the async API. Also removes explanatory comments from the diff and trims public Javadoc. Claude-Session: https://claude.ai/code/session_01VCkN66JyKNd2RQpFE46nCE
There was a problem hiding this comment.
🟡 Changes recommended
Outstanding review nits concern the misleading asyncClient name for a synchronous client and the unnecessary public exposure of a package-internal parsing helper.
2 open issues
What changed in this PR
Adds raw JSON schema operations to the synchronous and asynchronous collection clients, allowing callers to use schema options not represented by CollectionConfig.
Changes:
- Adds
createFromJson,getConfigAsJson, andlistAsJson, with collection-name validation before creation. - Shares typed and raw endpoint implementations through factories, preserving raw strings and ignoring create-response bodies.
- Adds validation and round-trip tests, plus async-indexing support for integration-test containers.
| File | Description |
|---|---|
src/test/java/io/weaviate/client6/v1/api/collections/GetConfigRequestTest.java |
Tests typed/raw reads and missing-collection handling. |
src/test/java/io/weaviate/client6/v1/api/collections/CreateCollectionRequestTest.java |
Tests payload serialization, validation, and failed futures. |
src/main/java/io/weaviate/client6/v1/internal/rest/SimpleEndpoint.java |
Returns string responses without parsing. |
src/main/java/io/weaviate/client6/v1/api/collections/WeaviateCollectionsClientAsync.java |
Adds asynchronous raw JSON operations. |
src/main/java/io/weaviate/client6/v1/api/collections/WeaviateCollectionsClient.java |
Adds synchronous raw JSON operations and documentation. |
src/main/java/io/weaviate/client6/v1/api/collections/ListCollectionRequest.java |
Adds an endpoint factory for typed or raw list responses. |
src/main/java/io/weaviate/client6/v1/api/collections/GetConfigRequest.java |
Adds an endpoint factory for typed or raw optional responses. |
src/main/java/io/weaviate/client6/v1/api/collections/CreateCollectionRequest.java |
Supports typed/raw payloads, validates names, and ignores response bodies. |
src/it/java/io/weaviate/integration/CollectionsITest.java |
Tests raw schema round trips and missing-name rejection. |
src/it/java/io/weaviate/containers/Weaviate.java |
Adds an async-indexing container setting. |
Review effort: Max
…ename asyncIndexingClient

What
Raw-JSON variants of the collection schema calls, on the sync and async clients:
Changes
CreateCollectionRequest,GetConfigRequestandListCollectionRequestare parameterized by payload type; aStringpayload/response is passed through unparsed (SimpleEndpoint.deserializeClass), so typed and raw calls share one request class.Voidinstead of parsing the echoed config, which a raw document may not fit intoCollectionConfig.createFromJsonreads the"class"key to return a handle; a document without one fails withIllegalArgumentExceptionbefore sending (a failed future on the async client).Breaking changes
The public
_ENDPOINTfields onCreateCollectionRequest,GetConfigRequestandListCollectionRequestare replaced byendpoint(...)factory methods.