Repository navigation
fix(collections): nest the quantizer inside a dynamic index - #613
Open
dudanogueira wants to merge 6 commits into
Open
dudanogueira wants to merge 6 commits into
dudanogueira wants to merge 6 commits into
Conversation
Weaviate parses quantizer settings out of the config map by exact key,
and returns them under the same names. The client used snake_case for
half of them, so rescoreLimit, trainingLimit and bitCompression were
dropped on write and came back null on read -- on every index type. A
user who set a rescore limit never set one.
rescore_limit -> rescoreLimit (BQ, SQ, RQ)
training_limit -> trainingLimit (PQ, SQ)
bit_compression -> bitCompression (PQ)
PQ's encoder needed a shape change rather than a rename: the server
nests it as encoder: {type, distribution} while the client had two flat
components. PQ.encoderType() and PQ.encoderDistribution() are kept as
derived accessors and the builder is unchanged, so only the canonical
constructor differs.
No alternate names: the snake_case spellings were never valid on the
wire, so no stored config uses them.
Closes #611
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WmY5dAGWCccWDoqkKNC2JU
A dynamic index is {distance, threshold, hnsw: {...}, flat: {...}} and
each sub-index carries its own quantizer, but the client put the
quantizer beside them and looked for it there on the way back. So a
dynamic index created through this client was silently unquantized, and
one quantized by other means read as quantization() == null.
VectorConfig has a single quantization slot, so a dynamic index is read
from hnsw when it has a quantizer and from flat otherwise, and always
written to hnsw -- the only sub-index accepting every quantizer type,
flat being limited to bq. Giving hnsw and flat different quantizers
stays inexpressible; that needs per-index slots on Hnsw and Flat.
The same nesting applies to the skipDefaultQuantization flag that the
update request has to preserve, so both call the shared QuantizerJson.
Also on Dynamic: "distance" is read and written (the server keeps one at
the dynamic level), and the response parser no longer assumes
"threshold" is present -- a dropped index has neither.
Closes #606
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WmY5dAGWCccWDoqkKNC2JU
![orca-security-eu[bot]](https://avatars.githubusercontent.com/in/276244?s=60&v=4){kind=small}
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
 Passed |
Secrets |  0  0  0  0 |
View in Orca |
There was a problem hiding this comment.
Pull request overview
This PR fixes quantization JSON mapping for dynamic vector indexes by correctly nesting quantizer settings under the dynamic sub-index (hnsw/flat) on both read and write paths, and aligns quantizer field names with what the Weaviate server actually parses. It also adds targeted unit/integration tests and a dedicated async-indexing IT container path required for dynamic indexes.
Changes:
- Fix dynamic-index quantizer placement/parsing by selecting the correct nested host object (
hnsw/flat) and preservingskipDefaultQuantizationin updates. - Add dynamic index support for
distance(read/write) and makethresholdparsing tolerant of missing fields in responses. - Update PQ/SQ/RQ/BQ quantizer JSON field names/shape to match server expectations and expand unit/integration test coverage.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/java/io/weaviate/client6/v1/internal/json/JSONTest.java | Adds/updates JSON round-trip fixtures for dynamic quantizer nesting and quantizer field name/shape changes. |
| src/test/java/io/weaviate/client6/v1/api/collections/DynamicQuantizationTest.java | New unit tests for dynamic quantizer read behavior across server-returned shapes. |
| src/main/java/io/weaviate/client6/v1/internal/json/QuantizerJson.java | New helper to locate the correct quantizer “host” object for dynamic vs non-dynamic indexes. |
| src/main/java/io/weaviate/client6/v1/api/collections/vectorindex/Dynamic.java | Adds distance, makes threshold optional on read, and updates dynamic index JSON adapter. |
| src/main/java/io/weaviate/client6/v1/api/collections/VectorConfig.java | Uses QuantizerJson to read/write quantizers at the correct depth for dynamic indexes. |
| src/main/java/io/weaviate/client6/v1/api/collections/quantizers/SQ.java | Renames SQ JSON fields to server-expected camelCase. |
| src/main/java/io/weaviate/client6/v1/api/collections/quantizers/RQ.java | Renames RQ JSON fields to server-expected camelCase. |
| src/main/java/io/weaviate/client6/v1/api/collections/quantizers/PQ.java | Nests PQ encoder settings under encoder and renames fields to server-expected camelCase. |
| src/main/java/io/weaviate/client6/v1/api/collections/quantizers/BQ.java | Renames BQ JSON fields to server-expected camelCase. |
| src/main/java/io/weaviate/client6/v1/api/collections/config/UpdateCollectionRequest.java | Writes skipDefaultQuantization at the same nested level the server reads for dynamic indexes. |
| src/it/java/io/weaviate/integration/CollectionsITest.java | Adds IT coverage for dynamic-index quantization round-trip and quantizer settings round-trip. |
| src/it/java/io/weaviate/containers/Weaviate.java | Adds builder support for ASYNC_INDEXING to run dynamic-index integration tests. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…te to hnsw The server returns every quantizer block under hnsw with enabled:false, so presence alone made hnsw win and hid an enabled quantizer on flat. On write, a dynamic index without hnsw put the quantizer under flat, which only accepts bq. Claude-Session: https://claude.ai/code/session_01VCkN66JyKNd2RQpFE46nCE
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Passed |
Infrastructure as Code | 0 0 0 0 |
View in Orca |
| Passed |
SAST | 0 0 0 0 |
View in Orca |
| Passed |
Secrets | 0 0 0 0 |
View in Orca |
| Passed |
Vulnerabilities | 0 0 0 0 |
View in Orca |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A dynamic index keeps its quantizer inside
hnsw/flat. The client wrote it beside them, where the server ignores it, and only looked at the top level when reading, soquantization()was alwaysnull. The dynamic-leveldistancewas also dropped, and a missingthresholdcaused an NPE on read.Fix
skipDefaultQuantizationon update) underhnsw. Read it fromhnswif a quantizer is enabled there, otherwise fromflat.distanceonDynamic; treatthresholdas optional.DynamicQuantizationTest, aJSONTestrow, andCollectionsITest.test_dynamicIndexQuantizationRoundTrip, which runs on its own container withASYNC_INDEXING=true.Breaking changes
Dynamic's canonical constructor gains adistancecomponent. The builder and accessors are unchanged.Closes #606