diff --git a/packages/zarr-metadata/README.md b/packages/zarr-metadata/README.md index 6b6b172aec..122d534a44 100644 --- a/packages/zarr-metadata/README.md +++ b/packages/zarr-metadata/README.md @@ -14,7 +14,7 @@ Two layers and an optional integration: specifications, plus types for [`zarr-extensions`](https://github.com/zarr-developers/zarr-extensions/) and a few widely-used-but-unspecified entities (e.g. consolidated metadata). - **Document models** (`zarr_metadata.model`): canonical frozen-dataclass - models of whole metadata documents, with structural validators, loc-aware + models of whole metadata documents, with validators, loc-aware parsers, and store-key (de)serialization. A document produced by `to_json` shares no mutable state with the model that produced it. - **Optional Pydantic integration** (`zarr_metadata.pydantic`, requires @@ -57,10 +57,16 @@ members that the strict model parser rejects. The model validators enforce the declared document structure and a small set of context-free consistency rules, including fixed format literals, finite JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one -`dimension_names` entry per array dimension. They do not interpret extension -names or configurations, resolve codec pipelines, or decide whether a data -type, chunk grid, codec, or storage transformer is supported. Those decisions -belong to consumer implementations. +`dimension_names` entry per array dimension. In a v3 document they also read +each extension point -- the data type, chunk grid, chunk key encoding, each +codec and each storage transformer -- through the definition that claims its +name in a scope, `CORE_AND_EXTENSIONS` unless a `context` is passed: a +configuration its definition refuses is refused, and a key it does not +declare is reported as `unknown_key`. A name nothing in the scope claims is +left unjudged, and whether to support it is the consumer's decision. The +validators do not judge fields against each other: a fill value against its +data type, a codec against the array it is handed, a chunk grid against the +shape. The Pydantic integration's generated JSON Schemas express independently checkable document structure and field constraints, but they are not a @@ -75,7 +81,7 @@ versus `shape`. Consumers should run the model parser after schema validation. At minimum, this library supports what Zarr-Python needs: the complete Zarr v2 and v3 specs, consolidated metadata, and a subset of the metadata defined in `zarr-extensions`. We are generally open to contributions that -add types, models, or structural validation for Zarr metadata with a +add types, models, or validation for Zarr metadata with a published spec. Runtime array behavior is out of scope: nothing here encodes or decodes diff --git a/packages/zarr-metadata/changes/4433.feature.md b/packages/zarr-metadata/changes/4434.feature.1.md similarity index 100% rename from packages/zarr-metadata/changes/4433.feature.md rename to packages/zarr-metadata/changes/4434.feature.1.md diff --git a/packages/zarr-metadata/changes/4436.feature.md b/packages/zarr-metadata/changes/4436.feature.md new file mode 100644 index 0000000000..05defe18e6 --- /dev/null +++ b/packages/zarr-metadata/changes/4436.feature.md @@ -0,0 +1,13 @@ +**Breaking:** the v3 validators read each extension point through its +definition: `validate_array_metadata_v3` judges the data type, the chunk +grid, the chunk key encoding, each codec and each storage transformer +against the definition that claims its name in a scope, +`CORE_AND_EXTENSIONS` unless a `context` is passed, so a gzip `level` of +99, or a key a codec does not declare, is a problem located in the +configuration, and a document holding one, accepted before, is refused. +So do the `is_*` and `parse_*` beside it and the v3 group validators, +with each array an inline `consolidated_metadata` holds, and the v3 +model classes' `from_json`, `from_key_value` and `to_key_value` take the +same `context`, so a model read in a scope is written in it. The +pydantic types read in `CORE_AND_EXTENSIONS`. A name nothing in scope +claims is left unjudged. diff --git a/packages/zarr-metadata/docs/api/index.md b/packages/zarr-metadata/docs/api/index.md index d52da16fa4..95e31491f5 100644 --- a/packages/zarr-metadata/docs/api/index.md +++ b/packages/zarr-metadata/docs/api/index.md @@ -7,7 +7,7 @@ title: API reference The package is organized to mirror the structure of the Zarr specifications: - [`zarr_metadata.model`](model.md) — frozen-dataclass document models, - structural validators, loc-aware parsers, and the `UNSET` sentinel + validators, loc-aware parsers, and the `UNSET` sentinel - [`zarr_metadata.pydantic`](pydantic.md) — optional Pydantic field types over the models - [`zarr_metadata.typed_json`](typed_json.md) — `check`, which type-checks diff --git a/packages/zarr-metadata/docs/index.md b/packages/zarr-metadata/docs/index.md index 0163dda5c1..1e0346393f 100644 --- a/packages/zarr-metadata/docs/index.md +++ b/packages/zarr-metadata/docs/index.md @@ -28,7 +28,7 @@ closely model the content of the Zarr specifications, such as: [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) and a few widely-used-but-unspecified entities (e.g. consolidated metadata). - **Document models** ([`zarr_metadata.model`](api/model.md)): canonical - frozen-dataclass models of whole metadata documents, with structural + frozen-dataclass models of whole metadata documents, with validators, loc-aware parsers, and store-key (de)serialization. A document produced by `to_json` shares no mutable state with the model that produced it. @@ -72,17 +72,23 @@ members that the strict model parser rejects. The model validators enforce the declared document structure and a small set of context-free consistency rules, including fixed format literals, finite JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one -`dimension_names` entry per array dimension. They do not interpret extension -names or configurations, resolve codec pipelines, or decide whether a data -type, chunk grid, codec, or storage transformer is supported. Those decisions -belong to consumer implementations. +`dimension_names` entry per array dimension. In a v3 document they also read +each extension point -- the data type, chunk grid, chunk key encoding, each +codec and each storage transformer -- through the definition that claims its +name in a scope, `CORE_AND_EXTENSIONS` unless a `context` is passed: a +configuration its definition refuses is refused, and a key it does not +declare is reported as `unknown_key`. A name nothing in the scope claims is +left unjudged, and whether to support it is the consumer's decision. The +validators do not judge fields against each other: a fill value against its +data type, a codec against the array it is handed, a chunk grid against the +shape. ## Scope At minimum, this library supports what Zarr-Python needs: the complete Zarr v2 and v3 specs, consolidated metadata, and a subset of the metadata defined in `zarr-extensions`. We are generally open to contributions that -add types, models, or structural validation for Zarr metadata with a +add types, models, or validation for Zarr metadata with a published spec. Runtime array behavior is out of scope: nothing here encodes or decodes diff --git a/packages/zarr-metadata/src/zarr_metadata/model/__init__.py b/packages/zarr-metadata/src/zarr_metadata/model/__init__.py index b7c62205ff..2c75d65b5c 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/__init__.py @@ -1,14 +1,17 @@ """In-memory models for Zarr metadata documents. Models are frozen dataclasses that hold a canonical, semantically lossless -representation of the JSON documents; they never interpret extension points -(codecs, chunk grids, data types). Validators check JSON structure, not domain validity. -Each document concept gets a `validate_*` function returning every problem -found (a `list[ValidationProblem]`, each with a machine-readable `kind`), an -`is_*` type guard, and a `parse_*` function that narrows or raises -`MetadataValidationError`. Model `from_json` / `from_key_value` constructors -raise `MetadataValidationError` for every ingestion failure, including -missing store keys and undecodable bytes. +representation of the JSON documents. Validators check a document's JSON +structure and, in a v3 document, read each extension point (codecs, chunk +grids, data types, ...) through the definition that claims its name in a +scope, `CORE_AND_EXTENSIONS` unless a `context` is passed; they do not judge +fields against each other. Each document concept gets a `validate_*` +function returning every problem found (a tuple of `ValidationProblem`, each +with a machine-readable `kind`), an `is_*` type guard, and a `parse_*` +function that narrows or raises `MetadataValidationError`. Model +`from_json` / `from_key_value` constructors raise `MetadataValidationError` +for every ingestion failure, including missing store keys and undecodable +bytes, and the v3 ones take the same `context`. """ from zarr_metadata._json import ( diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_array.py b/packages/zarr-metadata/src/zarr_metadata/model/_array.py index b899937bd9..cdf5a684c4 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_array.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_array.py @@ -26,6 +26,7 @@ from zarr_metadata.v2.array import ZARR_V2_ARRAY_METADATA_STORE_KEY from zarr_metadata.v2.attributes import ZARR_V2_ATTRIBUTES_STORE_KEY from zarr_metadata.v3._common import parse_metadata_field_v3 +from zarr_metadata.v3._registry import CORE_AND_EXTENSIONS from zarr_metadata.v3.array import ZARR_V3_ARRAY_METADATA_STORE_KEY if TYPE_CHECKING: @@ -40,6 +41,7 @@ from zarr_metadata.v2.attributes import ZarrV2AttributesStoreKey from zarr_metadata.v2.codec import ZarrV2CodecMetadata from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON + from zarr_metadata.v3._registry import Context from zarr_metadata.v3.array import ( ZarrV3ArrayMetadataJSON, ZarrV3ArrayMetadataStoreKey, @@ -158,8 +160,11 @@ class ZarrV3ArrayMetadata: content for an array. Extension points (`data_type`, `chunk_grid`, `chunk_key_encoding`, `codecs`, `storage_transformers`) are held as `ZarrV3MetadataField` values (currently `ZarrV3NamedConfig` name, - configuration, and obligation records) and are never interpreted; - `fill_value` is held verbatim in its JSON form. Equivalent extension + configuration, and obligation records). `from_json` and + `from_key_value` read each through the definition that claims its name + in a scope -- `CORE_AND_EXTENSIONS` unless a `context` is passed -- and + the model holds what they read as written; `fill_value` is held + verbatim in its JSON form. Equivalent extension spellings normalize to shorthand strings when configuration is empty and understanding is required. """ @@ -276,9 +281,11 @@ def to_json(self) -> ZarrV3ArrayMetadataJSON: return out @classmethod - def from_json(cls, data: object) -> ZarrV3ArrayMetadata: + def from_json( + cls, data: object, *, context: Context = CORE_AND_EXTENSIONS + ) -> ZarrV3ArrayMetadata: # A read model shares no mutable state with what it read. - parsed = copy.deepcopy(parse_array_metadata_v3(data)) + parsed = copy.deepcopy(parse_array_metadata_v3(data, context=context)) extra_fields: dict[str, ZarrV3ExtensionField] = { k: v for k, v in parsed.items() if k not in ARRAY_METADATA_STANDARD_KEYS_V3 } @@ -310,15 +317,20 @@ def must_understand_fields(self) -> dict[str, ZarrV3ExtensionField]: return must_understand_subset(self.extra_fields) @classmethod - def from_key_value(cls, mapping: Mapping[StoreKey, bytes]) -> ZarrV3ArrayMetadata: - return cls.from_json(load_store_json(mapping, ZARR_V3_ARRAY_METADATA_STORE_KEY)) + def from_key_value( + cls, mapping: Mapping[StoreKey, bytes], *, context: Context = CORE_AND_EXTENSIONS + ) -> ZarrV3ArrayMetadata: + return cls.from_json( + load_store_json(mapping, ZARR_V3_ARRAY_METADATA_STORE_KEY), context=context + ) def to_key_value( - self, *, indent: int | str | None = None + self, *, indent: int | str | None = None, context: Context = CORE_AND_EXTENSIONS ) -> Mapping[ZarrV3ArrayMetadataStoreKey, bytes]: # A model built by hand is not validated: its document is written only - # if it reads as `from_json` reads one, and every problem is raised. - document = parse_array_metadata_v3(self.to_json()) + # if it reads as `from_json` reads one in `context`, and every problem + # is raised. + document = parse_array_metadata_v3(self.to_json(), context=context) return {ZARR_V3_ARRAY_METADATA_STORE_KEY: dump_store_json(document, indent=indent)} diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_group.py b/packages/zarr-metadata/src/zarr_metadata/model/_group.py index f36bd03f47..fa56f633aa 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_group.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_group.py @@ -34,6 +34,7 @@ from zarr_metadata.v2.attributes import ZARR_V2_ATTRIBUTES_STORE_KEY from zarr_metadata.v2.consolidated import ZARR_V2_CONSOLIDATED_METADATA_STORE_KEY from zarr_metadata.v2.group import ZARR_V2_GROUP_METADATA_STORE_KEY +from zarr_metadata.v3._registry import CORE_AND_EXTENSIONS from zarr_metadata.v3.consolidated import ZARR_V3_CONSOLIDATED_METADATA_KEY from zarr_metadata.v3.group import ZARR_V3_GROUP_METADATA_STORE_KEY @@ -42,6 +43,7 @@ from zarr_metadata.v2.attributes import ZarrV2AttributesStoreKey from zarr_metadata.v2.consolidated import ZarrV2ConsolidatedMetadataStoreKey from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON, ZarrV2GroupMetadataStoreKey + from zarr_metadata.v3._registry import Context from zarr_metadata.v3.array import ZarrV3ExtensionField from zarr_metadata.v3.consolidated import ZarrV3ConsolidatedMetadataJSON from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON, ZarrV3GroupMetadataStoreKey @@ -72,8 +74,9 @@ class ZarrV3GroupMetadata: A canonical, semantically lossless representation of the `zarr.json` content for a group. The `consolidated_metadata` reference-implementation - convention is modeled as a typed field holding thin child models; every - other unknown top-level key lands in `extra_fields` verbatim. + convention is modeled as a typed field holding thin child models, each + array read in the scope the group is read in; every other unknown + top-level key lands in `extra_fields` verbatim. """ zarr_format: Literal[3] = field(default=3, init=False) @@ -135,9 +138,11 @@ def to_json(self) -> ZarrV3GroupMetadataJSON: return out @classmethod - def from_json(cls, data: object) -> ZarrV3GroupMetadata: + def from_json( + cls, data: object, *, context: Context = CORE_AND_EXTENSIONS + ) -> ZarrV3GroupMetadata: # A read model shares no mutable state with what it read. - parsed = copy.deepcopy(parse_group_metadata_v3(data)) + parsed = copy.deepcopy(parse_group_metadata_v3(data, context=context)) consolidated_raw: object = parsed.get(ZARR_V3_CONSOLIDATED_METADATA_KEY, UNSET) consolidated: ZarrV3ConsolidatedMetadata | UNSET if consolidated_raw is UNSET or consolidated_raw is None: @@ -146,7 +151,7 @@ def from_json(cls, data: object) -> ZarrV3GroupMetadata: # absence and never written back — repaired, not preserved. consolidated = UNSET else: - consolidated = ZarrV3ConsolidatedMetadata.from_json(consolidated_raw) + consolidated = ZarrV3ConsolidatedMetadata.from_json(consolidated_raw, context=context) extra_fields: dict[str, ZarrV3ExtensionField] = { k: v for k, v in parsed.items() @@ -171,15 +176,20 @@ def must_understand_fields(self) -> dict[str, ZarrV3ExtensionField]: return must_understand_subset(self.extra_fields) @classmethod - def from_key_value(cls, mapping: Mapping[StoreKey, bytes]) -> ZarrV3GroupMetadata: - return cls.from_json(load_store_json(mapping, ZARR_V3_GROUP_METADATA_STORE_KEY)) + def from_key_value( + cls, mapping: Mapping[StoreKey, bytes], *, context: Context = CORE_AND_EXTENSIONS + ) -> ZarrV3GroupMetadata: + return cls.from_json( + load_store_json(mapping, ZARR_V3_GROUP_METADATA_STORE_KEY), context=context + ) def to_key_value( - self, *, indent: int | str | None = None + self, *, indent: int | str | None = None, context: Context = CORE_AND_EXTENSIONS ) -> Mapping[ZarrV3GroupMetadataStoreKey, bytes]: # A model built by hand is not validated: its document is written only - # if it reads as `from_json` reads one, and every problem is raised. - document = parse_group_metadata_v3(self.to_json()) + # if it reads as `from_json` reads one in `context`, and every problem + # is raised. + document = parse_group_metadata_v3(self.to_json(), context=context) return {ZARR_V3_GROUP_METADATA_STORE_KEY: dump_store_json(document, indent=indent)} @@ -221,9 +231,11 @@ def to_json(self) -> ZarrV3ConsolidatedMetadataJSON: } @classmethod - def from_json(cls, data: object) -> ZarrV3ConsolidatedMetadata: + def from_json( + cls, data: object, *, context: Context = CORE_AND_EXTENSIONS + ) -> ZarrV3ConsolidatedMetadata: normalized = arrays_to_tuples(data) - problems = validate_consolidated_metadata_v3(normalized) + problems = validate_consolidated_metadata_v3(normalized, context=context) if len(problems) != 0: raise MetadataValidationError(problems) env = cast("Mapping[str, object]", normalized) @@ -231,9 +243,9 @@ def from_json(cls, data: object) -> ZarrV3ConsolidatedMetadata: for key, entry in cast("Mapping[str, object]", env["metadata"]).items(): node_type = cast("Mapping[str, object]", entry).get("node_type") if node_type == "array": - entries[key] = ZarrV3ArrayMetadata.from_json(entry) + entries[key] = ZarrV3ArrayMetadata.from_json(entry, context=context) else: - entries[key] = ZarrV3GroupMetadata.from_json(entry) + entries[key] = ZarrV3GroupMetadata.from_json(entry, context=context) return cls(metadata=entries) diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_validation.py b/packages/zarr-metadata/src/zarr_metadata/model/_validation.py index 12410814f0..ff4f9a1f76 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_validation.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_validation.py @@ -1,12 +1,18 @@ -"""Structural validation for Zarr metadata documents. - -Validators check JSON structure (key presence, value shapes, and fixed -literals like `zarr_format`), not domain validity. Each concept gets a -`validate_*` function returning every problem found, an `is_*` type guard, -and a `parse_*` function that narrows or raises `MetadataValidationError`. -The guards are `TypeGuard`s, not `TypeIs`: True narrows a value to its -document type, and False says nothing about its type, since a value can be -well typed and still not a valid document. +"""Validation for Zarr metadata documents. + +Validators check a document's JSON structure -- key presence, value +shapes, fixed literals like `zarr_format` -- and, in a v3 document, read +each extension point through the definition that claims its name in a +scope, so a configuration its definition refuses is refused here too. A +name nothing in the scope claims is left unjudged. Rules that read one +field against another -- a fill value against its data type, a codec +against the array it is handed, a grid against the shape -- are not +judged here. Each concept gets a `validate_*` function returning every +problem found, an `is_*` type guard, and a `parse_*` function that +narrows or raises `MetadataValidationError`. The guards are `TypeGuard`s, +not `TypeIs`: True narrows a value to its document type, and False says +nothing about its type, since a value can be well typed and still not a +valid document. Every `ValidationProblem` carries a machine-readable `kind` alongside its human-readable `message`, so consumers can dispatch on the failure mode @@ -18,7 +24,7 @@ import json from collections.abc import Mapping, Sequence -from typing import Final, TypeGuard, TypeVar, cast +from typing import Any, Final, TypeGuard, TypeVar, cast from zarr_metadata._json import ( MetadataValidationError, @@ -31,9 +37,16 @@ from zarr_metadata._json import prefixed as _prefix from zarr_metadata.v2.array import ZarrV2ArrayMetadataJSON from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON -from zarr_metadata.v3._common import ( - validate_metadata_field_v3, +from zarr_metadata.v3._definition import ( + ChunkGridDefinition, + ChunkKeyEncodingDefinition, + CodecDefinition, + DataTypeDefinition, + Definition, + StorageTransformerDefinition, + resolve, ) +from zarr_metadata.v3._registry import CORE_AND_EXTENSIONS, Context from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON @@ -240,10 +253,7 @@ def _is_canonical_array_metadata_v3(value: object) -> bool: return False if "dimension_names" in doc and not isinstance(doc["dimension_names"], tuple): return False - if not all( - _is_canonical_metadata_field_v3(doc[key]) - for key in ("data_type", "chunk_grid", "chunk_key_encoding") - ): + if not all(_is_canonical_metadata_field_v3(doc[key]) for key, _ in _EXTENSION_POINTS_V3): return False if not all( _is_canonical_metadata_field_v3(item) for item in cast("tuple[object, ...]", doc["codecs"]) @@ -315,11 +325,32 @@ def _validate_attributes(value: object) -> tuple[ValidationProblem, ...]: return tuple(problems) -def validate_array_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: - """Return every reason `value` is not a structurally-valid v3 array doc. +_EXTENSION_POINTS_V3: Final[tuple[tuple[str, type[Definition[Any]]], ...]] = ( + ("data_type", DataTypeDefinition), + ("chunk_grid", ChunkGridDefinition), + ("chunk_key_encoding", ChunkKeyEncodingDefinition), +) +"""A v3 array document's single extension points, and the kind each is read as.""" + +_EXTENSION_LISTS_V3: Final[tuple[tuple[str, type[Definition[Any]]], ...]] = ( + ("codecs", CodecDefinition), + ("storage_transformers", StorageTransformerDefinition), +) +"""Its lists of extension points, and the kind each entry is read as.""" - Checks structure, not domain validity. Unknown top-level keys are allowed - (they map to `extra_fields`). + +def validate_array_metadata_v3( + value: object, *, context: Context = CORE_AND_EXTENSIONS +) -> tuple[ValidationProblem, ...]: + """Return every reason `value` is not a valid v3 array document. + + Its structure, and each extension point read through the definition + that claims its name in `context`: a gzip `level` out of range, a key a + codec's configuration does not declare. A name nothing in `context` + claims is left unjudged. Unknown top-level keys are allowed (they map + to `extra_fields`); a reader must understand each one that does not + say `must_understand: false`, which the model reports as + `must_understand_fields`. """ if not isinstance(value, Mapping): return (ValidationProblem((), "expected a mapping", "invalid_type"),) @@ -331,22 +362,20 @@ def validate_array_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: problems.extend(_validate_dim_sequence(doc, "shape")) if "fill_value" in doc: problems.extend(_prefix("fill_value", validate_json(doc["fill_value"]))) - # Every extension *point* must be understood: ignoring a codec gives - # wrong bytes just as surely as ignoring a data type gives wrong - # values. The spec names only the first three - # (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L1580-L1581), - # which this package reads as an oversight rather than a licence. - # `must_understand: false` keeps its meaning where it has one: an + # Each extension point is read by `resolve`, which judges its envelope + # -- every extension *point* must be understood, so a `must_understand` + # of `false` is refused at each: ignoring a codec gives wrong bytes as + # surely as ignoring a data type gives wrong values, and the spec + # naming only the first three + # (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L1580-L1581) + # is read as an oversight rather than a licence -- and then its + # configuration, against the definition in `context` that claims its + # name. `must_understand: false` keeps its meaning where it has one: an # unknown top-level extension *field*, which a reader really can skip. - for key in ("data_type", "chunk_grid", "chunk_key_encoding"): + for key, kind in _EXTENSION_POINTS_V3: if key in doc: - problems.extend( - _prefix( - key, - validate_metadata_field_v3(doc[key], allow_must_understand_false=False), - ) - ) - for key in ("codecs", "storage_transformers"): + problems.extend(resolve(doc[key], kind, context, (key,))[1]) + for key, kind in _EXTENSION_LISTS_V3: if key in doc: entries = doc[key] if not _is_array(entries): @@ -359,17 +388,7 @@ def validate_array_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: ) ) for index, entry in enumerate(entries): - problems.extend( - _prefix( - key, - _prefix( - index, - validate_metadata_field_v3( - entry, allow_must_understand_false=False - ), - ), - ) - ) + problems.extend(resolve(entry, kind, context, (key, index))[1]) if "attributes" in doc: problems.extend(_validate_attributes(doc["attributes"])) if "dimension_names" in doc: @@ -399,19 +418,23 @@ def validate_array_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: return tuple(problems) -def is_array_metadata_v3(value: object) -> TypeGuard[ZarrV3ArrayMetadataJSON]: - """Whether `value` is a structurally-valid v3 array metadata document.""" +def is_array_metadata_v3( + value: object, *, context: Context = CORE_AND_EXTENSIONS +) -> TypeGuard[ZarrV3ArrayMetadataJSON]: + """Whether `value` is a v3 array document `validate_array_metadata_v3` finds nothing wrong with, written with tuples.""" return ( _is_canonical_json(value, finite=False) - and not validate_array_metadata_v3(value) + and not validate_array_metadata_v3(value, context=context) and _is_canonical_array_metadata_v3(value) ) -def parse_array_metadata_v3(value: object) -> ZarrV3ArrayMetadataJSON: +def parse_array_metadata_v3( + value: object, *, context: Context = CORE_AND_EXTENSIONS +) -> ZarrV3ArrayMetadataJSON: """Return `value` as `ZarrV3ArrayMetadataJSON`, or raise `MetadataValidationError`.""" normalized = arrays_to_tuples(value) - problems = validate_array_metadata_v3(normalized) + problems = validate_array_metadata_v3(normalized, context=context) if len(problems) != 0: raise MetadataValidationError(problems) return cast("ZarrV3ArrayMetadataJSON", normalized) @@ -525,13 +548,15 @@ def parse_array_metadata_v2(value: object) -> ZarrV2ArrayMetadataJSON: return cast("ZarrV2ArrayMetadataJSON", normalized) -def validate_consolidated_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: +def validate_consolidated_metadata_v3( + value: object, *, context: Context = CORE_AND_EXTENSIONS +) -> tuple[ValidationProblem, ...]: """Return every reason `value` is not a valid inline consolidated envelope. Locs are value-relative (the caller prefixes with `consolidated_metadata` where appropriate). Entries recurse into the array and group document - validators, so a validator verdict always agrees with what - `ZarrV3ConsolidatedMetadata.from_json` accepts. + validators, in `context`, so a validator verdict agrees with what + `ZarrV3ConsolidatedMetadata.from_json` accepts in the same scope. """ if not isinstance(value, Mapping): return (ValidationProblem((), "expected a mapping", "invalid_type"),) @@ -562,11 +587,17 @@ def validate_consolidated_metadata_v3(value: object) -> tuple[ValidationProblem, node_type = cast("Mapping[object, object]", entry).get("node_type") if node_type == "array": problems.extend( - _prefix("metadata", _prefix(key, validate_array_metadata_v3(entry_obj))) + _prefix( + "metadata", + _prefix(key, validate_array_metadata_v3(entry_obj, context=context)), + ) ) elif node_type == "group": problems.extend( - _prefix("metadata", _prefix(key, validate_group_metadata_v3(entry_obj))) + _prefix( + "metadata", + _prefix(key, validate_group_metadata_v3(entry_obj, context=context)), + ) ) else: problems.append( @@ -579,13 +610,17 @@ def validate_consolidated_metadata_v3(value: object) -> tuple[ValidationProblem, return tuple(problems) -def validate_group_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: - """Return every reason `value` is not a structurally-valid v3 group doc. - - Checks structure, not domain validity. Unknown top-level keys are allowed - (they map to `extra_fields`); a `consolidated_metadata` key, if present, - is deep-validated (envelope and entries) via - `validate_consolidated_metadata_v3`. +def validate_group_metadata_v3( + value: object, *, context: Context = CORE_AND_EXTENSIONS +) -> tuple[ValidationProblem, ...]: + """Return every reason `value` is not a valid v3 group document. + + Unknown top-level keys are allowed (they map to `extra_fields`); a + reader must understand each one that does not say `must_understand: + false`, which the model reports as `must_understand_fields`. A + `consolidated_metadata` key, if present, is deep-validated (envelope + and entries) via `validate_consolidated_metadata_v3`, each array in it + read as `validate_array_metadata_v3` reads one, in `context`. """ if not isinstance(value, Mapping): return (ValidationProblem((), "expected a mapping", "invalid_type"),) @@ -609,21 +644,27 @@ def validate_group_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: problems.extend( _prefix( "consolidated_metadata", - validate_consolidated_metadata_v3(doc["consolidated_metadata"]), + validate_consolidated_metadata_v3(doc["consolidated_metadata"], context=context), ) ) return tuple(problems) -def is_group_metadata_v3(value: object) -> TypeGuard[ZarrV3GroupMetadataJSON]: - """Whether `value` is a structurally-valid v3 group metadata document.""" - return _is_canonical_json(value, finite=False) and not validate_group_metadata_v3(value) +def is_group_metadata_v3( + value: object, *, context: Context = CORE_AND_EXTENSIONS +) -> TypeGuard[ZarrV3GroupMetadataJSON]: + """Whether `value` is a v3 group document `validate_group_metadata_v3` finds nothing wrong with, written with tuples.""" + return _is_canonical_json(value, finite=False) and not validate_group_metadata_v3( + value, context=context + ) -def parse_group_metadata_v3(value: object) -> ZarrV3GroupMetadataJSON: +def parse_group_metadata_v3( + value: object, *, context: Context = CORE_AND_EXTENSIONS +) -> ZarrV3GroupMetadataJSON: """Return `value` narrowed to `ZarrV3GroupMetadataJSON`, or raise `MetadataValidationError`.""" normalized = arrays_to_tuples(value) - problems = validate_group_metadata_v3(normalized) + problems = validate_group_metadata_v3(normalized, context=context) if len(problems) != 0: raise MetadataValidationError(problems) return cast(ZarrV3GroupMetadataJSON, normalized) diff --git a/packages/zarr-metadata/src/zarr_metadata/pydantic.py b/packages/zarr-metadata/src/zarr_metadata/pydantic.py index d333f629be..2cb69f50d2 100644 --- a/packages/zarr-metadata/src/zarr_metadata/pydantic.py +++ b/packages/zarr-metadata/src/zarr_metadata/pydantic.py @@ -7,8 +7,10 @@ model class — the instances ARE the core classes, so values interoperate freely with non-pydantic code (equality, isinstance, nesting). Validation delegates to the library: a raw document routes through `from_json` (the -single source of truth for structural validation and normalization, so -pydantic's field-level coercion can never bypass it), an existing model +single source of truth for validation and normalization, so pydantic's +field-level coercion can never bypass it), reading v3 extension points in +`CORE_AND_EXTENSIONS`, since a field type holds no scope; a reader with a +scope of its own calls `from_json(..., context=...)` itself. An existing model instance passes through unchanged, and serialization emits the canonical document via `to_json`. `MetadataValidationError` subclasses `ValueError`, so a failed parse surfaces as a pydantic `ValidationError` carrying the diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/definition.py b/packages/zarr-metadata/src/zarr_metadata/v3/definition.py index 0809534690..420926d983 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/definition.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/definition.py @@ -106,6 +106,13 @@ def acme_lz4_rules(configuration: AcmeLz4Configuration) -> Iterator[ValidationPr ) SCOPE = CORE_AND_EXTENSIONS.extended_with(ACME_LZ4) +A scope reads whole documents as well as fields: +`validate_array_metadata_v3(document, context=SCOPE)`, from +`zarr_metadata.model`, reads each extension point of a v3 array document +through the definitions in `SCOPE`, and so do the model's `from_json` and +`from_key_value`. A fill value is not judged against its data type there: +no rule here reads one field against another. + The TypedDict says what a key it does not declare is: with `closed=True`, a problem, as above; with `extra_items=`, a key holding that type; with `closed=False`, anything at all. One that says none of diff --git a/packages/zarr-metadata/tests/model/test_array.py b/packages/zarr-metadata/tests/model/test_array.py index 4586e9c26c..9f4097969a 100644 --- a/packages/zarr-metadata/tests/model/test_array.py +++ b/packages/zarr-metadata/tests/model/test_array.py @@ -713,7 +713,9 @@ def test_v2_roundtrip_json_model_json() -> None: ZarrV3ArrayMetadata.create_default( shape=(2,), attributes={"a": {"b": [1]}}, - codecs=(ZarrV3NamedConfig(name="blosc", configuration={"opts": {"level": 1}}),), + # A name nothing in the scope claims, so reading it back judges only + # the document, and its configuration can nest. + codecs=(ZarrV3NamedConfig(name="acme.nested", configuration={"opts": {"level": 1}}),), extra_fields={"ext": {"must_understand": False, "cfg": {"x": [1]}}}, ), id="v3", diff --git a/packages/zarr-metadata/tests/model/test_extension_points.py b/packages/zarr-metadata/tests/model/test_extension_points.py new file mode 100644 index 0000000000..77424fb567 --- /dev/null +++ b/packages/zarr-metadata/tests/model/test_extension_points.py @@ -0,0 +1,220 @@ +"""The model's v3 validators read each extension point through its definition. + +A document's structure is the model's own judgment; what a codec, data +type, chunk grid or chunk key encoding holds in its configuration is its +definition's, found in a scope (`CORE_AND_EXTENSIONS` unless a caller +passes another) and reported at the field's place in the document. A +name nothing in the scope claims is left unjudged. +""" + +from __future__ import annotations + +import math +from typing import Any, cast + +import pytest + +from zarr_metadata._json import arrays_to_tuples +from zarr_metadata.model import ( + MetadataValidationError, + ZarrV3ArrayMetadata, + ZarrV3GroupMetadata, + is_array_metadata_v3, + parse_array_metadata_v3, + validate_array_metadata_v3, + validate_group_metadata_v3, +) +from zarr_metadata.v3.codec.gzip import GZIP_CODEC +from zarr_metadata.v3.definition import CORE, CORE_AND_EXTENSIONS, CodecDefinition, Context + +BYTES = {"name": "bytes", "configuration": {"endian": "little"}} + + +def _document(**fields: object) -> dict[str, Any]: + """A default v3 array document with `fields` in it, arrays as tuples as a reader's are.""" + document = {**ZarrV3ArrayMetadata.create_default(shape=(4,)).to_json(), **fields} + return cast("dict[str, Any]", arrays_to_tuples(document)) + + +LENIENT_GZIP = CodecDefinition( + name="gzip", + configuration=GZIP_CODEC.configuration, + kind="bytes_bytes", + size="dynamic", + rules=lambda configuration: [], +) +"""A reader's own gzip, which takes any level: a scope can grow, and substitute.""" + + +@pytest.mark.parametrize( + ("document", "context"), + [ + (_document(), CORE_AND_EXTENSIONS), + ( + _document(codecs=[BYTES, {"name": "gzip", "configuration": {"level": 5}}]), + CORE_AND_EXTENSIONS, + ), + ( + _document(codecs=[BYTES, {"name": "acme.lz9", "configuration": {"anything": 1}}]), + CORE_AND_EXTENSIONS, + ), + (_document(codecs=[BYTES, {"name": "zstd", "configuration": {"level": 99}}]), CORE), + ( + _document(codecs=[BYTES, {"name": "gzip", "configuration": {"level": 99}}]), + CORE.extended_with(LENIENT_GZIP), + ), + (dict(ZarrV3ArrayMetadata.create_default(shape=(0, 4)).to_json()), CORE_AND_EXTENSIONS), + ], + ids=[ + "default", + "a-codec-its-definition-allows", + "a-name-nothing-claims", + "an-extension-outside-the-scope", + "a-reader-s-own-definition", + "a-zero-length-array", + ], +) +def test_a_document_reads_through_the_definitions_in_its_scope( + document: dict[str, Any], context: Context +) -> None: + # What the scope holds judges; what it does not is left to the reader. + # The model reads and writes in the same scope as the validators. + assert validate_array_metadata_v3(document, context=context) == () + assert is_array_metadata_v3(document, context=context) + assert parse_array_metadata_v3(document, context=context) is not None + model = ZarrV3ArrayMetadata.from_json(document, context=context) + written = model.to_key_value(context=context) + assert ZarrV3ArrayMetadata.from_key_value(written, context=context) == model + + +@pytest.mark.parametrize( + ("fields", "loc", "kind"), + [ + ( + {"codecs": [BYTES, {"name": "gzip", "configuration": {"level": 99}}]}, + ("codecs", 1, "configuration", "level"), + "invalid_value", + ), + ( + {"codecs": [BYTES, {"name": "gzip", "configuration": {"level": 5, "clevel": 9}}]}, + ("codecs", 1, "configuration", "clevel"), + "unknown_key", + ), + ( + { + "data_type": { + "name": "numpy.datetime64", + "configuration": {"unit": "s", "scale_factor": 0}, + } + }, + ("data_type", "configuration", "scale_factor"), + "invalid_value", + ), + ( + {"chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [-1]}}}, + ("chunk_grid", "configuration", "chunk_shape", 0), + "invalid_value", + ), + ( + { + "codecs": [ + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": [2], + "codecs": [BYTES, {"name": "gzip", "configuration": {"level": 99}}], + "index_codecs": [BYTES], + }, + } + ] + }, + ("codecs", 0, "configuration", "codecs", 1, "configuration", "level"), + "invalid_value", + ), + ], + ids=["codec", "unknown-key-in-a-configuration", "data-type", "chunk-grid", "nested-in-a-shard"], +) +def test_error_a_configuration_its_definition_refuses( + fields: dict[str, Any], loc: tuple[str | int, ...], kind: str +) -> None: + # Located at the field's place in the document; the document is + # refused by every judging entry point, and by the model reading it. + document = _document(**fields) + assert [(problem.loc, problem.kind) for problem in validate_array_metadata_v3(document)] == [ + (loc, kind) + ] + assert not is_array_metadata_v3(document) + with pytest.raises(MetadataValidationError): + ZarrV3ArrayMetadata.from_json(document) + + +def test_error_an_array_in_a_group_s_consolidated_metadata() -> None: + array = _document(codecs=[BYTES, {"name": "gzip", "configuration": {"level": 99}}]) + group = { + **ZarrV3GroupMetadata.create_default().to_json(), + "consolidated_metadata": { + "kind": "inline", + "must_understand": False, + "metadata": {"a": array}, + }, + } + assert [(problem.loc, problem.kind) for problem in validate_group_metadata_v3(group)] == [ + ( + ("consolidated_metadata", "metadata", "a", "codecs", 1, "configuration", "level"), + "invalid_value", + ) + ] + + +def test_an_array_in_a_group_s_consolidated_metadata_reads_in_the_group_s_scope() -> None: + array = _document(codecs=[BYTES, {"name": "gzip", "configuration": {"level": 99}}]) + group = { + **ZarrV3GroupMetadata.create_default().to_json(), + "consolidated_metadata": { + "kind": "inline", + "must_understand": False, + "metadata": {"a": array}, + }, + } + lenient = CORE.extended_with(LENIENT_GZIP) + assert validate_group_metadata_v3(group, context=lenient) == () + model = ZarrV3GroupMetadata.from_json(group, context=lenient) + written = model.to_key_value(context=lenient) + assert ZarrV3GroupMetadata.from_key_value(written, context=lenient) == model + + +def test_error_a_model_is_not_written_in_a_scope_that_refuses_it() -> None: + # The writer judges in the scope it is given, as the reader does: the + # default one refuses the level the reader's own gzip took. + document = _document(codecs=[BYTES, {"name": "gzip", "configuration": {"level": 99}}]) + model = ZarrV3ArrayMetadata.from_json(document, context=CORE.extended_with(LENIENT_GZIP)) + with pytest.raises(MetadataValidationError) as raised: + model.to_key_value() + assert [(problem.loc, problem.kind) for problem in raised.value.problems] == [ + (("codecs", 1, "configuration", "level"), "invalid_value") + ] + + +def test_error_an_envelope_is_judged_once() -> None: + # `resolve` judges the envelope and reads the configuration: one + # report, and the configuration still judged. + document = _document( + codecs=[BYTES, {"name": "gzip", "configuration": {"level": 5}, "must_understand": False}] + ) + assert [(problem.loc, problem.kind) for problem in validate_array_metadata_v3(document)] == [ + (("codecs", 1, "must_understand"), "invalid_value") + ] + + +def test_error_a_field_that_is_not_json_is_not_read() -> None: + # Not JSON is the first verdict: the NaN is reported, and the + # configuration beside it is not read. + document = _document( + codecs=[ + BYTES, + {"name": "gzip", "configuration": {"level": 99}, "must_understand": math.nan}, + ] + ) + assert [(problem.loc, problem.kind) for problem in validate_array_metadata_v3(document)] == [ + (("codecs", 1, "must_understand"), "invalid_value") + ]