feat(zarr-metadata): must_understand false refused at every extension point (layer 1b)

[d-v-b]

🤖 AI text below 🤖

Mirror of zarr-developers#4433, where this is reviewed: the same head commit, on main now that layers 0 and 1 have merged upstream. It stays open here as the base of the fork's stack (#349, #350 and #355 build on it); review fixes land on the upstream branch and this branch follows.

One reading of the spec, in its own PR so it can be decided on its own: an array document may not declare any of its extension points ignorable.

What it changes. validate_array_metadata_v3 now refuses must_understand: false on a codec or a storage transformer, as it already did on a data type, a chunk grid and a chunk key encoding. Ignoring a codec gives wrong bytes as surely as ignoring a data type gives wrong values. The spec names only those three points, which this package reads as an oversight rather than a licence. must_understand: false keeps its meaning where it has one: an unknown top-level extension field, which a reader really can skip. The JSON schema that zarr_metadata.pydantic generates for an array document says the same: every extension point has the mandatory envelope, and the test that holds schema and runtime together covers all five.

What breaks. A document that declares a codec or a storage transformer ignorable, which the package accepted before, now has a problem at codecs.N.must_understand or storage_transformers.N.must_understand, so is_array_metadata_v3 says no and parse_array_metadata_v3, from_json and from_key_value raise. The fragment is marked Breaking:. zarr itself does not read through these validators, so what zarr opens is unchanged.

Not in this PR. A metadata field read on its own, outside a document (validate_metadata_field_v3, parse_metadata_field_v3, ZarrV3NamedConfig, the pydantic ZarrV3MetadataField), still takes must_understand: false, and the envelope's allow_must_understand_false keyword stays. The envelope's bool is public API: seventeen TypedDicts and the ZarrV3NamedConfig model declare it. Making the envelope refuse false itself would retype all of them, which is worth doing only once this reading is accepted. Until then a model built by hand can hold what its reader refuses: ZarrV3ArrayMetadata.create_default().update(codecs=(ZarrV3NamedConfig(name="bytes", configuration={}, must_understand=False),)) builds, and its to_key_value, which validates what it writes as the reader does, raises with the problem at codecs.0.must_understand. Closing that gap in the types is the follow-up if the reading is accepted.

Review question: is refusing must_understand: false at codecs and storage transformers the right reading of the spec?

The stack. This is layer 1b of the zarr-metadata stack whose layers 0 (zarr-developers#4420, zarr-developers#4421, zarr-developers#4422) and 1 (zarr-developers#4432, the TypedDict checker) have merged. It is based on main and stands alone. The later layers are open here: metadata fields read against their definitions (#350, upstream as zarr-developers#4434, with its core reviewable alone in #349), then the field in a document (#355). They apply the same refusal when they judge a field's envelope, and a test there holds the two readers together at all five points, so if this reading is rejected, those call sites change with it.

🤖 Generated with Claude Code

[d-v-b]

🤖 AI text below 🤖

Merged upstream with zarr-developers#4434 (squash 8dc50b81b), which carried these commits; its upstream twin, zarr-developers#4433, is closed.