fix(chunk-grids): one invariant for zero-length axes across model, clamps, and metadata by d-v-b · Pull Request #4334 · zarr-developers/zarr-python

d-v-b · 2026-09-09T18:20:46Z

This PR tightens our representation of chunk grids to disallow the representation of 0-length chunks. This has compatibility implications: previous versions of zarr could access arrays with 0-length chunks and exercise a subset of the Zarr API against them (reading / writing attributes), but not write chunks. I'm still thinking about the right compatibility shim for this, hence the draft status,

Previous PRs that removed 0-length chunks caused issues downstream so I'd like some eyes on this change (cc @TomNicholas)

🤖 AI text below 🤖

Target: 3.4.1 (patch). A chunk edge length is now always an integer of at least 1, while an array extent may still be 0. Stored metadata that older releases wrote with a chunk size of 0, false or true is read by one lenient module, and nothing that zarr 3.4.0 accepted and handled correctly is rejected, changed, or newly warns (the one exemption is the experimental rectilinear metadata API, listed below).

Summary

What	Before (3.4.0)	With this PR
Zarr format 2 array with stored `chunks: [0]` on an axis that has grown	opens; `append` reports success, stores no chunk, reads back fill values (silent data loss)	opens with a warning that the axis holds only the fill value; `append` stores the data and valid metadata
Zarr format 3 array with stored `chunk_shape: [0]` / `[false]` (3.0.x, 3.1.x)	does not open	opens; silent on an empty axis, warns on a grown one
Sharded Zarr format 3 array with outer `chunk_shape: [0]` (3.0.x, 3.1.x)	does not open	opens; the outer chunk is a multiple of the inner chunk size
Stored JSON `true` as a chunk size	regular: kept as `True`, written back as `true`; rectilinear: `ValueError`	read as 1, silently
`create_array(shape=(0, 20), chunks=(5, 5), shards=-1)`	`ValueError` (not divisible)	shard shape `(5, 20)`
`create_array(shape=(0,), chunks=[[5, 10, 5]])` (rectilinear, flag on)	`ValueError` (does not sum to 0); zarr 3.2 allowed it	creates the array; the edges describe the chunks the axis grows into

Design

One invariant, one full-span rule

FixedDimension requires size >= 1 (was >= 0); its special cases for a size of 0 are removed. "One chunk spanning an axis of length span" is defined once, in zarr.core.chunk_grids.full_span_chunk_size(span, unit):

unit * max(1, ceildiv(span, unit))

unit is the size the chunk must be a multiple of: the inner chunk size when the chunk is a shard, 1 otherwise. Every spelling that derives a chunk from a span uses it (chunks=-1, chunks=False, the starting point of chunks="auto", shards=-1, shards=False), and so does the reading of a stored chunk size of 0. normalize_chunks_nd takes a per-axis unit, which resolve_outer_and_inner_chunks passes for shard shapes. For a rectilinear dimension of extent 0, any non-empty list of positive edges is kept (the same state resize(0) produces).

Stored documents: one lenient module

src/zarr/core/metadata/upgrades.py is the only place invalid metadata is read leniently. Each upgrade is a pure function from a stored JSON document to a valid one, and ArrayV2Metadata.from_dict / ArrayV3Metadata.from_dict apply them before the constructors run, so every path that parses a stored document (single array open, group members, consolidated metadata) gets the same readings. Type tests there are exact JSON types; no NumPy.

Stored value	Where	Written by	Read as	Warns
`0` or `false`, axis of length 0	Zarr format 2 `chunks`, `regular` `chunk_shape`	2.18.7 through 3.3.0 for arrays created with a zero-length axis	one chunk spanning the axis: `unit` (1, or the inner chunk size)	no (3.4.0 opened the Zarr format 2 ones without a warning)
`0` or `false`, axis of positive length	same	the arrays above after the axis grew; explicit `chunks=(0,)`/`False` in 2.18.7 to 3.2.1	one chunk spanning the axis	yes: no chunk can have been stored, so the axis holds only the fill value
`true`	regular chunk size; rectilinear edges and run-length sizes; sharding inner `chunk_shape` (nested codecs too)	3.0.x, 3.2.0/3.2.1	1	no
integral float such as `4.0`	rectilinear edges and run-length sizes	3.2.x	the `int` it equals	no
any other float (regular or inner chunk size, run-length count, below 1, fractional)		no known writer	rejected by the constructors

A warning is given once per document, after the upgraded document has passed the constructor (so an invalid document raises its own error rather than a warning), and names the array by its full path, including for members of consolidated metadata. Messages describe the metadata, never which release wrote it.

Strict vs lenient

Stored documents are read leniently only through upgrades.py; the metadata constructors keep 3.4.0's acceptance in this patch release (ArrayV2Metadata(chunks=(0,)), NumPy integers in ArrayV2Metadata, ShardingCodec(chunk_shape=...) are all still accepted). One rule, parse_chunk_edge in zarr.core.common, checks every bare chunk size, explicit edge and run-length size in metadata: 3.4.0's isinstance(value, int) test, with a bool read as the int it equals. An array built directly from code-built ArrayV2Metadata with a chunk size of 0 (for example through create_hierarchy) reads that chunk size the way a stored 0 is read, silently; the metadata object itself is still stored as given. The strict constructors are a separate PR for 3.5.0.

Writing after an upgraded read

Metadata read from an upgraded document remembers the document as stored (_stored_document, set in one place, mark_upgraded). Before the first non-empty chunk write through such a handle, the array:

re-reads its current stored document and upgrades it;
if there is no document any more, writes chunks as 3.4.0 does;
if the upgraded current document lays out chunks differently from the handle (another writer resized the array keeping the chunk size of 0, say), raises ValueError asking to reopen the array, and stores nothing;
if it still needs upgrading, stores the upgrade through an internal diff-and-upsert (zarr.core.metadata.io: only documents that differ are written); if another writer has stored valid metadata since, it is left as written.

update_attributes, resize and append store the upgrade as before. Every reader of the store then agrees with the chunks written. In a ZipStore the first write adds a second entry for the metadata document, as every metadata update there does.

Groups never touch member documents

A group write (attributes, update_attributes, del group[name], consolidate_metadata, create_hierarchy) stores only that group's own documents. It never reads, writes or re-saves a member's documents. The consolidated copy of a member read from an upgraded document is written back exactly as it was stored, so every reader of the consolidated metadata still reads it as upgraded, and the member's own first chunk write (above) is the only place its document is upgraded on disk. Group writes with legacy members therefore cost what they cost in 3.4.0 (no member I/O). Once the array stores its upgrade through a handle that shares the group's metadata, later group writes store the upgraded form.

What users see

Warning (ZarrUserWarning) only where action is needed, for example:

Array 'file:///data/x.zarr/a': The stored chunk shape [0] is invalid: chunk sizes must be integers of at least 1. It is read as [3], reading 0 in dimension 0 as one chunk spanning the dimension (3), and as no chunk can be stored under a chunk size of 0, the array holds only its fill value. To store valid metadata, open the array writable and call array.update_attributes({}); if a group holds consolidated metadata for the array, then also call zarr.consolidate_metadata on that group.
Silent where 3.4.0 read the same value: 0/false on an empty axis (as 2.18.7 and 3.3.0 wrote for every empty Zarr format 2 array), true, float rectilinear edges. With -W error, datasets that 3.4.0 opened still open.
Error on a write through a handle whose stored chunk layout changed underneath it: ValueError asking to reopen, nothing stored.
FixedDimension(size=0, ...) raises ValueError (internal class; no public path that worked in 3.4.0 reaches it).
RegularChunkGridMetadata reads a bool edge as its int (3.4.0 kept True and wrote JSON true) and rejects a string or mapping chunk shape as a whole (3.4.0 rejected them with an unrelated comparison error).
Experimental rectilinear API (exempt from the patch rule, per the standing rule that rectilinear APIs are experimental): RectilinearChunkGridMetadata, its from_dict and expand_rle read a bool edge as its int and reject NumPy integers and float edges such as 4.0 with a TypeError. 3.4.0 kept them as given, so it could not store a NumPy integer, could not read back a bool, and stored a float as a JSON float, which the spec does not allow. Stored array documents with 3.2.x float edges still open, through the upgrade module.

3.4.0 compatibility evidence

Behaviour table: 244 inputs (metadata constructors, stored documents, creation calls with every chunk spelling, NumPy inputs, zarr.testing strategies, ZipStore resizes) were run against a real zarr==3.4.0 install and against each branch. 96 give 3.4.0's result, 121 were rejected by 3.4.0, and 27 differ; each of the 27 is documented (3.4.0 mishandled it, the parsed value is equal and only its JSON spelling changes, it is internal, or it is the experimental rectilinear API). check_patch.py re-runs the probes: OK, 0 undocumented changes, 0 warnings-only changes.
Writes: a 906-case write matrix (both formats; shapes including zero-length axes; several dtypes; "auto", -1, explicit, None, True, 0 chunks; sharding; fill values; compressors) stores byte-identical documents and chunks to 3.4.0 in 905 cases. The remaining case is shards=-1 on shape (7, 13) with inner chunks (1, 3), which 3.4.0 rejected.
Downstream: xarray's zarr backend tests (test_backends.py, test_backends_datatree.py, -k zarr) give 1495 passed on both 3.4.0 and the combined patch branches, also with -W error::zarr.errors.ZarrUserWarning. VirtualiZarr's test suite gives identical results against both (657 passed; the same 2 environment failures). These runs were made on the combined patch branches before the final group-write change, which only removed member reads and writes.
Other implementations reading what these branches write (both formats, -1/False/"auto" on empty and grown axes, sharded -1/"auto"/explicit, a non-multiple shards=-1): tensorstore 0.1.85 and zarrs read all 19 arrays with the expected data. zarrita reads every non-empty one; it fails on any zero-size array ("Input contains an empty iterator"), including one written by 3.4.0, so that is a zarrita limitation. Rectilinear arrays created on an empty axis read in zarrs; tensorstore and zarrita do not implement the rectilinear grid.
Real legacy stores: 54 stores written by real installs of zarr 3.0.10, 3.1.6, 3.2.1, 3.3.0 and 3.4.0 (both formats, sharded, consolidated, axes grown by resize, append or a write) open on this branch, take an append without losing data, and reopen with warnings turned into errors, i.e. the append stored valid metadata.

Tests

tests/test_metadata/test_upgrades.py: one table of stored documents and their upgrades and warnings, one test per rejection, end-to-end round trips over both formats, the first-write re-save (including a stale handle that must keep newer metadata, a stale handle whose chunk layout changed, an empty write that stores nothing, and a missing document), and groups: group writes store a legacy member's consolidated copy as stored, concurrent deletions leave no member document behind, and a member's first write stores its upgrade.
tests/test_metadata/test_io.py: the internal diff and upsert (stores only what differs, stores nothing for identical documents, leaves the store untouched when encoding fails).
tests/test_array_stateful.py (ArrayLifecycle): one array through create, append, resize (to and from 0), write and re-save, against a NumPy model that tracks cells beyond the shape exactly as chunks keep them. It draws both formats, every chunk spelling, sharding, rectilinear grids and stored chunk sizes of 0, and checks after each step that no chunk sits under a still-invalid stored document. It runs in the slow hypothesis job (just hypothesis, now including this file).

Known limitations

A consolidated copy of a member that has since been re-saved through its own handle stays stale until zarr.consolidate_metadata runs, as any consolidated copy does in 3.4.0.

Merge order and release

fix: read mixed regular/rectilinear chunk grids written by zarr 3.2.x #4375 and test: sample the full rectilinear chunk grid declaration space #4377 contain this PR; their diffs include it until it merges.
fix: make the chunk normalizer the one judge of a chunk specification #4376 is independent but conflicts in two small hunks. Resolution: in resolve_outer_and_inner_chunks, keep fix: make the chunk normalizer the one judge of a chunk specification #4376's single call and add this PR's unit, outer = normalize_chunks_nd(shard_spec, array_shape, unit=chunks.chunk_shape); in tests/test_unified_chunk_grid.py, keep both imports (import re and from functools import partial).
A follow-up PR for 3.5.0 makes the metadata constructors strict; it contains all four patch PRs.

🤖 Generated with Claude Code

…, clamps and metadata Invariant: a chunk edge length is always >= 1; a dimension's extent may be 0, in which case the dimension has zero chunks (ceildiv(0, size) == 0). Zero-length-axis bugs have recurred since 2017 (#150, #241, #303, zarr-developers#972, zarr-developers#1977, zarr-developers#2434, zarr-developers#3711, zarr-developers#4305, zarr-developers#4307, zarr-developers#4328) because the layers disagreed on this invariant and every span-derived chunk spelling clamped on its own: - The metadata layer (common.py, metadata/v3.py) required chunk edges >= 1, but the in-memory FixedDimension allowed size == 0 with four special-case branches left over from zarr-developers#2434, so normalization could build a grid the metadata constructor then rejected. FixedDimension now rejects size < 1 and the four `if self.size == 0` branches are gone. VaryingDimension already required edges > 0 and is unchanged. - `chunks=-1`, `chunks=False`, `chunks="auto"` (_guess_regular_chunks, both the typesize == 0 early return and the np.maximum line) and `shards="auto"` each derived "one chunk covering the axis" independently. They now all go through one helper, `_full_span_chunk_size(span) = max(span, 1)`, which is the single definition of that phrase for a possibly zero-length axis. - Zarr format 2 metadata had no chunk >= 1 check, so a legacy `chunks: [0]` document opened fine and read uninitialised memory after a resize. It now raises a clear ValueError at parse time, matching the format 3 grid. - Rectilinear grids had no creation-time spelling for a zero-length axis: normalize_chunks_1d required sum(edges) == span, which no list of positive edges can satisfy for span 0, even though the same state is reachable via resize((0,)) and round-trips through reopen. For span == 0 any non-empty list of positive edges is now accepted verbatim, producing the same VaryingDimension(edges, extent=0) that resize produces; the strict sum check is kept for span > 0. Tests: the per-spelling regression test from zarr-developers#4328 is replaced by one matrix over {-1, False, "auto", 1, (1,...), [[2, 2]]} x {(0,), (0, 4), (4, 0), (0, 0), ()} x {v2, v3} x {no shards, shards="auto" with and without a byte budget, explicit shards}, with separate small tests for each error case. Tests that constructed FixedDimension(size=0) now assert it raises, and a zero-extent test covers the behaviour the old special cases were guarding. Assisted-by: ClaudeCode:claude-fable-5-1

zarr-python 2.18.7 writes `chunks: [0]` for `zarr.zeros((0,), chunks=False)` and for `chunks=(0,)`, so stores with that document exist. Rejecting them at open would turn a previously-readable array into an error; leaving the 0 in place read uninitialised memory after a resize. Normalize the edge to 1 with a ZarrUserWarning instead — the same grid every other "one chunk spans the axis" spelling produces — and keep rejecting a zero edge on an axis that has data. Assisted-by: ClaudeCode:claude-fable-5-1

Assisted-by: ClaudeCode:claude-fable-5-1

Measured against zarr 2.18.7: `zeros((0,), chunks=False)`, `chunks=-1` and `chunks=(0,)` all write `chunks: [0]`, after which nchunks, read, write, append, resize and reopen-then-read every raise ZeroDivisionError. There was never a working behaviour to preserve; normalizing the edge to 1 makes such arrays usable for the first time. Say so in the comment and fragment instead of claiming the stores were previously readable. Assisted-by: ClaudeCode:claude-fable-5-1

codecov · 2026-09-09T18:40:11Z

Codecov Report

❌ Patch coverage is 98.83721% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.51%. Comparing base (f58644a) to head (30d7ca6).
⚠️ Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
src/zarr/core/metadata/upgrades.py	97.94%	3 Missing ⚠️
src/zarr/core/chunk_grids.py	93.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4334      +/-   ##
==========================================
+ Coverage   94.37%   94.51%   +0.14%     
==========================================
  Files          93       94       +1     
  Lines       13171    13406     +235     
==========================================
+ Hits        12430    12671     +241     
+ Misses        741      735       -6

Files with missing lines	Coverage Δ
src/zarr/core/_json.py	`100.00% <100.00%> (ø)`
src/zarr/core/array.py	`98.21% <100.00%> (+0.12%)`	⬆️
src/zarr/core/common.py	`91.71% <100.00%> (+1.23%)`	⬆️
src/zarr/core/group.py	`95.25% <100.00%> (+0.03%)`	⬆️
src/zarr/core/metadata/io.py	`100.00% <100.00%> (ø)`
src/zarr/core/metadata/v2.py	`90.65% <100.00%> (+1.27%)`	⬆️
src/zarr/core/metadata/v3.py	`96.50% <100.00%> (+1.05%)`	⬆️
src/zarr/core/chunk_grids.py	`96.74% <93.33%> (-0.04%)`	⬇️
src/zarr/core/metadata/upgrades.py	`97.94% <97.94%> (ø)`

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Assisted-by: Codex:GPT-6

TomNicholas · 2026-09-16T15:54:05Z

I was slightly ahead of you here - I think as long as I release my VZ PR before you release this then VZ at least will not break as a result of this change in Zarr-python.

… format 3 The compatibility policy for a stored chunk size of 0 on a zero-length axis covered Zarr format 2 only, so arrays written by zarr-python 3.0 and 3.1 with `chunk_shape: [0]` — or `[false]`, which 3.0 wrote for `chunks=False` — still could not be opened at all. `ArrayV3Metadata` now applies the same policy to a regular chunk grid: a stored chunk size of 0 on a zero-length axis is read as 1 with a `ZarrUserWarning`, and a zero chunk size on a positive-length axis is left for the chunk grid parser to reject. It runs in `__init__` rather than in the grid parser because the policy needs the array shape, which chunk grid metadata does not carry. Both warnings now say how to store a corrected chunk size — open the array writable and call `array.update_attributes({})`, which rewrites the whole document from the parsed metadata — from one shared constant. The Zarr format 2 warning also named only zarr-python 2.x; measured against real installs, every 3.x release before 3.4 wrote a zero chunk size for an empty array too (3.3.0 for `chunks=-1` and `chunks=False`). Tested against stores written by zarr 3.0.10, 3.1.6 and 3.3.0: they open, append without losing data, and re-save to a chunk size that reopens without a warning. Assisted-by: ClaudeCode:claude-opus-5

… in one routine The legacy zero-chunk policy was written out twice: inline in `ArrayV2Metadata.__init__`, and again in a Zarr format 3 helper. The V2 copy also zipped with `strict=False` and re-appended any trailing chunk entries only so that a separate length check, `parse_metadata`, could report a dimensionality mismatch after construction. `parse_stored_chunk_shape` in `zarr.core.metadata.common` is now the one place a stored chunk shape is checked against its array's shape, for both formats: one entry per axis, every integer chunk size at least 1, and a size of 0 (or JSON `false`) on a zero-length axis read as 1 with a warning that names the writer and how to re-save. Non-integer entries, such as edge lists, pass through for the caller's own parser. `ArrayV2Metadata.__init__` calls it directly and `parse_metadata` is gone. The Zarr format 3 adapter only locates a regular grid's `chunk_shape` in the stored document and hands it over; it still runs in `ArrayV3Metadata.__init__` because chunk grid metadata has no array shape. The Zarr format 3 import changes that existed only for the old helper are reverted. Tests for the policy now target the routine: one table of valid and legacy inputs, and one test per rejection (dimension mismatch, zero on a non-empty axis, negative). They replace metadata-level tests in test_v2.py and test_v3.py that only re-tested the same rules; the end-to-end tests still cover both formats' wiring against stored arrays. Assisted-by: ClaudeCode:claude-opus-5

…nk grids `parse_stored_chunk_shape` passed non-integer entries through "for the caller's own parser", which made a regular-grid policy look like a general chunk shape routine and let it decide what a 0-length chunk means for grids it does not own. A rectilinear grid, or any other grid, is free to define its own semantics for 0-length chunks. It is now `parse_stored_regular_chunk_shape`, typed `Sequence[int]`, with no pass-through, and its docstring says it applies to Zarr format 2 `chunks` and Zarr format 3 `regular` grids only. The Zarr format 3 caller hands it a chunk shape only when the grid is named `regular` and every entry is an integer (`_is_regular_chunk_shape`); anything else is not a regular chunk shape and goes to the chunk grid parser untouched. Assisted-by: ClaudeCode:claude-opus-5

… into fix/mixed-regular-chunk-grid-4374 # Conflicts: # src/zarr/core/metadata/v3.py

…place zarr-developers#4334 read a zero chunk size on an empty axis in `ArrayV3Metadata.__init__` and zarr-developers#4375 read a 3.2.x mixed grid inside `parse_chunk_grid`, each with its own predicate, warning text and re-save advice. Both are compatibility readings of a stored `regular` grid, so `_read_stored_regular_chunk_grid` now dispatches to both, `parse_chunk_grid` accepts only what the spec allows, and both warnings use `RESAVE_METADATA_HINT`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… into test/rectilinear-declaration-space

With zarr-developers#4334 a rectilinear dimension of extent 0 takes any non-empty list of positive edges at creation, the same state 3.2.x wrote and resize(0) produces. The strategies asserted extent > 0 and `chunk_grids` fell back to a regular grid for any empty axis, so no property saw that state. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…panning chunk A stored chunk size of 0 was tolerated only on a zero-length axis and rejected otherwise, because the metadata supposedly could not say how the stored chunks were laid out. But a chunk size of 0 gives a grid of zero chunks, so no release could store a chunk under it, and the writers of that metadata let the axis grow: 3.4.0 appends to a Zarr format 2 array created empty by 3.3.0 (shape grows, no chunk written), and 3.1.6 records a Zarr format 3 resize and the resize half of a failed append. Measured with real installs. Those arrays open in 3.4.0, attributes included; the rejection would have made them unopenable. `parse_stored_regular_chunk_shape` now reads a stored 0 (or JSON `false`) on any axis as one chunk spanning it, `max(extent, 1)`, which is what the `-1`/`False` spec that wrote it meant. On a grown axis the warning also says that data written to it was not saved. Negative sizes are still rejected. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The only state machine that touched arrays compared zarr on one store with zarr on a MemoryStore, so a chunk grid bug showed up identically on both sides; it had no append rule, covered Zarr format 3 only, and kept every empty axis at 0 when resizing, which is where the zero-length bugs live. `ArrayLifecycle` checks one array against a NumPy model across both formats, every chunk spelling (-1, False, "auto", ints, sharded, rectilinear) and the stored chunk size of 0 that releases before 3.4 wrote, including on an axis those releases grew. Rules append, resize (growing and shrinking to and from 0), write and re-save the metadata; the invariant reopens the array and compares shape, values and whether the legacy warning is due. Deliberately breaking the grown-axis policy, the legacy warning, or append on an empty axis each fails it. `resize` keeps partly retained chunks whole, so cells cut off by a shrink can come back with their old values when the axis grows (as in 2.x); the model marks such cells unknown until written instead of encoding that. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… into fix/mixed-regular-chunk-grid-4374 # Conflicts: # src/zarr/core/metadata/v3.py

… into test/rectilinear-declaration-space

The warning for a stored chunk size of 0 said which zarr-python releases wrote it. The check runs on every metadata construction, so metadata built in code, such as VirtualiZarr's kerchunk writer passing an empty array's shape as `chunks`, was told it came from zarr-python 2.x. The warning now says what the chunk size is read as and, on an axis of positive length, that the axis holds only the fill value. `legacy_writers` is gone, and the docstrings no longer narrate release history; the changelog keeps it. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… into fix/mixed-regular-chunk-grid-4374 # Conflicts: # src/zarr/core/metadata/v2.py # src/zarr/core/metadata/v3.py

… machine The model now tracks cells beyond the array's shape: a shrinking resize deletes exactly the chunks outside the new grid, kept chunks keep their out-of-bounds cells, and a write covering every in-bounds cell of an unsharded chunk resets its out-of-bounds cells to the fill value. A resize that never deletes chunks now fails the test. Sharding is its own chunk spelling, a stored chunk size of 0 applies to sharded arrays too (in the outer grid), re-saving metadata only runs when the store warns, and the test takes its settings from the repository's hypothesis profiles under the slow_hypothesis marker, like the other stateful tests. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…dates An upgrade now returns the upgraded document and how it read it, and `from_dict` warns with those readings after the metadata constructor accepts the upgraded document. A stored document that is still invalid after an upgrade raises its own error instead of first warning how it was read. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e for edges Stored regular chunk shapes, in both formats and in a sharding codec's inner chunk shape, are read entry by entry by one rule in upgrades.py: an int >= 1 is kept, `true` is read as 1 (zarr 3.0.10 also wrote it as the inner chunk size of a shard, which the sharding codec used to read silently), and 0 or `false` as one chunk spanning the axis. Integral floats are not read: no release wrote a regular grid with them. A document warns once, naming the array where the caller knows its path. Metadata constructors check every chunk edge length with one function, `parse_chunk_edge`, which tells a wrong type (TypeError) from a wrong value (ValueError); this covers bare sizes, rectilinear edges, RLE sizes and the sharding codec's inner chunk shape. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

`_guess_regular_chunks` clamped zero-length axes with `np.maximum`, restating `full_span_chunk_size` with unit 1. Fold the zero-span normalizer tests into the `normalize_chunks_1d` tables. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add it to `just hypothesis`. Re-saving metadata is always enabled and must leave valid metadata unchanged; appends favour an axis stored with chunk size 0, which raises how often a legacy axis grows before it is re-saved. A fixed example pins that a write to a shard kept by a shrink leaves its cells beyond the shape alone. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

An array read from a stored document that the upgrades had to correct (a chunk size of 0, `false` or `true`) wrote chunks under the corrected layout while the store kept the old document, so zarr 3.2.1-3.4.0 then read only the fill value, and tensorstore and zarrs could not open it. `from_dict` now marks the metadata it read from an upgraded document (`_stored_document_upgraded`, a field outside the document and equality), and `AsyncArray._set_selection`, which every chunk write goes through (`setitem` now included), stores that metadata first and keeps the stored copy. Concurrent first writes store the same document. The array lifecycle state machine now ends the legacy state on a write and checks that no chunk is stored under a document that is still upgraded on read; `resave_metadata` runs only while the stored document is invalid, and re-saving valid metadata is checked once, at creation. The changelog also says which releases stored a chunk size of 0 on an axis of positive length. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Warnings about an upgraded document name the array by its store path, `str(StorePath(store, path))`, on every path that reads one: opening an array, `AsyncArray.from_dict`, a group read with or without consolidated metadata, and consolidated members, which were named relative to the group. `GroupMetadata.from_dict` and `ConsolidatedMetadata.from_dict` take the group's path for that. The consolidated test now also opens the group with `use_consolidated=False` and checks each array's name, so dropping the path on either read path fails a test. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…in messages - `_validate_chunk_shapes` decides "edge list or bare size" with one `match`: a list or tuple is an edge list, anything else is a bare size checked by `parse_chunk_edge`, so a string dimension such as `"10"` is rejected as not an int instead of being iterated per character. Stored rectilinear `from_dict` makes the same JSON split and passes the dimension to `expand_rle`, so an invalid edge or RLE count inside a list names its dimension. - Messages say "dimension" throughout, lowercase after the colon ("Dimension 0: chunk edge length must be >= 1, got 0"); the upgrade warnings read "0 in dimension 0 as one chunk spanning the dimension". - The upgrades test for JSON arrays with `list` only (a stored document holds no tuples), and `_read_chunk_size` says what `span=None` means. - `ArrayV2Metadata.from_dict` copies the document once. - The rejected stored chunk shapes get one test per error case. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…riting A handle read from an upgraded document upserts the upgrade of what the store holds on its first non-empty chunk write: a stale handle no longer rolls back newer metadata, and an empty write stores nothing. consolidate_metadata does the same for each upgraded member before writing the consolidated document. Both go through two internal primitives in zarr.core.metadata.io: diff_documents compares a node's stored documents with those its metadata would store, value by value, and upsert_metadata encodes first, then stores only the documents that differ and returns the changes. The upgraded-document flag is a ClassVar set on the instance by one helper, mark_upgraded, which also warns; the upgrades are keyed by Zarr format; the Metadata.to_dict and V2 from_dict changes are reverted. _append goes through AsyncArray.setitem and _setitem is removed. A chunk shape that is not a list or tuple is rejected as a whole, and every expand_rle error names its dimension. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e machine An empty write stores no metadata, so it no longer ends the legacy state. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…metadata constructors (patch release) Nothing in a patch release may reject a chunk size that zarr 3.4.0 accepted. The one chunk edge rule (`parse_chunk_edge`, which also reads RLE repeat counts) now reads any integral number as the `int` it equals: an `int`, a `bool`, a NumPy integer or an integral float, so rectilinear grids written with float edges (`[[4.0, 2]]`) read again and the grid constructors accept NumPy integers; a fractional float is still rejected. A regular chunk shape may again be any iterable. `ArrayV2Metadata(chunks=...)` and `ShardingCodec(chunk_shape=...)` read their chunk shape with `parse_shapelike`, as in 3.4.0: a scalar, NumPy integers, bools and 0 are accepted, and a 0 is written back as given (the kerchunk writer in VirtualiZarr relies on `chunks=(0,)` for empty inlined arrays). Stored documents with a chunk size of 0 are still read by the upgrades. The strict rule returns in the next minor release. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e's first write A handle read from an upgraded document re-reads the stored document before its first chunk write. If that document no longer needs an upgrade, store nothing: re-encoding it could differ from how another implementation wrote it, and nothing about it needs fixing. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…release Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The patch release widened the one chunk edge rule to accept integral floats so that stored rectilinear grids written with float edges (`[[4.0, 2]]`, as zarr 3.2 wrote for float edges) keep opening. That also made a stored regular chunk shape `[10.0]` open, which zarr 3.4.0 rejected and the next minor release would reject again. The constructor rule now accepts integers of any integer type (`int`, `bool`, NumPy integers) and rejects floats. A new document upgrade reads an integral JSON float of at least 1 as an `int` where zarr 3.2 stored one: the explicit edges and run-length encoded sizes of a rectilinear chunk grid, with the standard warning. Bare sizes, repeat counts, regular chunk shapes and inner chunk shapes stay for the constructors to reject. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s anything A node whose metadata no array or group can be built from now fails with the store untouched, even with overwrite=True. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…pgraded copies - Readings that give what zarr 3.4.0 read (0/false on an empty axis, JSON true, float rectilinear edges) are silent; the metadata is still marked upgraded, so the first write re-saves it. - JSON true is read as 1 in stored rectilinear edges and RLE sizes and in nested sharding codecs' inner chunk shapes. - A write through a handle whose stored document now lays out chunks differently raises and stores nothing; with no stored document it writes as before. - Storing group metadata refreshes each upgraded consolidated member from its own stored document (the one place: save_metadata); consolidate_metadata no longer needs its own pass. - Metadata built in code with chunk size 0 is read through the upgrades, so an array can be built from it, as in 3.4.0. - Chunk edges in metadata follow the 3.4.0 integer rule (int; bool read as its value); a string or mapping is rejected as a chunk shape as a whole. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…y axes Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…shape is invalid A stored outer chunk size of 0 of a sharded array is read in multiples of the inner chunk size. When that inner size is itself 0 or `false`, the unit is unknown: the upgrade now leaves the 0 for the constructor, which rejects it with the same `ValueError` zarr 3.4.0 raised, instead of a ZeroDivisionError. `_read_codec` now matches the sharding codec like `_inner_chunk_shape` does. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…pt it A group handle kept the consolidated copies of upgraded members flagged, so every later group write re-read every such member, one after another, and read each twice on the first write (once to parse, once to diff). - `save_metadata` returns the metadata it stored; `AsyncGroup._save_metadata` (attrs, `update_attributes`, `delitem`, `consolidate_metadata`) and `Group.update_attributes_async` adopt it, so later writes read no member. - The refresh visits only nested groups and flagged members, and reads them with one `asyncio.gather`. The documents it reads are the ones the upsert diffs against (`read_documents` + `upsert_metadata(..., stored)`), and the array write path does the same. - `parse_stored_array` (documents -> silently marked metadata) replaces `read_stored_array`'s `(metadata, bool)` tuple, and is the one "read, mark, don't warn" path, also for code-built metadata. - A flagged member whose own document is gone or cannot be read (invalid, replaced by a group) keeps its consolidated copy instead of failing every group write. - `parse_array_metadata` of a metadata object reads it as a stored document only when `ArrayV2Metadata.chunks` holds a 0, the one chunk size that constructors still accept and the upgrades change. This removes the per-construction `to_dict` + upgrade (AsyncArray() back to 3.4.0 speed) and building arrays from codec configurations holding NumPy scalars works again, as in 3.4.0. - `create_hierarchy` documents that it stores a group's consolidated metadata as given. - Tests: second group write reads no member; nested member refresh; member without a readable document; NumPy-scalar codec configuration; stale handle whose inner chunk shape alone changed (kills `_chunk_layout` returning only the outer grid). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A group write replaced the handle's metadata with a copy holding the consolidated members it read again, so the handle no longer shared its consolidated dicts with subgroup handles taken earlier: an array deleted through such a subgroup was stored again by the parent's next write, and reappeared on reopen. `_refresh_consolidated` now returns, along with the copy it encodes, the members it replaced (with the consolidated dict that holds each), and `save_metadata` writes them back into those dicts once the store succeeds, unless the member was deleted or replaced meanwhile. `save_metadata` returns nothing again, and the group callers keep their metadata objects. A failed store leaves the members flagged, so a retry reads them again. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…solidated member A group write reads each upgraded consolidated member again from its own document, and kept the consolidated copy when that document could not be read. That also swallowed the rectilinear chunks flag error, so a member whose document now declares a rectilinear grid had its stale copy stored as valid. The flag error is now `RectilinearChunksDisabledError`, a `ValueError`, and the refresh lets it propagate: the group write raises and stores nothing. Unreadable documents keep their consolidated copy as before. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…stored Group writes (attributes, `update_attributes`, deleting a member, `consolidate_metadata`, `create_hierarchy`) read each upgraded member of the consolidated metadata again from its own document and stored its upgrade. Those member writes raced concurrent deletions: two concurrent `delitem`s on a consolidated group recreated a deleted array's metadata document in most runs, which zarr 3.4.0 never did. A group write now stores only the group's own documents and reads none. Array metadata read from a document that had to be upgraded keeps that document (`_stored_document`, set by `mark_upgraded`, replacing the `_stored_document_upgraded` flag), and consolidated metadata stores such a member as it was stored. Every reader of the consolidated metadata then reads the member as upgraded again, and the array's own first chunk write stores the upgrade of its current document (or refuses a changed chunk grid) as before: the only write that upgrades a member document. Removed: the refresh and adoption of consolidated members in `save_metadata` (`_refresh_consolidated`, `_refresh_array`, `_Refreshed`), and `Group.update_attributes_async`'s detour through `save_metadata`. Tests pinning the refresh are replaced by tests that a group write reads nothing and writes only the group's documents, stores the member's consolidated copy byte for byte as stored, that concurrent deletions leave no member, and that the first write through consolidated metadata stores the member's upgrade or refuses a changed grid. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…tadata Setting attributes of an array read from a document that had to be upgraded stored the upgraded document with the new attributes, but left the metadata marked with the document it was read from. A consolidated group handle shares that metadata, so its next write stored the old document in the consolidated metadata and the new attributes were lost from it (zarr 3.4.0 kept them). Every write of an array's own documents (creating, resizing, setting attributes) now goes through `AsyncArray._save_metadata`, which clears the mark afterwards, as the first chunk write does after storing the upgrade. Both clear it through `_stored_document_replaced`, the one place that declares it: the first chunk write clears it also when it stores nothing (the store already holds a valid document, or none), so it cannot be folded into the save. The deep copy in `mark_upgraded` stays: the metadata's attributes share objects with the document it was read from, which the caller also holds. A test pins it. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ment mark Setting attributes on or resizing an array read from a document that needed an upgrade clears the `_stored_document` mark only after the store accepts the new metadata. A test now injects a store failure for the array's own document and checks the mark and the stored bytes survive, for Zarr formats 2 and 3. The `_save_metadata` docstring now says only what the method does. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zarr-developers#4334 moved stored-document leniency into document upgrades (`zarr.core.metadata.upgrades`) and made the metadata constructors strict. Resolution: the zero-chunk-size branch of `_read_stored_regular_chunk_grid` is dropped (upgrades.py owns it now); the mixed-grid branch is kept unchanged so this commit changes no behaviour of zarr-developers#4375, and its warning uses `RESAVE_HINT`. The regular and rectilinear validators range-check with zarr-developers#4334's `parse_chunk_edge`, and two tests expect its message. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

read-the-docs-community · 2026-09-26T20:18:12Z

Documentation build overview

📚 zarr-metadata | 🛠️ Build #34778775 | 📁 Comparing 30d7ca6 against latest (cad2561)

🔍 Preview build

3 files changed

± api/model/index.html
± api/pydantic/index.html
± api/v2/index.html

d-v-b force-pushed the fix/zero-length-single-invariant branch from 7c5688b to 302b638 Compare September 9, 2026 18:28

d-v-b added 4 commits September 9, 2026 20:28

chore: rename changelog fragment to the PR number

921be0d

Assisted-by: ClaudeCode:claude-fable-5-1

d-v-b force-pushed the fix/zero-length-single-invariant branch from 302b638 to 047a92e Compare September 9, 2026 18:28

d-v-b added 2 commits September 12, 2026 14:59

Merge branch 'main' into fix/zero-length-single-invariant

7a2a71a

docs: qualify zero-chunk compatibility history

2297c62

Assisted-by: Codex:GPT-6

d-v-b added this to the 3.5.0 milestone Sep 16, 2026

Merge branch 'main' into fix/zero-length-single-invariant

2873178

d-v-b added 5 commits September 16, 2026 18:11

Merge branch 'main' into fix/zero-length-single-invariant

d868213

Merge branch 'main' into fix/zero-length-single-invariant

4456d80

d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Sep 25, 2026

Merge branch 'fix/zero-length-single-invariant' (zarr-developers#4334)…

e32fd4b

… into fix/mixed-regular-chunk-grid-4374 # Conflicts: # src/zarr/core/metadata/v3.py

d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Sep 25, 2026

Merge branch 'fix/zero-length-single-invariant' (zarr-developers#4334)…

f5ef3df

… into test/rectilinear-declaration-space

d-v-b and others added 3 commits September 25, 2026 12:28

Merge branch 'main' into fix/zero-length-single-invariant

64b3550

d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Sep 25, 2026

Merge branch 'fix/zero-length-single-invariant' (zarr-developers#4334)…

b2e3c99

… into fix/mixed-regular-chunk-grid-4374 # Conflicts: # src/zarr/core/metadata/v3.py

d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Sep 25, 2026

Merge branch 'fix/zero-length-single-invariant' (zarr-developers#4334)…

42230e8

… into test/rectilinear-declaration-space

This was referenced Sep 25, 2026

fix: read mixed regular/rectilinear chunk grids written by zarr 3.2.x #4375

Draft

test: sample the full rectilinear chunk grid declaration space #4377

Draft

d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Sep 25, 2026

Merge branch 'fix/zero-length-single-invariant' (zarr-developers#4334)…

8d672b6

… into fix/mixed-regular-chunk-grid-4374 # Conflicts: # src/zarr/core/metadata/v2.py # src/zarr/core/metadata/v3.py

d-v-b and others added 26 commits September 25, 2026 22:28

test: prefer zero-length axes for stored chunk size 0 in the lifecycl…

9e29333

…e machine An empty write stores no metadata, so it no longer ends the legacy state. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: describe the 4334 changes to metadata constructors for a patch …

2d0121d

…release Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(codecs): pickle a ShardingCodec with an inner chunk size of 0

6fde402

Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test: expect the lifecycle machine's upgrade warning only on non-empt…

f78e115

…y axes Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(group): say what consolidated metadata stores for an upgraded array

65f3f8e

Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This was referenced Sep 26, 2026

fix: make the chunk normalizer the one judge of a chunk specification #4376

Draft

fix(metadata)!: make chunk sizes strict in metadata constructors and chunk specifications #4431

Draft

Navigation Menu

Uh oh!

fix(chunk-grids): one invariant for zero-length axes across model, clamps, and metadata - #4334

fix(chunk-grids): one invariant for zero-length axes across model, clamps, and metadata#4334
d-v-b wants to merge 46 commits into
zarr-developers:mainfrom
d-v-b:fix/zero-length-single-invariant

d-v-b commented Sep 9, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 9, 2026 •

edited

Loading

Uh oh!

TomNicholas commented Sep 16, 2026

Uh oh!

read-the-docs-community Bot commented Sep 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

d-v-b commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design

One invariant, one full-span rule

Stored documents: one lenient module

Strict vs lenient

Writing after an upgraded read

Groups never touch member documents

What users see

3.4.0 compatibility evidence

Tests

Known limitations

Merge order and release

Uh oh!

codecov Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

TomNicholas commented Sep 16, 2026

Uh oh!

read-the-docs-community Bot commented Sep 26, 2026

Documentation build overview

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

d-v-b commented Sep 9, 2026 •

edited

Loading

codecov Bot commented Sep 9, 2026 •

edited

Loading