Conversation
A SCRIP file stores one (lon, lat) pair per face-corner, so the corner table is n_face * n_corners rows even though adjacent faces share almost every vertex. The reader deduped that table with Polars, which needs the whole thing resident: 53.6 GiB of float64 for a 300M-element np4 grid, before the join copy. That is why large np4 SCRIP grids fail to open. When the grid is opened with chunks=, dedup with a dask shuffle instead, which spills partition-by-partition, and build the inverse index lazily with map_blocks against the (small) unique-node table. The eager path is unchanged.
Without a distributed client dask picks an in-memory shuffle. On the 226M-face ne2048np4 grid that peaked at 17.9 GiB against 12.1 GiB for the disk shuffle, for the same answer. Pin it, but only when the caller has not chosen a method and no client is running, since p2p is the better choice on a real cluster. Also drop the intermediate DataFrame once its two columns are out. The unique-node table is 3.4 GiB at this size, so holding it past its last use is not free. Together these take the full open from ~31.0 to ~29.3 GiB peak (3 runs each).
The block lookup relied on a sort-after-join to keep each block aligned with its inputs. Polars' own docs say not to rely on an observed join order without requesting one, so request it with maintain_order="left" and check the row count survived. Misalignment here would not raise -- it would build every face from the wrong corners. Lift the closure to module level so that contract is testable, and widen the tests: parametrize over chunks= spellings including "auto" (what callers actually pass), compare corners in winding order rather than sorted, cover the radians path, and assert the connectivity is still lazy after open so a future .compute() in the reader fails here rather than only on a multi-GB file.
This was referenced Sep 19, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1774
Overview
Dedups with a dask shuffle and builds the inverse index lazily via
map_blockswhen corners are dask-backed; eager path unchanged. A 24 GB grid that killed eager opens in ~60 s, and the dedup stage runs on the 63 GB np4 grid in 135 s at 38 GiB peak. Unblocks the MCP server path with no server change.Expected Usage
General
Testing & Benchmarking
Documentation and Examples
docs/api.rst; internal (private) function names start with an underscore (_)AI Disclosure
AI Usage: