refactor(datasets): use the shared scverse-misc dataset registry + downloader by timtreis · Pull Request #1213 · scverse/squidpy

timtreis · 2026-06-15T11:09:06Z

Replaces squidpy's internal pooch-based dataset registry/downloader with the shared infrastructure from scverse-misc (scverse-misc[datasets]>=0.1.1).

Public API unchanged: sq.datasets.* keep the same signatures and behavior.
~750 lines of duplicated registry/download code deleted; pooch is now transitive.
datasets.yaml stays in squidpy; the anndata / image / visium_10x loaders register via register_loader.

Replace squidpy's internal pooch-based registry/downloader with the shared scverse_misc.datasets system (scverse-misc[datasets]): - _registry.py: build a scverse_misc DatasetRegistry from datasets.yaml, folding squidpy-specific shape/library_id into the generic metadata mapping. Drops squidpy's duplicated FileEntry/DatasetEntry/DatasetRegistry/DatasetType. - _downloader.py: register squidpy's domain loaders (image -> ImageContainer, visium_10x -> read.visium, spatialdata -> read_zarr) via register_loader and override the built-in anndata loader for the shape warning. The pooch download/verify/extract machinery now lives in scverse-misc. - _datasets.py: public API unchanged; type dispatch uses plain strings. - pyproject: drop direct pooch dep (now via scverse-misc[datasets]). Net ~750 lines deleted. Public API (sq.datasets.*) is unchanged. Depends on scverse/scverse-misc#40. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scverse-misc now ships a generic spatialdata loader, so squidpy no longer needs its own; it registers only its domain loaders (image, visium_10x) plus the anndata shape-warning override. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@internet

Drop the redundant 'visium' prefix in visium() so downloads land in <datasetdir>/visium_10x/<sample>/ like every other type (was doubly nested under visium/visium_10x). Update the hires-image path assertion accordingly. Verified: all @internet datasets tests pass (8 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

registry.{anndata,image,spatialdata}_datasets were squidpy's old registry properties, removed in the migration. Use dataset_names(type) instead. Visium samples now cache to <datasetdir>/visium_10x/<sample>/ via the public API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scverse-misc dropped its DatasetRegistry/Fetcher/FetchContext classes for a typed data model + functions. Adapt: - _registry: parse_registry() -> (base_url, dict[str, DatasetEntry]); get_registry() returns the dict, get_base_url() the base. shape/library_id/doc_header now live in entry.metadata. - _downloader: loaders are (entry, target, download, **kwargs); DatasetDownloader wraps fetch(); visium uses pooch.Untar instead of a manual tarfile loop. - _datasets/tests updated to the dict + metadata shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

for more information, see https://pre-commit.ci

codecov · 2026-06-23T23:18:20Z

Codecov Report

❌ Patch coverage is 85.29412% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.13%. Comparing base (ac08971) to head (1f3d061).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/squidpy/datasets/_downloader.py	80.95%	6 Missing and 2 partials ⚠️
src/squidpy/datasets/_registry.py	87.50%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1213      +/-   ##
==========================================
+ Coverage   77.08%   77.13%   +0.05%     
==========================================
  Files          63       63              
  Lines        9368     9202     -166     
  Branches     1576     1549      -27     
==========================================
- Hits         7221     7098     -123     
+ Misses       1545     1520      -25     
+ Partials      602      584      -18

Files with missing lines	Coverage Δ
src/squidpy/datasets/_datasets.py	`87.93% <100.00%> (+6.39%)`	⬆️
src/squidpy/datasets/_registry.py	`85.71% <87.50%> (-0.53%)`	⬇️
src/squidpy/datasets/_downloader.py	`80.39% <80.95%> (+8.63%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- replace DatasetDownloader class + get_downloader singleton + module wrapper with a single download() over scverse_misc.datasets.fetch - restore pooch as a direct dependency (used via pooch.Untar in the visium loader); drop now-unused direct pyyaml dependency - drop redundant visium() name validation, keeping its specific message

for more information, see https://pre-commit.ci

selmanozleyen · 2026-06-26T06:59:23Z

+        raise ValueError(f"Unknown Visium sample: {sample_id}. Available samples: {dataset_names('visium_10x')}")

-    return downloader.download(sample_id, base_dir, include_hires_tiff=include_hires_tiff)
+    # downloads land in <datasetdir>/visium_10x/<sample_id>/


this used to be /x/visium/x/ on old code but it's fine I guess to move the cache path. It's also seen in

- expected_image_path = (Path(settings.datasetdir) / "visium" / sample / "image.tif").resolve() + expected_image_path = (Path(settings.datasetdir) / "visium_10x" / sample / "image.tif").resolve()

but good to document

selmanozleyen · 2026-06-26T07:04:15Z

-    def visium_datasets(self) -> list[str]:
-        """Return names of all Visium datasets (alias for visium_10x_datasets)."""
-        return self.visium_10x_datasets
+def get_registry() -> dict[str, DatasetEntry]:


now we can have a prettier name for _parsed() and get rid of these two functions. No need to have two separate functions get_registry() and get_base_url()

selmanozleyen · 2026-06-26T07:05:19Z

-    def visium_datasets(self) -> list[str]:
-        """Return names of all Visium datasets (alias for visium_10x_datasets)."""
-        return self.visium_10x_datasets
+def get_registry() -> dict[str, DatasetEntry]:


dict[str, DatasetEntry]: should be MappingProxy because we don't want it to be mutable

selmanozleyen · 2026-06-26T07:05:50Z

-  "spatialdata>=0.7.2",          # 0.7.2 dropped xarray-schema (pkg_resources break, #1115)
+  # dataset registry + downloader now live in scverse-misc
+  "scverse-misc[datasets]>=0.1.1",
+  "spatialdata>=0.7.2",            # 0.7.2 dropped xarray-schema (pkg_resources break, #1115)


why the diff here, can you undo it?

"spatialdata>=0.7.2", # 0.7.2 dropped xarray-schema (pkg_resources break, #1115)

selmanozleyen · 2026-06-26T07:27:35Z

+    if sample_id not in get_registry():
+        raise ValueError(f"Unknown Visium sample: {sample_id}. Available samples: {dataset_names('visium_10x')}")


valid-but-wrong-type name (e.g. imc, an AnnData dataset) passes the guard, then fails deep inside the anndata loader with a confusing TypeError: read_h5ad() got an unexpected keyword argument 'include_hires_tiff' instead of a clear error.

Suggested change

if sample_id not in get_registry():

raise ValueError(f"Unknown Visium sample: {sample_id}. Available samples: {dataset_names('visium_10x')}")

visium_samples = dataset_names("visium_10x")

if sample_id not in visium_samples:

raise ValueError(f"Unknown Visium sample: {sample_id}. Available samples: {visium_samples}")

selmanozleyen · 2026-06-26T07:28:59Z

+
+    def test_unknown_dataset(self):
+        assert "nonexistent" not in get_registry()
+


Maybe let's add a test like this given that it fails

class TestVisiumValidation: """Offline validation of the ``visium()`` sample-name guard (no network).""" def test_unknown_sample_raises(self): with pytest.raises(ValueError, match="Unknown Visium sample"): visium("definitely_not_a_sample") def test_valid_but_wrong_type_rejected(self): # `imc` is a valid registry entry but an AnnData dataset, not a Visium 10x # sample. It must be rejected before any download is attempted. with pytest.raises(ValueError, match="Unknown Visium sample"): visium("imc")

selmanozleyen

See my comments. But this is a nice win!

- visium(): guard against the visium_10x names specifically so a valid-but- wrong-type name (e.g. "imc") fails with a clear error instead of dying deep in the anndata loader with an "unexpected keyword argument" TypeError - get_registry(): return a read-only MappingProxyType so the lru_cached singleton can't be mutated by callers; rename _parsed() -> _load() - cap scverse-misc[datasets] to <0.2 (0.1.x API still churning) - document the <datasetdir>/<type>/ cache-path change (visium/->visium_10x/, images/->image/) in the release notes - restore the include_hires_tiff metadata-toggle test; add offline regression tests for the visium guard; comment the intentional global anndata override

for more information, see https://pre-commit.ci

selmanozleyen · 2026-07-07T13:46:38Z

We don't really use this file. Maybe we should remove it? Or make a note each release to somehow add these notes into the autogenerated logs

selmanozleyen

LGTM

timtreis and others added 5 commits June 24, 2026 01:01

timtreis force-pushed the feat/datasets-via-scverse-misc branch from fd53f80 to 947e68e Compare June 23, 2026 23:02

[pre-commit.ci] auto fixes from pre-commit.com hooks

07387e4

for more information, see https://pre-commit.ci

timtreis and others added 2 commits June 24, 2026 01:45

[pre-commit.ci] auto fixes from pre-commit.com hooks

c517bbd

for more information, see https://pre-commit.ci

timtreis marked this pull request as ready for review June 23, 2026 23:45

timtreis requested a review from selmanozleyen June 24, 2026 08:52

selmanozleyen reviewed Jun 26, 2026

View reviewed changes

selmanozleyen requested changes Jun 26, 2026

View reviewed changes

timtreis and others added 4 commits July 7, 2026 14:28

Merge branch 'main' into feat/datasets-via-scverse-misc

b04a031

[pre-commit.ci] auto fixes from pre-commit.com hooks

6b3561a

for more information, see https://pre-commit.ci

Merge branch 'main' into feat/datasets-via-scverse-misc

bb79072

selmanozleyen reviewed Jul 7, 2026

View reviewed changes

timtreis requested a review from selmanozleyen July 7, 2026 14:28

selmanozleyen approved these changes Jul 7, 2026

View reviewed changes

chore(datasets): drop notes-dev.md changelog entry (no longer used)

1f3d061

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor(datasets): use the shared scverse-misc dataset registry + downloader#1213

refactor(datasets): use the shared scverse-misc dataset registry + downloader#1213
timtreis wants to merge 13 commits into
scverse:mainfrom
timtreis:feat/datasets-via-scverse-misc

timtreis commented Jun 15, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jun 23, 2026 •

edited

Loading

Uh oh!

selmanozleyen Jun 26, 2026

Uh oh!

selmanozleyen Jun 26, 2026

Uh oh!

selmanozleyen Jun 26, 2026

Uh oh!

selmanozleyen Jun 26, 2026 •

edited

Loading

Uh oh!

selmanozleyen Jun 26, 2026

Uh oh!

selmanozleyen Jun 26, 2026

Uh oh!

selmanozleyen left a comment

Uh oh!

selmanozleyen Jul 7, 2026

Uh oh!

selmanozleyen left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		if sample_id not in get_registry():
		raise ValueError(f"Unknown Visium sample: {sample_id}. Available samples: {dataset_names('visium_10x')}")

-    if sample_id not in get_registry():
-        raise ValueError(f"Unknown Visium sample: {sample_id}. Available samples: {dataset_names('visium_10x')}")
+    visium_samples = dataset_names("visium_10x")
+    if sample_id not in visium_samples:
+        raise ValueError(f"Unknown Visium sample: {sample_id}. Available samples: {visium_samples}")


		def test_unknown_dataset(self):
		assert "nonexistent" not in get_registry()

Uh oh!

Conversation

timtreis commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

selmanozleyen Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

selmanozleyen Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

selmanozleyen Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

selmanozleyen Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

selmanozleyen Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

selmanozleyen Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

selmanozleyen left a comment

Choose a reason for hiding this comment

Uh oh!

selmanozleyen Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

selmanozleyen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

timtreis commented Jun 15, 2026 •

edited

Loading

codecov Bot commented Jun 23, 2026 •

edited

Loading

selmanozleyen Jun 26, 2026 •

edited

Loading