Skip to content

Add num_chunk_per_batch to job kwargs and use it for zarr v3 shard writes - #4833

Open
samuelgarcia wants to merge 10 commits into
SpikeInterface:mainfrom
samuelgarcia:parallel_executor_grouped_chunks
Open

samuelgarcia wants to merge 10 commits into
SpikeInterface:mainfrom
samuelgarcia:parallel_executor_grouped_chunks

Conversation

@samuelgarcia

@samuelgarcia samuelgarcia commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

See #4831

Totally untested just a draft proof of concept.

@chrishalcrow
@h-mayorquin

@alejoe91 alejoe91 added the concurrency Related to parallel processing label Oct 6, 2026
@alejoe91 alejoe91 added this to the 0.106.0 milestone Oct 6, 2026
@alejoe91 alejoe91 changed the title draft proof of concept of num_chunk_per_batch for the zarr shards Add num_chunk_per_batch ro job kwargs (for the zarr v3 shards) Oct 12, 2026
@alejoe91 alejoe91 changed the title Add num_chunk_per_batch ro job kwargs (for the zarr v3 shards) Add num_chunk_per_batch to job kwargs (for the zarr v3 shards) Oct 12, 2026
@alejoe91
alejoe91 marked this pull request as ready for review October 30, 2026 10:40
@alejoe91

Copy link
Copy Markdown
Member

@samuelgarcia @h-mayorquin the last commit adds some core tests and sets num_chunk_per_job as the number of chunks in a shard, so that each job is going to deal with full shards. I can benchmark memory usage with this approach with respect to the previous "full-shard" one and report back here

@alejoe91 alejoe91 changed the title Add num_chunk_per_batch to job kwargs (for the zarr v3 shards) Add num_chunk_per_batch to job kwargs and use it for zarr v3 shard writes Oct 30, 2026
@alejoe91

Copy link
Copy Markdown
Member

Hi guys, this is ready to review on my end @samuelgarcia @chrishalcrow @h-mayorquin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

concurrency Related to parallel processing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants