perf(service): parallelize intra-task tile merge (MAPCO-11325) - #260
Open
shimoncohen wants to merge 1 commit into
Open
perf(service): parallelize intra-task tile merge (MAPCO-11325)#260shimoncohen wants to merge 1 commit into
shimoncohen wants to merge 1 commit into
Conversation
The service merged tiles one coord at a time in a sequential x/y loop. Merge is the hot path and each coord is independent, so run each batch's coords through Parallel.ForEach, one chunk at a time. - Coords are enumerated per bounds and processed in chunks of batchMaxSize. Each chunk is merged in parallel into a ConcurrentBag, then flushed to the target before the next chunk starts, so peak memory stays bounded to ~one chunk (count-based; with the default batchMaxSize=1000 this is ~the previous 20MB byte cap for jpeg tiles). - Within a chunk the target is only read (writes happen after the parallel phase), and GetCorrespondingTile is thread-safe per data source, so the merge relies on the thread-safe GpkgClient connection from MAPCO-11320. - Degree of parallelism is configurable via GENERAL:parallel:numOfThreads (0/unset -> Environment.ProcessorCount). Depends on and is stacked over MAPCO-11320 (logic-1-gpkgclient-perf). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
shimoncohen
marked this pull request as ready for review
August 5, 2026 20:20
shimoncohen
requested review from
CL-SHLOMIKONCHA,
almog8k,
asafmas-rnd,
razbroc and
syncush
August 5, 2026 20:20
asafmas-rnd
approved these changes
Aug 6, 2026
Contributor
|
Check that Parallel doesn't call other parallel's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SVC-1 of MAPCO-11318. Stacked on #255 (MAPCO-11320) — base branch is
logic-1-gpkgclient-perf, not master, so this diff shows only the SVC-1 change. Retarget to master after #255 merges.Change
The service merged tiles sequentially in a nested x/y loop. Merge is the hot path and each coord is independent, so:
batchMaxSize, merging each chunk's coords withParallel.ForEachinto aConcurrentBag, then flushing the chunk to the target before starting the next.batchMaxSize=1000, that's ≈ the previous 20 MB byte cap for jpeg tiles. Per the design decision, the precise byte cap is approximated by the count cap.GetCorrespondingTileis thread-safe per data source — so this relies on the thread-safeGpkgClientconnection from perf(gpkg): connection reuse + prepared bulk-insert + keyset paging (MAPCO-11320) #255.GENERAL:parallel:numOfThreads(0/unset →Environment.ProcessorCount).Scope — delivers P9 of MAPCO-11325 only
MAPCO-11325 scopes four points; this PR implements P9 (parallelize the intra-task merge). The rest remain open in that ticket and are intentionally out of scope here:
MainRunner: one task at a time process-wide → per-type bounded workers). Job-state/heartbeat complexity; separate change.Separately, the count-based memory cap here is an approximation of the old byte-accurate 20 MB bound; tracked as its own follow-up in MAPCO-11367.
Testing
Full suite: 1142 passed, 0 failed. The batch-flush test now uses dense bounds and asserts count-based (chunked) flushing.
Depends on
🤖 Generated with Claude Code