atelet: parallel snapshot upload - #1143
Open
Benjamin Elder (BenTheElder) wants to merge 2 commits into
Open
Conversation
A single stream to GCS tops out near 100 MiB/s no matter how it is chunked, and for a memory-heavy actor the snapshot upload is most of a suspend. Measured from a pod on a GKE worker node against the snapshot bucket, 300 MiB through this same non-seekable streaming path: one stream, 16-128 MiB chunks 82-107 MiB/s 2 parts + compose 150-163 MiB/s 4 parts + compose 233-257 MiB/s 8 parts + compose 224-231 MiB/s So an object past 64 MiB is now cut into 32 MiB parts, uploaded concurrently and composed server-side, matching what the S3 path already does with the upload manager (same part size and concurrency constants, so both backends peak at 128 MiB of buffering). The parts are byte ranges of the SAME stream rather than independently produced pieces, so the composed object is byte-identical to what one PutObject would have written: the sparse-extent format and the whole download path are unaffected. An earlier comment in this file called parallel parts "a format change" -- that was wrong and is corrected here. Small objects are left alone, because they lose: at 24 MiB (an idle golden) one request measured 258-314 ms against 310-410 ms for 2 parts and 332-407 ms for 4, the round trips no longer amortised. The crossover was bracketed by those two sizes rather than measured, so the threshold sits at the conservative end. Tested against a GCS emulator (fake-gcs-server, which implements compose): byte-exact round trips at 3 MiB, exactly the threshold, one byte past it, and several parts with a partial last part -- plus an assertion that no part or intermediate objects are left behind.
The memory-ranges upload is ~75% of a micro-VM suspend (the rest of a
checkpoint -- pause, CH snapshot, teardown -- is 0.39s). Piping one
compressed stream into a part splitter left both halves serialized:
zstd's streaming encoder only compresses one block at a time however
much concurrency it is given, and the parts could not start until that
one producer had made them.
Fan the whole pipeline out by file range instead. Each part reads its own
extents with ReadAt, compresses them, and streams into its own object
over its own connection, so every stream starts at once; the parts are
consecutive byte ranges of the same sparse-extent stream, so the composed
object is exactly what the single-stream writer produced and the download
path is untouched.
Three things this needed:
* Independent zstd FRAMES per chunk. Concatenated frames decode as one
stream, so parallel compression needs no format change -- and it costs
no ratio either: re-splitting one real memory image into 1/2/3/6/12/24
frames moved the compressed size by -0.3% to +0.9%, which is noise.
* A client per part. One storage.Client keeps a single HTTP/2
connection per host and multiplexes every part onto it: at 8 streams
that is 334 MB/s against 518 MB/s with a connection each.
* 64 MiB of source per part. Smaller parts stop amortizing the ~0.25s a
part costs before it transfers, and double the compose.
Measured end to end on a GKE worker node, median of the runs, suspending
an actor with a 541 MB working set:
single stream 4.66s
parts from one compressed stream 3.60s
range-parallel 1.51s
A small actor stays under the split threshold and takes the unchanged
single-request path, but still gains from the parallel compressor: a
~100 MB image went 0.79s -> 0.62s.
Levels above SpeedFastest are not worth it: re-compressing real guest
memory images at default/better gives up 6-9% of size for more time than
the smaller upload saves.
Benjamin Elder (BenTheElder)
force-pushed
the
atelet-parallel-snapshot-upload
branch
from
August 22, 2026 05:43
bcc2a94 to
072385d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GCS doesn't offer something like the S3 upload manager package (#1068) but the same thing holds ... uploading decent sized chunks in parallel can really speed up snapshot upload, we just have to implement it ourselves.
This PR implements parallel uploading for GCS.
Along the way, I discovered that atelet's current approach to compression bottlenecks parallel upload, so the second commit splits that into chunks as well.
For some test data, we wind up with epsilon the same bytes than before but with a 451 MB working set we go from 4.66s (before this PR) to 1.51s (after both commits).
For a tiny counter demo with snapshot mode full (NOTE: the default counter demo we use for e2e uses only data mode for uploads, and full only for pause), we go from 0.79s to 0.62s (22% faster), purely from the parallelized compression (second commit) as it stays under the upload chunk size.
Also relevant: #1130