Repository navigation

[feature](cloud) Spill to object storage in cloud mode and report the traffic in SHOW DATA - #68032

Open

mrhhsg wants to merge 23 commits into

apache:masterfrom

mrhhsg:feat/cloud-spill-s3

Commits on Sep 24, 2026

[feature](cloud) Spill to object storage in cloud mode and report the traffic in SHOW DATA
mrhhsg
committed
4ae9f70
[feature](cloud) Key remote spill by backend_id and address the review of the S3 spill path
mrhhsg
committed
2b3ffe0
[refactor](spill) Split SpillDataDir into local and remote stores
mrhhsg
committed
1c2530a
[fix](spill) Charge the upload budget by buffer capacity, refuse cancelled queries, drop spill of deleted instances
mrhhsg
committed
9380aff
[fix](spill) Namespace remote spill by instance and pair the upload gate with every buffer
mrhhsg
committed
9f5d145
[fix](spill) Show the spill data currently held in object storage instead of the upload total
mrhhsg
committed
4e6c701
[fix](spill) Order spill stats reports, drop the ungated empty-object fallback, bound the instance lookup
mrhhsg
committed
2b0a0d1
[fix](spill) Make the instance-id lookup back off for a minute and harden the reporter test
mrhhsg
committed
d6496d3
[fix](spill) Close spill parts synchronously and let every FE cache the spill size for SHOW DATA
mrhhsg
committed
7bd0218
[refactor](spill) Use the bound remote store member in SpillFileManager
mrhhsg
committed
843d3a4
[refactor](spill) Poll the spill size for SHOW DATA from the BEs and drop the meta-service spill stats
mrhhsg
committed
ce151a6
[feature](spill) Record remote spill bytes in the audit log and list the spill size as a SHOW DATA row
mrhhsg
committed
bef9437
[refactor](spill) Lay out remote spill objects as spill/{host}/{query_id}/...
mrhhsg
committed
93681e5
[refactor](spill) Name the remote spill directory of a BE {ip}_{port}
mrhhsg
committed
289b6d5
[fix](be) Release local states before the object pool when a RuntimeState is destroyed
mrhhsg
committed
d598b91
[improvement](spill) Coalesce adjacent blocks into one GET when reading remote spill files
mrhhsg
committed
0bad0c5
[fix](spill) Poll the remote spill stats on their own schedule
mrhhsg
committed
f3a5d53
[fix](spill) Fence the spill recycler with a BE heartbeat and keep undeleted spill charged
mrhhsg
committed
0217179
[fix](spill) Delete a directory marker under spill/ alone and count the spill sweep in the recycler statistics
mrhhsg
committed
71a21e2
[refactor](spill) Delete remote spill only when its SpillFile is destroyed
mrhhsg
committed
422b839

Commits on Sep 28, 2026

Commits on Oct 7, 2026

[test](cloud) Cover remote spill billing and statistics paths
mrhhsg
committed
7ad11da

Navigation Menu

Repository navigation

[feature](cloud) Spill to object storage in cloud mode and report the traffic in SHOW DATA - #68032

[feature](cloud) Spill to object storage in cloud mode and report the traffic in SHOW DATA#68032
mrhhsg wants to merge 23 commits into
apache:masterfrom
mrhhsg:feat/cloud-spill-s3

Commits on Sep 24, 2026

[feature](cloud) Spill to object storage in cloud mode and report the traffic in SHOW DATA

[feature](cloud) Key remote spill by backend_id and address the review of the S3 spill path

[refactor](spill) Split SpillDataDir into local and remote stores

[fix](spill) Charge the upload budget by buffer capacity, refuse cancelled queries, drop spill of deleted instances

[fix](spill) Namespace remote spill by instance and pair the upload gate with every buffer

[fix](spill) Show the spill data currently held in object storage instead of the upload total

[fix](spill) Order spill stats reports, drop the ungated empty-object fallback, bound the instance lookup

[fix](spill) Make the instance-id lookup back off for a minute and harden the reporter test

[fix](spill) Close spill parts synchronously and let every FE cache the spill size for SHOW DATA

[refactor](spill) Use the bound remote store member in SpillFileManager

[refactor](spill) Poll the spill size for SHOW DATA from the BEs and drop the meta-service spill stats

[feature](spill) Record remote spill bytes in the audit log and list the spill size as a SHOW DATA row

[refactor](spill) Lay out remote spill objects as spill/{host}/{query_id}/...

[refactor](spill) Name the remote spill directory of a BE {ip}_{port}

[fix](be) Release local states before the object pool when a RuntimeState is destroyed

[improvement](spill) Coalesce adjacent blocks into one GET when reading remote spill files

[fix](spill) Poll the remote spill stats on their own schedule

[fix](spill) Fence the spill recycler with a BE heartbeat and keep undeleted spill charged

[fix](spill) Delete a directory marker under spill/ alone and count the spill sweep in the recycler statistics

[refactor](spill) Delete remote spill only when its SpillFile is destroyed

Commits on Sep 28, 2026

[fix](spill) Discard the writer of a destroyed spill file and close Iceberg spill writers explicitly

[fix](spill) Bound the remote footer probe by the coalesced read size

Commits on Oct 7, 2026

[test](cloud) Cover remote spill billing and statistics paths