feat(services): add S3CompatibleImageFileStorage (first cloud impl of ImageFileStorageBase)

[goanpeca]

Summary

First concrete cloud impl of ImageFileStorageBase. The ABC was introduced in #1650 with cloud storage explicitly named as the design rationale ("if someone wants to use cloud storage for their images, they should be able to replace the image storage service easily."), but no implementation has ever landed.

S3CompatibleImageFileStorage lives at invokeai/app/services/image_files/image_files_s3.py, parallel to image_files_disk.py. It works against AWS S3 by default and any S3-compatible store (Backblaze B2, MinIO, etc.) when s3_endpoint_url is set. It is selected via a new opt-in storage_backend: Literal["disk", "s3"] field on InvokeAIAppConfig, defaulting to "disk", so behavior is unchanged for existing users. No new runtime dep; boto3 is already pulled in by the download service.

To serve images from a backend with no local filesystem path, the read path now streams instead of buffering:

• ImageFileStorageBase gains open_stream() and get_local_path(); both disk and S3 implement them.
• The image-serving routes use FileResponse (sendfile) when a real local path exists (disk) and fall back to a chunked StreamingResponse over open_stream() otherwise (S3). Blocking lookups run in a threadpool, and the inline Content-Disposition is built RFC 5987-safe (header-injection and path-separator hardened).
• Bulk download streams object bytes straight into the zip rather than loading whole images into memory.

Non-ASCII workflow/graph metadata is stored as hex-encoded S3 user-metadata within the S3 metadata size budget.

Related Issues / Discussions

• Closes [enhancement]: Add S3-compatible ImageFileStorage (first cloud impl of ImageFileStorageBase) #9121, closes [enhancement]: Store output images to S3(-like) object storage #1425. Implements the design intent of Add node-based invocation system #1650.

Open Questions

Opening as draft so these can be discussed against concrete code. Carried over from #9121:

1. LRU cache — DiskImageFileStorage keeps an in-process LRU; the S3 impl is stateless. Ship the cache now, or v1 without and follow up after profiling?
2. pil_compress_level — disk reads it from config at save time; S3 currently uses PIL's default. Thread it through?
3. get_path contract — typed Path, but S3 has no real path. We return a synthetic s3://...; long-term answer is presigned URLs. OK as a stub?
4. Follow-up scope — S3CompatibleObjectSerializer (latents) and S3PresignedUrlService (frontend direct-fetch). Keep them as separate PRs?

Testing

Unit / router (no network, runs in CI):

uv run pytest \
  tests/app/services/image_files/test_image_files_s3.py \
  tests/app/routers/test_image_serving.py \
  tests/app/services/bulk_download/test_bulk_download.py \
  tests/test_config.py

• 32 backend tests: save/read/stream/delete, metadata hex round-trip, credential mapping (AWS + B2 convenience names), rollback on partial failure. Uses a hand-rolled _FakeS3Client injected via client= (no new test dep, no moto).
• 10 streaming-serving route tests: FileResponse vs StreamingResponse, 404 mapping, Content-Disposition injection/path-separator safety.
• Bulk-download streaming and storage_backend config selection.

Real-bucket integration (opt-in, gated): tests/integration/test_image_files_s3_real.py runs the full save → read → metadata → delete pipeline against an actual bucket. Marked slow and skipped unless the CI_TEST_S3_* env vars are set, so normal/CI runs never touch a real bucket. Parametrized over AWS-style and B2-style credential env names.

CI_TEST_S3_BUCKET=... CI_TEST_S3_REGION=... \
CI_TEST_S3_ACCESS_KEY_ID=... CI_TEST_S3_SECRET_ACCESS_KEY=... \
# B2: also set CI_TEST_S3_ENDPOINT_URL=https://s3.<region>.backblazeb2.com
uv run pytest -m s3_integration tests/integration/test_image_files_s3_real.py -v

A manual workflow_dispatch CI job (.github/workflows/s3-integration.yml) runs this against repo secrets. We can offer a Backblaze B2 test bucket + scoped key for maintainers to wire up.

QA Instructions

export INVOKEAI_STORAGE_BACKEND=s3
export INVOKEAI_S3_BUCKET=<bucket>
export INVOKEAI_S3_ENDPOINT_URL=<empty for AWS, or e.g. https://s3.us-west-004.backblazeb2.com>
export INVOKEAI_S3_REGION=<region>
# AWS_ACCESS_KEY_ID/SECRET, or B2_APPLICATION_KEY_ID/KEY for B2
invokeai-web

Generate an image, confirm gallery preview + workflow round-trip, confirm images/<name>.png and thumbnails/<name>.webp land in the bucket, delete from gallery and confirm both keys disappear. Smoke-tested against AWS us-east-1 and Backblaze B2 us-west-004.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• ❗Changes to a redux slice have a corresponding migration — N/A
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR) — happy to do this once a release is targeted

[Copilot]

Pull request overview

Adds an opt-in S3-compatible image file storage backend that implements ImageFileStorageBase, parallel to the existing on-disk implementation. The backend is selected via a new storage_backend config field defaulting to "disk", so behavior is unchanged for existing users. boto3 is reused (already a transitive dep), with conveniences for AWS S3, Backblaze B2, and any S3-compatible store.

Changes:

• New S3CompatibleImageFileStorage plus user-metadata-based workflow/graph lookups and a synthetic s3:// get_path stub.
• New storage_backend, s3_bucket, s3_endpoint_url config fields and dispatch in dependencies.initialize().
• Documentation pages (mkdocs and starlight) plus unit tests against a hand-rolled fake S3 client.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
invokeai/app/services/image_files/image_files_s3.py	New S3-compatible image storage implementation.
invokeai/app/services/config/config_default.py	Adds storage_backend, s3_bucket, s3_endpoint_url settings.
invokeai/app/api/dependencies.py	Selects disk vs. S3 image storage at startup.
tests/app/services/image_files/test_image_files_s3.py	Unit tests using a fake S3 client.
tests/test_config.py	Tests for the new storage_backend config field.
docs/src/content/docs/configuration/object-storage.mdx	Starlight docs page for the new backend.
docs-old/configuration/object-storage.md	mkdocs docs page mirror.
mkdocs.yml	Adds the new docs page to navigation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 10 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 8 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 23 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 23 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 23 changed files in this pull request and generated 7 comments.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 4 comments.

[goanpeca]

Thanks for an awesome project!

One honest note: this PR grew a fair bit beyond the original "add an S3 backend" idea. To actually serve images from a backend that has no local filesystem path, I ended up adding a streaming read path (open_stream / get_local_path on ImageFileStorageBase, a FileResponse-vs-StreamingResponse split in the image routes, and chunked bulk-download), plus header-safe metadata handling and some subfolder hardening. It's all in service of the same feature, but it's more surface area than a single "backend" PR usually carries.

If that's more than you'd like to take in one go, I'm very happy to split it into smaller, focused PRs. One natural cut would be landing the streaming-serving refactor first (it stands on its own and also benefits the existing disk backend), then adding S3CompatibleImageFileStorage on top. Totally your call though, just let me know whatever is easiest for you to review and merge.

Either way I'm glad to keep iterating, rebase, or adjust anything you'd like. And the offer still stands to provide a Backblaze B2 test bucket plus a scoped key if you'd like the real-bucket integration test wired into CI.

Thanks again for the time and attention here!

[Pfannkuchensack]

Findings

Low- HEAD /i/{image_name}/full downloads the entire object on the S3 backend. invokeai/app/api/routers/images.py:399 registers a .head route that runs the same _serve_image handler. For a non-local backend, _serve_image calls open_stream and returns a StreamingResponse (invokeai/app/api/routers/images.py:88-92).

Evidence chain:

1. _serve_image resolves no local path for S3 (get_local_path returns None), so it calls run_in_threadpool(images.open_stream, ...), which issues get_object - a full S3 GET - in invokeai/app/services/image_files/image_files_s3.py:152.
2. It then returns StreamingResponse(_iter_and_close(stream)).
3. Starlette's StreamingResponse.stream_response has no HEAD short-circuit - it always iterates self.body_iterator (confirmed in installed starlette 0.46.2, responses.py:238-251). Only FileResponse honors HEAD via send_header_only (responses.py:340).
4. Consequence: every HEAD to /full on S3 issues a full object GET and reads/streams the whole payload - billed S3 egress for a request that is supposed to be header-only. The disk path (FileResponse) does not have this problem, so behavior diverges by backend.

To expose this issue, add a test that issues a HEAD request to /i/{name}/full against an open_stream-only (non-local) backend and asserts the body is not streamed (e.g. open_stream is not invoked, or no body chunks are produced).

That is only needed for install with a Proxy that could cache. The Frontend and Backend does not use a HEAD request.

Medium (test gap) - the most common 404 path, a missing image record, is never tested. In production a bogus image_name (the typical 404) fails at image_records.get() inside images.get_local_path() - the first call in _serve_image - raising ImageRecordNotFoundException, which invokeai/app/api/routers/images.py:422 catches and turns into 404. But every router test in invokeai/tests/app/routers/test_image_serving.py sets images.get_local_path.return_value = None and only ever raises ImageFileNotFoundException from open_stream. The get_local_path-raises-ImageRecordNotFoundException branch (the real missing-image case) is unverified, and so is open_stream raising ImageRecordNotFoundException.

To expose this issue, add a test that makes images.get_local_path raise ImageRecordNotFoundException and asserts the route returns 404 (and that open_stream is never called).

Low - S3-backed responses are emitted without Content-Length (chunked), unlike the disk and previous behavior. The old handler used Response(content, ...), which sets Content-Length; the disk path now uses FileResponse, which also sets it. The S3 path uses StreamingResponse over a boto StreamingBody (invokeai/app/api/routers/images.py:91) with no length, so responses go out Transfer-Encoding: chunked. The get_object response already carries ContentLength, but it is discarded in invokeai/app/services/image_files/image_files_s3.py:159. Consequence: clients lose payload size up front (download-progress UI, some proxy/caching paths), and HEAD responses report no size. Severity is low (functional content still served).

To expose this issue, add a test asserting a Content-Length header is present on a served image; it passes for the disk/FileResponse path and fails for the S3/streaming path.

Open Questions

• save() rollback can delete a pre-existing object on a same-name overwrite. In invokeai/app/services/image_files/image_files_s3.py:246-250, if the main put_object succeeds (overwriting an existing key) and the thumbnail put_object then fails, _delete_quietly(key, thumb_key) deletes both keys - destroying what was a previously valid image. This is only reachable if save() is ever called twice with the same image_name. Image names are freshly generated UUIDs, so I could not establish a real trigger; flagging rather than asserting a defect. If re-save with an existing name is reachable anywhere, this becomes a real data-loss finding.

• Residual: switching storage_backend does not migrate existing images (old images 404 after switch) - operational, should be documented. The S3 read path issues two image_records.get() lookups per /full request (once in get_local_path, once in open_stream); minor and not worth blocking on.

[lstein]

@goanpeca There are comments on this PR that need to be addressed. Are you still actively working on it?

[goanpeca]

Hi @lstein yes! will review them now :)

Sorry for the delay!

[goanpeca]

@Pfannkuchensack thanks for the review.

HEAD /full on S3

Fixed. HEAD /i/{image_name}/full now uses a header-only path for non-local backends and calls get_content_length() instead of open_stream(), so it does not issue an S3 get_object or stream the body. Added a regression test asserting open_stream is not called for HEAD.

Missing image record 404 test gap

Fixed. Added coverage for images.get_local_path() raising ImageRecordNotFoundException, asserting the route returns 404 and open_stream() is not called. Also covered open_stream() raising ImageRecordNotFoundException.

S3 responses missing Content-Length

Fixed. S3 open_stream() now preserves the object ContentLength, and the image-serving route sets Content-Length on streamed responses when available. Added tests asserting the header is present for the non-local streaming path.

Open Question: rollback can delete pre-existing object

Fixed. S3 saves are now create-only using IfNoneMatch="*", and rollback only deletes keys that the current save call successfully created. Added regression coverage to ensure a same-name save cannot overwrite or delete pre-existing image/thumbnail objects.

Open Question: switching storage_backend does not migrate existing images

Documented. Added an operational note clarifying that changing storage_backend changes where reads/writes look, but does not migrate existing files between disk and the bucket.

Residual: duplicate image_records.get() lookup on S3 GET

Left as-is. This is a minor extra SQLite lookup and not worth adding new service API shape in this PR unless we see measurable impact or want to do a broader image-serving refactor.

[Pfannkuchensack]

Picking this up to get it merge-ready. Summary of what I changed and why:

Merged main and resolved conflicts. Conflicts were in config_default.py,
test_config.py, and uv.lock, all from independent additions on both sides —
resolved by keeping both (the S3 validator/tests alongside main's base_url
proxy validator/tests) and regenerating uv.lock so boto3 and its deps land
cleanly.

Fixed a startup crash introduced by the merge. main added three abstract
methods to ImageFileStorageBase (image_root, thumbnail_root,
evict_cache_paths) as part of the image-subfolder-move feature. The S3 backend
predated them, so after the merge it could no longer be instantiated
(Can't instantiate abstract class ...) and the server failed to boot with
storage_backend="s3". Implemented them on the S3 backend: synthetic s3://
roots (identification only, mirroring get_path) and a no-op
evict_cache_paths (this backend keeps no local image cache).

Disabled the subfolder feature for S3. Object storage has no directory
structure and there is no in-place move/reorganize operation for it, so the
filesystem-only image-move service (os.replace/fsync/empty-dir cleanup)
can't work on S3, and switching image_subfolder_strategy would strand existing
objects. For storage_backend="s3":

• ImageMoveService is not created (the image_moves API returns 503),
• writes are forced flat regardless of image_subfolder_strategy,
• the strategy selector and the Image Storage Maintenance panel are hidden in
  Settings.

Docs. Documented that storage_backend is a deploy-time choice with no
migration between disk and s3 (switching orphans the gallery), that S3 is
intended for hosted/multi-replica deployments decoupling output from a single
server's filesystem, and that image_subfolder_strategy / storage maintenance
are disk-only. Corrected the now-inaccurate object-layout section and regenerated
the settings reference.

All backend gates, frontend tsc/lint, and the docs build (internal links
included) pass.

I have tested all the PR offers. Works fine, is a bit slow on the save but that was to be expected.
Test with a docker mimio instance.