Skip to content

refactor(storage): migrate S3 clients to tigrisdata/storage-go - #949

Open
Xe wants to merge 4 commits into
masterfrom
Xe/storage-go-migration
Open

refactor(storage): migrate S3 clients to tigrisdata/storage-go#949
Xe wants to merge 4 commits into
masterfrom
Xe/storage-go-migration

Conversation

@Xe

@Xe Xe commented Jun 18, 2026

Copy link
Copy Markdown
Owner

Summary

Migrates all AWS S3 SDK v2 client construction in the repo to github.com/tigrisdata/storage-go and deletes the now-redundant within.website/x/tigris helper package. The library's storage.Client embeds *s3.Client, so standard S3 operations (PutObject/GetObject/HeadObject/DeleteObject/ListObjectsV2) work unchanged — most call sites only needed a type swap. Its tigrisheaders subpackage already provides verbatim copies of every helper (and the Region type) that the deleted tigris package exposed, none of which were used anywhere beyond tigris.Client.

storage.New credential resolution is backward-compatible: it uses TIGRIS_STORAGE_ACCESS_KEY_ID/_SECRET_ACCESS_KEY when set, otherwise falls back to AWS LoadDefaultConfig.

Per-file changes

Low-level storage.Client (storage.New(...)):

  • store/s3api.goWithGlobalEndpoint()
  • autocert/s3cache/s3cache.goWithGlobalEndpoint(), WithPathStyle(true)
  • cmd/relayd/telemetry.goWithGlobalEndpoint(), WithPathStyle(true); also drops the dead --telemetry-path-style flag (construction always hardcoded path style)
  • cmd/uploud/main.goWithGlobalEndpoint(), WithPathStyle(true)
  • cmd/future-sight/main.goWithEndpoint/WithRegion/WithPathStyle/WithAccessKeypair from existing flags
  • cmd/xedn/uplodr/main.go — Tigris client via WithFlyEndpoint(); Backblaze B2 client via WithEndpoint/WithRegion/WithPathStyle/WithAccessKeypair

High-level simplestorage:

  • cmd/stickers/main.go — switched to storage-go/simplestorage: Head for the existence probe and PresignURL(http.MethodGet, key, time.Hour) for the redirect URL. Deletes the bespoke Presigner type and all AWS SDK imports. Kept WithFlyEndpoint() so presigned URLs retain the xedn.fly.storage.tigris.dev host that the files.xeiaso.net rebrand rewrites.

Removed: tigris/tigris.go (package deleted).

Deps: go.mod/go.sumtigrisdata/storage-go promoted to a direct dependency; aws-sdk-go-v2/config and aws-sdk-go-v2/credentials demoted to indirect.

Behavioral notes / decisions

  • Endpoint pinning: clients that previously took their endpoint from the AWS_ENDPOINT_URL_S3 env var now pin the global endpoint (t3.storage.dev). uplodr's Tigris client keeps the fly endpoint it already used. This was a deliberate choice; it removes runtime endpoint override-ability for those services.
  • future-sight loses its custom AppID user-agent and verbose ClientLogMode request logging, which storage.New does not expose (observability only — no functional impact).
  • fix(confyg) is included as a prerequisite: a pre-existing go vet error (%#q applied to the int from peekRune) was failing the repo's pre-commit npm test hook and blocking all commits. Fixed by printing the offending character with %q on an explicit rune conversion.

⚠️ Verify before merge/deploy

autocert/s3cache (used only by sakurajima's autocert cache) now hardcodes t3.storage.dev. If sakurajima previously pointed its cert cache at a different endpoint (e.g. fly.storage.tigris.dev) via AWS_ENDPOINT_URL_S3, the existing cert cache would appear empty and certs would be re-issued (Let's Encrypt rate-limit risk). The org standard appears to be t3.storage.dev (per kube/alrest/gitea/gitea-config.yaml), so this is likely low-risk — but it can't be confirmed from code alone.

Testing

  • go build ./... — clean
  • go vet ./... on all migrated packages — clean
  • go test ./... — passes (full suite runs via the pre-commit hook on every commit)
  • A cynical review verified against the library source that: storage.New injects no always-on Tigris middleware (B2/MinIO safe), simplestorage.Head propagates 404 errors (not-found fallback intact), and PresignURL + WithFlyEndpoint produces the virtual-hosted host the URL rebrand depends on.

🤖 Assisted-by: Claude Opus 4.8 via Claude Code

Xe added 4 commits June 18, 2026 12:51
go vet rejected the %#q verb applied to the int returned by peekRune.
Print the offending character with %q on an explicit rune conversion so
the error reads as a quoted character literal and vet is satisfied.

Assisted-by: Claude Opus 4.8 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
…orage-go

Route low-level S3 client construction through
github.com/tigrisdata/storage-go (storage.Client embeds *s3.Client, so
call sites are unchanged) and delete the now-redundant
within.website/x/tigris package, whose helpers and Region type are
provided verbatim by storage-go/tigrisheaders.

Tigris clients that previously resolved their endpoint from the
AWS_ENDPOINT_URL_S3 env var now pin the global endpoint
(t3.storage.dev); uplodr's Tigris client keeps the fly endpoint it
already used. The Backblaze B2 and future-sight clients move to
storage.New with explicit endpoint/region/path-style/keypair options.

While migrating relayd, drop the telemetry-path-style flag: it was
already dead (construction hardcoded path style) and remains so.

future-sight loses its custom AppID user-agent and verbose
ClientLogMode logging, which storage.New does not expose.

Assisted-by: Claude Opus 4.8 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
…presigning

Replace the low-level S3 client, the bespoke Presigner type, and the
HeadObject existence probe with the high-level
storage-go/simplestorage client: Head for existence checks and
PresignURL for the redirect URL. This drops every AWS SDK import from
the command.

Keep the fly endpoint so presigned URLs retain the
xedn.fly.storage.tigris.dev host that the files.xeiaso.net rebrand
rewrites.

Assisted-by: Claude Opus 4.8 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Record the plan for migrating S3 client construction to
tigrisdata/storage-go and removing the within.website/x/tigris package,
alongside the existing plan documents.

Assisted-by: Claude Opus 4.8 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
@Xe

Xe commented Jun 18, 2026

Copy link
Copy Markdown
Owner Author

/reviewbot

@github-actions

Copy link
Copy Markdown
Contributor

The latest Buf updates on your PR. Results from workflow Go / build (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedJun 18, 2026, 5:43 PM

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

### Review Summary
  • autocert/s3cache/s3cache.go: Migrated S3 client construction to storage.New with WithGlobalEndpoint() and WithPathStyle(true). Swapped *s3.Client to *storage.Client.
  • cmd/future-sight/main.go: Replaced manual s3.New with storage.New, correctly passing flags. Dropped cosmetic AppID user-agent and verbose logging (documented in PR body).
  • cmd/relayd/telemetry.go: Updated client construction to storage.New. Correctly dropped the dead --telemetry-path-style flag.
  • cmd/stickers/main.go: Successfully migrated to simplestorage, removing the bespoke Presigner struct and entirely eliminating AWS SDK imports while preserving the fly endpoint for URL rebranding.
  • cmd/uploud/main.go: Clean migration to storage.New with global endpoint and path style.
  • cmd/xedn/uplodr/main.go: Unified Tigris and Backblaze B2 client construction via storage.New. Correctly adapted mkB2Client to accept context and return an error.
  • store/s3api.go: Replaced awsConfig.LoadDefaultConfig with storage.New, dropped AWS config dependency, kept necessary aws/s3 imports for input structs.
  • tigris/tigris.go: Correctly deleted the redundant helper package after migrating all call sites to storage-go.
  • internal/confyg/read.go: Fixed go vet error by applying %q to an explicit rune conversion instead of an int.
  • go.mod / go.sum: Promoted tigrisdata/storage-go to direct dependency, updated AWS SDK versions, and tidied indirect dependencies as expected.

Suggestions

The migration is thorough and aligns perfectly with the added plan document. The behavioral trade-offs (endpoint pinning, dropped logging/UA, removed dead flags) are appropriately scoped and documented in the PR body. No functional regressions are apparent, and the type swaps are safe since storage.Client embeds *s3.Client. Ready to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant