refactor(storage): migrate S3 clients to tigrisdata/storage-go - #949
Open
Xe wants to merge 4 commits into
Open
Conversation
go vet rejected the %#q verb applied to the int returned by peekRune. Print the offending character with %q on an explicit rune conversion so the error reads as a quoted character literal and vet is satisfied. Assisted-by: Claude Opus 4.8 via Claude Code Signed-off-by: Xe Iaso <xe@tigrisdata.com>
…orage-go Route low-level S3 client construction through github.com/tigrisdata/storage-go (storage.Client embeds *s3.Client, so call sites are unchanged) and delete the now-redundant within.website/x/tigris package, whose helpers and Region type are provided verbatim by storage-go/tigrisheaders. Tigris clients that previously resolved their endpoint from the AWS_ENDPOINT_URL_S3 env var now pin the global endpoint (t3.storage.dev); uplodr's Tigris client keeps the fly endpoint it already used. The Backblaze B2 and future-sight clients move to storage.New with explicit endpoint/region/path-style/keypair options. While migrating relayd, drop the telemetry-path-style flag: it was already dead (construction hardcoded path style) and remains so. future-sight loses its custom AppID user-agent and verbose ClientLogMode logging, which storage.New does not expose. Assisted-by: Claude Opus 4.8 via Claude Code Signed-off-by: Xe Iaso <xe@tigrisdata.com>
…presigning Replace the low-level S3 client, the bespoke Presigner type, and the HeadObject existence probe with the high-level storage-go/simplestorage client: Head for existence checks and PresignURL for the redirect URL. This drops every AWS SDK import from the command. Keep the fly endpoint so presigned URLs retain the xedn.fly.storage.tigris.dev host that the files.xeiaso.net rebrand rewrites. Assisted-by: Claude Opus 4.8 via Claude Code Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Record the plan for migrating S3 client construction to tigrisdata/storage-go and removing the within.website/x/tigris package, alongside the existing plan documents. Assisted-by: Claude Opus 4.8 via Claude Code Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Owner
Author
|
/reviewbot |
Contributor
|
The latest Buf updates on your PR. Results from workflow Go / build (pull_request).
|
Contributor
There was a problem hiding this comment.
### Review Summary
- autocert/s3cache/s3cache.go: Migrated S3 client construction to
storage.NewwithWithGlobalEndpoint()andWithPathStyle(true). Swapped*s3.Clientto*storage.Client. - cmd/future-sight/main.go: Replaced manual
s3.Newwithstorage.New, correctly passing flags. Dropped cosmeticAppIDuser-agent and verbose logging (documented in PR body). - cmd/relayd/telemetry.go: Updated client construction to
storage.New. Correctly dropped the dead--telemetry-path-styleflag. - cmd/stickers/main.go: Successfully migrated to
simplestorage, removing the bespokePresignerstruct and entirely eliminating AWS SDK imports while preserving the fly endpoint for URL rebranding. - cmd/uploud/main.go: Clean migration to
storage.Newwith global endpoint and path style. - cmd/xedn/uplodr/main.go: Unified Tigris and Backblaze B2 client construction via
storage.New. Correctly adaptedmkB2Clientto acceptcontextand return an error. - store/s3api.go: Replaced
awsConfig.LoadDefaultConfigwithstorage.New, dropped AWS config dependency, kept necessaryaws/s3imports for input structs. - tigris/tigris.go: Correctly deleted the redundant helper package after migrating all call sites to
storage-go. - internal/confyg/read.go: Fixed
go veterror by applying%qto an explicitruneconversion instead of anint. - go.mod / go.sum: Promoted
tigrisdata/storage-goto direct dependency, updated AWS SDK versions, and tidied indirect dependencies as expected.
Suggestions
The migration is thorough and aligns perfectly with the added plan document. The behavioral trade-offs (endpoint pinning, dropped logging/UA, removed dead flags) are appropriately scoped and documented in the PR body. No functional regressions are apparent, and the type swaps are safe since storage.Client embeds *s3.Client. Ready to merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates all AWS S3 SDK v2 client construction in the repo to
github.com/tigrisdata/storage-goand deletes the now-redundantwithin.website/x/tigrishelper package. The library'sstorage.Clientembeds*s3.Client, so standard S3 operations (PutObject/GetObject/HeadObject/DeleteObject/ListObjectsV2) work unchanged — most call sites only needed a type swap. Itstigrisheaderssubpackage already provides verbatim copies of every helper (and theRegiontype) that the deletedtigrispackage exposed, none of which were used anywhere beyondtigris.Client.storage.Newcredential resolution is backward-compatible: it usesTIGRIS_STORAGE_ACCESS_KEY_ID/_SECRET_ACCESS_KEYwhen set, otherwise falls back to AWSLoadDefaultConfig.Per-file changes
Low-level
storage.Client(storage.New(...)):store/s3api.go—WithGlobalEndpoint()autocert/s3cache/s3cache.go—WithGlobalEndpoint(),WithPathStyle(true)cmd/relayd/telemetry.go—WithGlobalEndpoint(),WithPathStyle(true); also drops the dead--telemetry-path-styleflag (construction always hardcoded path style)cmd/uploud/main.go—WithGlobalEndpoint(),WithPathStyle(true)cmd/future-sight/main.go—WithEndpoint/WithRegion/WithPathStyle/WithAccessKeypairfrom existing flagscmd/xedn/uplodr/main.go— Tigris client viaWithFlyEndpoint(); Backblaze B2 client viaWithEndpoint/WithRegion/WithPathStyle/WithAccessKeypairHigh-level
simplestorage:cmd/stickers/main.go— switched tostorage-go/simplestorage:Headfor the existence probe andPresignURL(http.MethodGet, key, time.Hour)for the redirect URL. Deletes the bespokePresignertype and all AWS SDK imports. KeptWithFlyEndpoint()so presigned URLs retain thexedn.fly.storage.tigris.devhost that thefiles.xeiaso.netrebrand rewrites.Removed:
tigris/tigris.go(package deleted).Deps:
go.mod/go.sum—tigrisdata/storage-gopromoted to a direct dependency;aws-sdk-go-v2/configandaws-sdk-go-v2/credentialsdemoted to indirect.Behavioral notes / decisions
AWS_ENDPOINT_URL_S3env var now pin the global endpoint (t3.storage.dev).uplodr's Tigris client keeps the fly endpoint it already used. This was a deliberate choice; it removes runtime endpoint override-ability for those services.future-sightloses its customAppIDuser-agent and verboseClientLogModerequest logging, whichstorage.Newdoes not expose (observability only — no functional impact).fix(confyg)is included as a prerequisite: a pre-existinggo veterror (%#qapplied to theintfrompeekRune) was failing the repo's pre-commitnpm testhook and blocking all commits. Fixed by printing the offending character with%qon an explicitruneconversion.autocert/s3cache(used only by sakurajima's autocert cache) now hardcodest3.storage.dev. If sakurajima previously pointed its cert cache at a different endpoint (e.g.fly.storage.tigris.dev) viaAWS_ENDPOINT_URL_S3, the existing cert cache would appear empty and certs would be re-issued (Let's Encrypt rate-limit risk). The org standard appears to bet3.storage.dev(perkube/alrest/gitea/gitea-config.yaml), so this is likely low-risk — but it can't be confirmed from code alone.Testing
go build ./...— cleango vet ./...on all migrated packages — cleango test ./...— passes (full suite runs via the pre-commit hook on every commit)storage.Newinjects no always-on Tigris middleware (B2/MinIO safe),simplestorage.Headpropagates 404 errors (not-found fallback intact), andPresignURL+WithFlyEndpointproduces the virtual-hosted host the URL rebrand depends on.🤖 Assisted-by: Claude Opus 4.8 via Claude Code