Add S3 integration tests

[PeterDowdy]

This PR adds S3 integration tests through minIO as a mocked S3 provider. It covers most of the happy and sad paths in the project, with moderate rigour. It doesn't check transient errors (since the library just dies on them anyways), TLS verification (since that seems like it's drifting out of testing S3), or very large synthetic files (to keep runtime short).

The minIO test depends on an env var so developers without minIO can safely bypass it.

This effort surfaced a few fairly small gaps in S3 file-handling that could be filled:

• No retries (as mentioned)
• A single failure in multipart aborts the whole multipart operation (related)
• SigV2 write unimplemented (seems pretty legacy though)
• CRAM UR: tag can't reach S3
• CRAM reference caching only ever uses local disk (not so much a gap as a possible useful feature)
• bgzip CLI can't write to S3
• Loading a .crai always downloads it to disk

Assisted-by: Claude:claude-sonnet-5

[whitwham]

Thank you for this. I won't have time to look at it for the next few weeks but I will get round to it.

[PeterDowdy]

Thank you for this. I won't have time to look at it for the next few weeks but I will get round to it.

No rush!

[whitwham]

Looking at this I have a some comments:

Firstly (and most importantly) is looks like minio is not longer in development and has been archived. The tests get the latest docker image, if the project is no longer maintained it should probably get a specific image (just in case someone replaces the latest image with something nefarious).

Secondly, the copyright mentions the Broad Institute. From your profile you do not appear to work for them so that needs changing.

Thirdly (and this one is more subjective) your test file itself seems large and wordy. It is going to make it harder to check and maintain.

All that being said we are thinking about using it for our maintenance checks (rather than the regular ones).