Make package publication retries safe

[thomashoneyman]

Closes #645 by making package publication retries safe when publication is interrupted between writing:

1. the package tarball to storage,
2. package metadata to the registry repository, and
3. the manifest to the registry index.

We don't delete any artifacts after a downstream failure (since we've historically treated them as immutable). Instead, we do forward reconciliation: verify and reuse the existing tarball, repair missing metadata or manifest state, and reject any conflicting artifact or changed source.

Problem

Publication spans several independent systems and we can't do atomic commits. Previously, if the tarball upload succeeded but writing metadata or the registry-index manifest failed, retrying publication encountered an existing storage object and stopped. This left the package in a permanently incomplete state that required manual repair.

Approach

Reconcile existing storage

When an upload fails, we first check whether the requested package version actually made it into storage or not. If it does, the stored tarball is downloaded and verified against the hash and size of the newly produced tarball:

• If they match, the artifact is reused and publication continues.
• A missing, corrupt, or mismatched artifact is rejected and we error out (so we can still end up needing manual repair)
• Existing storage never gets overwritten.

This also handles uploads that succeeded (the package was written) but returned an error to the caller.

Repair incomplete downstream state

When metadata already contains the package version, then on publish we verify the stored tarball using the immutable hash and size recorded in metadata.

If the registry-index manifest is missing, the manifest embedded in the stored tarball must match the manifest produced from the current source. This prevents a moved or modified Git ref from changing an already-uploaded release.

Once verified, publication idempotently reissues both Git writes. Reissuing both writes is intentional. A failed push can leave a local checkout with a commit that has not reached its remote; an idempotent write gives the repository effect another opportunity to push it.

Preserve immutability

Retries fail instead of reconciling in these cases:

• storage contains a tarball with different bytes or hash,
• the current source manifest differs from the manifest in the stored tarball, or
• the registry index already contains a different manifest for the package version.

A package version is only treated as fully published once both metadata and its registry-index manifest are present.

Why forward reconciliation instead of rollback?

Basically, we treat the tarball as an immutable publication artifact; once it has been verified, completing the missing downstream writes is safer and more reliable than deleting and recreating it.

This makes retries convergent:

no durable state
  -> storage
  -> storage + metadata
  -> storage + metadata + manifest

Each retry either advances toward the complete state or rejects a conflicting immutable artifact, in which case something has gone horribly wrong and we have to go fix it.

Tests

I updated our Run-based testing to go through each of these conditions:

• successful publication,
• an upload that completes but reports failure,
• metadata failure leaving storage-only state,
• manifest failure leaving storage and metadata state,
• successful retry through each incomplete state,
• corrupt stored tarball rejection,
• changed source manifest rejection,
• conflicting pre-existing tarball rejection, and
• final convergence to complete storage, metadata, and manifest state.

[thomashoneyman]

I ran some timings and this was really slow, especially in the retry scenarios I added. Instead, we can just use a tmpdir and run this the same as production.

[thomashoneyman]

The failure ref is just so we can inject errors like this into the evaluation and see how it propagates out.