Skip to content

fix: harden release reruns and pin checks - #507

Merged
uniqueg merged 4 commits into
devfrom
fix/506-release-hardening
Nov 19, 2026
Merged

uniqueg merged 4 commits into
devfrom
fix/506-release-hardening

Conversation

@uniqueg

@uniqueg uniqueg commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

Closes #506 (part of elixir-cloud-aai/.github#92). This ports elixir-cloud-aai/trs-filer#311 (fix for trs-filer#310) as is, plus trs-filer's earlier handling of empty chart dependencies:

  1. Re-runs of a published release. publish has a new step, "Check release state", right after "Check chart" and "Set up Docker Buildx" and before "Log in to Docker Hub", "Attest build provenance", "Sign image" and "Tag image". It runs scripts/release_assets.py precheck <tag>, which works like state but also requires a published release to have the Sigstore bundle of each asset already. For a published release, it also requires docker buildx imagetools inspect "${IMAGE_NAME}:${VERSION}" to point to the run's DIGEST. Otherwise the step fails and says to release a new patch version. As a result, a re-run of all jobs of a published release no longer moves X.Y.Z/latest to a new digest. Re-running only the failed jobs still passes, and drafts and new releases are unchanged.
  2. Pin check against uv sources and the lock. scripts/check_release_pins.py now also rejects the following. https://pypi.org/simple is the only allowed index.
    • [tool.uv.sources] entries of these kinds: git, url, path/editable, workspace, a non-PyPI or undefined index, and unknown forms
    • non-PyPI indexes: [[tool.uv.index]], index-url, extra-index-url, find-links
    • uv.lock packages whose source is not PyPI (the app itself is exempt)
    • a missing or unreadable uv.lock
  3. Empty chart dependencies. dependencies: [] or {} in Chart.yaml now counts as no dependencies, as in trs-filer. Before, it was reported as an unreadable flow-style dependency. New test: test_chart_dependencies_empty.

With this, scripts/release_assets.py and scripts/check_release_pins.py are identical to trs-filer's again, apart from one trs-filer-specific comment.

The workflow test is also slightly stricter than trs-filer's. Its gh stand-in answers only the step's own releases request (repos/$GH_REPO/releases?per_page=100), and the test checks the step's GH_REPO. Sourcery found this gap on elixir-cloud-aai/drs-filer#135, which ports the same change.

Docs: the header comment of release.yml, the step's own comment, and "Versioning and releases" in docs/development.md (the two release-state checks, re-runs, and the new pin-check findings).

Effect on dev: none. dev pins released FOCA and foca-plugins, every uv.lock package except protes itself comes from PyPI, and python scripts/check_release_pins.py exits 0. tests/test_check_release_pins.py::test_release_checkout still skips on non-release checkouts.

Verification

  • coverage run -m pytest -W ignore::DeprecationWarning, then coverage report: total coverage 100%, and all tests outside the Helm tests passed (1969 passed, 11 skipped). The 211 failures are all in tests/test_chart.py (209) and tests/test_config_examples.py::test_chart_examples_valid (2). Both need the library chart built with helm dependency build, which isn't available on this machine. CI runs them.
  • pytest tests/test_check_release_pins.py tests/test_release_assets.py tests/test_workflows.py: 228 passed, 1 skipped (test_release_checkout, not a release checkout).
  • test_chart_dependencies_empty failed before the fix (3 cases) and passes after it.
  • pre-commit run --all-files passed: ruff check/format, semgrep and its rule tests, gitleaks, actionlint with shellcheck, ast-grep and its rule tests, mypy, and pre-commit-hooks.
  • tests/test_workflows.py::test_release_state_check_runs runs the step's script against stand-ins for gh and docker. Cases: no release, draft, published with the run's digest (passes), published with another digest or without an image tag (fails), published with another chart (fails), no bundle, and mutable release.
  • No fork dry run now. The release workflow changed, so a fork dry run comes before proTES's next release, as decided on fix: harden release reruns and pin checks #506: publish a test release, re-run all jobs, with DOCKERHUB_ORG set on the fork. No release is planned now.
  • CI passed on f7909c5 (run): all checks are green, and Integration (TESK), Push image and Publish image were skipped, as intended on PRs. On 6cc3703, before the Sourcery fixes, the same set passed: Lint, Docs, both Test jobs, Chart, Smoke test, Integration (label, service info), Scan image, Release readiness, CI passed, CodeQL, Codecov and Read the Docs.

Review: one round of adversarial and quality review (full lane, since this changes the release workflow). Neither found anything high or medium.

Sourcery opened five threads, all now answered and resolved:

  • Fixed in bee9c40: the new-digest case of the workflow test now requires the digest-mismatch error. Before, any failure passed it, including a failed inspection caused by wrong docker arguments. When I changed the step's --format argument as a test, the case failed.
  • Fixed in f7909c5: the docstrings of check_release_pins.py now say that an empty [] or {} counts as no dependencies.
  • Answered: [tool.uv] default-index. uv 0.12.13 rejects that key in pyproject.toml as an unknown field. The default index there is an [[tool.uv.index]] entry, which is already checked, and an index set on the command line shows up in uv.lock.
  • Answered: a manual publish racing the run. The race existed before this PR, and this PR narrows its window, as answered in trs-filer#311.
  • Declined: [ ] and { } with inner spaces. The check fails closed with a readable finding, this chart uses block style, and the script's parsing should change in all four repos at once.

Both fixes are also in drs-filer#135 and cloud-registry#419.

Declined review findings

  • Adversarial (low): on a published release that isn't immutable, a digest mismatch still gives the remedy "release a new patch version", although deleting the release and re-running would also work. Declined, as in trs-filer#311: the digest check is reached only if the re-packaged chart is byte-identical. Otherwise precheck fails first and gives the remedy that matches the immutable flag. In practice it can't be reached.
  • Adversarial (low): "Check release state" inspects the image tag anonymously, before "Log in to Docker Hub", so a Docker Hub rate limit can fail it on a re-run of failed jobs. Declined, to stay identical to trs-filer. The error says to re-run the job, and nothing has been signed or tagged at that point. For fork dry runs, the fork's Docker Hub repository must be public, like the real one.
  • Quality (low): docs/development.md explains the precheck asset rule only in the script and the step comment. Declined: same as trs-filer, and the user-facing section already describes the effect.

Open questions

None.

🤖 Generated with Claude Code

uniqueg and others added 2 commits November 19, 2026 00:23
Port elixir-cloud-aai/trs-filer#311:

- publish: new step "Check release state" right after "Check chart",
  before Docker login, attestation, signing and tagging. It runs
  `release_assets.py precheck` (a published release must already have
  this run's chart and its Sigstore bundle) and, for a published
  release, requires the `X.Y.Z` image tag to point to the run's digest,
  so a re-run of all jobs fails before it moves any tag.
- check_release_pins.py: also reject `[tool.uv.sources]` overrides,
  package indexes other than https://pypi.org/simple, `uv.lock`
  packages from other sources (the app itself excepted) and a missing
  `uv.lock`.
- check_release_pins.py: treat empty chart `dependencies: []` or `{}`
  as no dependencies (as in trs-filer), so the script is identical in
  all four repos again.
- Tests and docs (release.yml header, docs/development.md).

Closes #506

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The `gh` stand-in of `test_release_state_check_runs` returned the
releases for any arguments, so a "Check release state" step that queried
the wrong repository or endpoint still passed. It now answers only the
step's request (`repos/$GH_REPO/releases?per_page=100`), and
`test_release_state_checked_first` checks that the step takes `GH_REPO`
from the run. Found by Sourcery on
elixir-cloud-aai/drs-filer#135.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@uniqueg uniqueg self-assigned this Nov 19, 2026
@sourcery-ai

sourcery-ai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

Implementation

Extend release pin validation to reject non-PyPI uv sources, indexes and lockfile packages, and treat empty chart dependency lists or maps as having no dependencies.

  • scripts/check_release_pins.py L16-25, L103-110, L253-381 (+5 small edits)

Add a precheck mode that verifies published release assets before signing, while retaining the existing post-signing release-state check.

  • scripts/release_assets.py L24-29, L118-125, L170-173 (+7 small edits)

Configuration

Add an early release-state and image-digest check so published-release reruns fail before Docker login, attestation, signing or tagging.

  • .github/workflows/release.yml L14-20, L394-431

Documentation

Document the early release checks and rerun behavior, and explain the expanded release pin requirements.

  • docs/development.md L584-587, L624-637, L771-777 (+1 small edits)

Tests

Add coverage for empty chart dependencies, uv source and lockfile validation, and lockfile error cases.

  • tests/test_check_release_pins.py L51-70, L248-262, L643-974 (+3 small edits)

Test precheck behavior for published assets, missing bundles, unexpected assets and unpublished releases.

  • tests/test_release_assets.py L317-406

Test that the workflow runs release checks before publishing actions and rejects unsafe reruns or digest mismatches.

  • tests/test_workflows.py L581-730 (+3 small edits)

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@read-the-docs-community

read-the-docs-community Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 proTES | 🛠️ Build #36000271 | 📁 Comparing f7909c5 against latest (8314ccc)

  🔍 Preview build  

1 file changed
± development/index.html

@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (8314ccc) to head (f7909c5).

Additional details and impacted files
@@            Coverage Diff            @@
##               dev      #507   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           37        37           
  Lines         3475      3544   +69     
=========================================
+ Hits          3475      3544   +69     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@uniqueg
uniqueg marked this pull request as ready for review November 19, 2026 00:35
@sourcery-ai

sourcery-ai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Linked issues

Checked at commit f7909c5

#506 fix: harden release reruns and pin checks · 4 of 5 met · 1 to check by hand

# Requirement Status Notes
1 Published-release reruns fail before attesting, signing, or tagging; failed-job reruns with the same digest still pass. ✅ Met The workflow test checks the step order and exercises published reruns with matching and mismatching digests, as well as failed-job reruns.
2 Reject non-PyPI uv sources and lockfile packages, while allowing PyPI-only locks and exempting the app itself. ✅ Met The pin checks cover source kinds, index settings, lockfile sources, and the project exemption; tests include a PyPI-only lock.
3 Document the new release behavior in the release workflow header and release documentation. ✅ Met Both the release.yml header and docs/development.md describe the checks and rerun behavior.
4 Treat empty chart dependencies written as [] or {} as no dependencies. ✅ Met The parser handles empty flow-style lists and maps, with tests for both.
5 Do a fork dry run before this repository's next release. 👀 Check by hand (manual process) The PR says the dry run has not happened yet; verify that it is completed before the next release.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 5 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path=".github/workflows/release.yml" line_range="405" />
<code_context>
+      # on a re-run of failed jobs, which reuses both. Drafts and new
+      # releases pass; "Create release" checks the release again (keep the
+      # fetch of the releases the same in both steps)
+      - name: Check release state
+        env:
+          GH_TOKEN: ${{ github.token }}
</code_context>
<issue_to_address>
**Published release points to wrong image**

When a release is published after the precheck snapshot but before `Tag image` runs, the workflow uses the stale precheck result, so `Tag image` moves `X.Y.Z` (and possibly `latest`) to this run’s digest; the later `Create release` check happens too late to prevent the published release from pointing at a different image.

Re-fetch and validate the release state and image digest before moving tags, and abort tagging if the release has been published or its digest differs.

Also at `.github/workflows/release.yml:414-415`, `.github/workflows/release.yml:418`.
</issue_to_address>

### Comment 2
<location path="scripts/check_release_pins.py" line_range="110" />
<code_context>
+UV_SOURCE_KINDS = ("git", "url", "path", "workspace")
+# Settings of `[tool.uv]` that name package indexes (or, `find-links`,
+# other places to find packages), each a URL or a list of URLs
+UV_INDEX_SETTINGS = ("index-url", "extra-index-url", "find-links")


</code_context>
<issue_to_address>
**Non-PyPI default index passes the release check**

When `[tool.uv].default-index` is configured with a non-PyPI URL while locked packages remain sourced from PyPI, `UV_INDEX_SETTINGS` omits `default-index`, so `_uv_pins` reports no finding and `_lock_pins` passes; the release check accepts the non-PyPI configuration.

Include `default-index` in `UV_INDEX_SETTINGS` so `_uv_pins` checks it.

Also at `scripts/check_release_pins.py:111`.
</issue_to_address>

### Comment 3
<location path="scripts/check_release_pins.py" line_range="139" />
<code_context>
             item_indent = None
-            if in_dependencies and value.strip():
+            # An empty list (or map) has no dependencies to check
+            if in_dependencies and value.strip() not in ("", "[]", "{}"):
                 dependencies.append(None)
             continue
</code_context>
<issue_to_address>
**Valid empty dependencies block releases**

When `Chart.yaml` uses an empty flow-style dependency collection with internal whitespace, such as `[ ]` or `{ }`, `_chart_dependencies` recognizes only the exact strings `[]` and `{}`, so it appends `None`; `find_dev_pins` reports that dependencies cannot be checked and blocks release readiness.

Normalize whitespace inside the flow-style value before checking whether the collection is empty.
</issue_to_address>

### Comment 4
<location path="tests/test_workflows.py" line_range="695-697" />
<code_context>
+        ),
+        "docker": (
+            '[[ "$*" == "buildx imagetools inspect $IMAGE_NAME:1.2.3 '
+            '--format {{ json .Manifest }}" ]] && '
+            + (f'cat "{tmp_path}/manifest.json"' if tagged else "exit 1")
+        ),
+    }
+    for name, body in stubs.items():
</code_context>
<issue_to_address>
**Wrong inspection command passes test**

When the `new-digest` case runs and the workflow invokes `docker` with unexpected arguments, `test_release_state_check_runs` treats this as the expected inspection failure; its nonzero-status assertion and generic “release a new patch version” check pass even when the command arguments are wrong, so the test does not verify the digest-mismatch branch.

Make the test distinguish an unexpected invocation from an unavailable tag, and assert the mismatch-specific output for the `new-digest` case.
</issue_to_address>

### Comment 5
<location path="scripts/check_release_pins.py" line_range="139" />
<code_context>
             item_indent = None
-            if in_dependencies and value.strip():
+            # An empty list (or map) has no dependencies to check
+            if in_dependencies and value.strip() not in ("", "[]", "{}"):
                 dependencies.append(None)
             continue
</code_context>
<issue_to_address>
**Empty chart dependencies misdescribed**

The module docstring still says flow-style `[...]` and `{...}` dependencies cannot be read, but `_chart_dependencies` now accepts empty `[]` and `{}`. Users relying on that description get an incorrect account of which charts the release check rejects.

Clarify that only non-empty flow-style dependencies are reported as unreadable.
</issue_to_address>

Sourcery assessment

Approval pending. 4 findings to address first.

Blocking findings: .github/workflows/release.yml:405, scripts/check_release_pins.py:110, scripts/check_release_pins.py:139, tests/test_workflows.py:697


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread .github/workflows/release.yml Resolved
Comment thread scripts/check_release_pins.py Resolved
Comment thread scripts/check_release_pins.py Resolved
Comment thread tests/test_workflows.py Resolved
Comment thread scripts/check_release_pins.py Resolved
uniqueg and others added 2 commits November 19, 2026 01:52
In `test_release_state_check_runs`, the `new-digest` case passed for any
failure that says "release a new patch version", which the error of a
failed image inspection says too. A wrong `docker` invocation in "Check
release state" therefore went unnoticed. The case now requires the
digest-mismatch error (`points to <digest>`). Found by Sourcery on
#507.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The docstrings of `check_release_pins.py` still said that chart
dependencies in flow style can't be read, but an empty list or map
(`[]`, `{}`) now counts as no dependencies. Found by Sourcery on
#507.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uniqueg added a commit to elixir-cloud-aai/drs-filer that referenced this pull request Nov 19, 2026
In `test_release_state_check_runs`, the `new-digest` case passed for any
failure that says "release a new patch version", which the error of a
failed image inspection says too. A wrong `docker` invocation in "Check
release state" therefore went unnoticed. The case now requires the
digest-mismatch error (`points to <digest>`). Found by Sourcery on
elixir-cloud-aai/protes#507.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uniqueg added a commit to elixir-cloud-aai/drs-filer that referenced this pull request Nov 19, 2026
The docstrings of `check_release_pins.py` still said that chart
dependencies in flow style can't be read, but an empty list or map
(`[]`, `{}`) now counts as no dependencies. Found by Sourcery on
elixir-cloud-aai/protes#507.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uniqueg added a commit to elixir-cloud-aai/drs-filer that referenced this pull request Nov 19, 2026
* fix: harden release reruns and pin checks

Port elixir-cloud-aai/trs-filer#311:

- publish: new step "Check release state" right after "Check chart",
  before Docker login, attestation, signing and tagging. It runs
  `release_assets.py precheck` (a published release must already have
  this run's chart and its Sigstore bundle) and, for a published
  release, requires the `X.Y.Z` image tag to point to the run's digest,
  so a re-run of all jobs fails before it moves any tag.
- check_release_pins.py: also reject `[tool.uv.sources]` overrides,
  package indexes other than https://pypi.org/simple, `uv.lock`
  packages from other sources (the app itself excepted) and a missing
  `uv.lock`.
- Tests and docs (release.yml header, docs/development.md).

Closes #131

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: check the release lookup of the gh stub

The `gh` stand-in of `test_release_state_check_runs` returned the
releases for any arguments, so a "Check release state" step that queried
the wrong repository or endpoint still passed. It now answers only the
step's request (`repos/$GH_REPO/releases?per_page=100`), and
`test_release_state_checked_first` checks that the step takes `GH_REPO`
from the run. Found by Sourcery on
#135.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: check the digest error of the release step

In `test_release_state_check_runs`, the `new-digest` case passed for any
failure that says "release a new patch version", which the error of a
failed image inspection says too. A wrong `docker` invocation in "Check
release state" therefore went unnoticed. The case now requires the
digest-mismatch error (`points to <digest>`). Found by Sourcery on
elixir-cloud-aai/protes#507.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: describe empty flow-style chart deps

The docstrings of `check_release_pins.py` still said that chart
dependencies in flow style can't be read, but an empty list or map
(`[]`, `{}`) now counts as no dependencies. Found by Sourcery on
elixir-cloud-aai/protes#507.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@uniqueg
uniqueg merged commit e0ace4c into dev Nov 19, 2026
53 checks passed
@uniqueg
uniqueg deleted the fix/506-release-hardening branch November 19, 2026 02:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: harden release reruns and pin checks

1 participant