feat(ci): forward-port backwards compatibility e2e workflows to v5#22930
Merged
benesjan merged 2 commits intoMay 5, 2026
Conversation
## Summary
Wires `CONTRACT_ARTIFACTS_VERSION`-based backwards-compat testing into
the release flow (F-550), and restructures the release pipeline so a
compat regression actually blocks stable/RC publishing.
On v4 releases, runs all simple e2e tests (excluding prover,
block_building, and epochs) against contract artifacts from every prior
stable tag >= v4.2.0, validating that new client code still works with
old contract artifacts ("new pxe / old contracts").
## How the release pipeline is wired now
Three tiers of jobs in `.github/workflows/ci3.yml`:
1. **`ci`** — verification build. On a release tag, runs `./bootstrap.sh
ci-release` which now does `semver check && build` only (no publish).
2. **`ci-compat-e2e`** — runs the compat test matrix on a dedicated EC2
instance via `./bootstrap.sh ci-compat-e2e`. Discovers prior stable tags
from `.release-please-manifest.json`, pre-populates
`.legacy-contracts/<version>/` on the host, and runs e2e tests with
`CONTRACT_ARTIFACTS_VERSION=<ver>` set. Triggered on any `refs/tags/v*`
except `-commit.`, plus on PRs with the `ci-compat-e2e` or
`ci-release-pr` labels.
3. **`ci-release-publish`** — the actual publish (`./bootstrap.sh
ci-release-publish` → `build && release`: npm publish, Docker push,
GitHub release, aztec-up scripts, etc.). `needs: [ci, ci-compat-e2e]`
with an `if:` that publishes only when ci succeeded and either the tag
is nightly or ci-compat-e2e succeeded.
### Tag class behavior
| Tag class | ci-compat-e2e | ci-release-publish |
|----------------------------------------|---------------------|------------------------------------------|
| Stable / RC (e.g. `v4.2.1`, `-rc.1`) | Runs, blocking | Gated on
compat-e2e passing |
| Nightly (e.g. `-nightly.20260417`) | Runs, observational | Publishes
regardless (`continue-on-error: true` keeps nightly green) |
| Dev `-commit.*` (ci-release-pr flow) | Skipped on tag push; runs on PR
if label applied | Skipped (`-commit.` filter) |
## Escape hatches
- `ci-skip-compat-e2e` PR label — marks compat-e2e failures non-blocking
on release PRs.
- `ci-compat-e2e` PR label — forces the job to run on a non-release PR.
# Does this thing actually work?
When I trigger it on this PR with `ci-release-pr` the thing seems to run
correctly - `ci_compat_e2e` workflow runs and is green. This tests that
the workflow itself is well setup and that the tags get resolved
correctly (i.e. it finds the correct tag and runs the tests).
What remains is to see if this gets run on nightly release in
observation mode (I got an agreement to not block on nightlies if there
is a failure since there might be flakes) and then see on the next
stable release in strict mode (if you get red `ci_compat_e2e` job then
no release should happen).
# Future work
- Forward port this to v5 (while not needed there as the code will be
dormant don't want to diverge too much)
- Would be nice to get a notification on slack if we get failure on
nightly
# How confident am I this thing is not non-sense
I have no experience with the build and release scripts but I reviewed
the diff and it makes sense to me.
…les (#22900) For example, this failure is a false positive: https://github.com/AztecProtocol/aztec-packages/actions/runs/25204430819/job/73902953105 This happened because the contract used to e2e test nested utility calls didn't exist back in 4.2.0. It's not a failure: it's a test that should trivially pass. We do however log such cases, in case they help us trace subtler issues in the future.
This was referenced May 4, 2026
2 tasks
benesjan
added a commit
that referenced
this pull request
May 12, 2026
## Summary - Drop OIDC auth from `ci-compat-e2e` and run with `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` instead. - The OIDC role (`pipeline-exec-aztecprotocol-aztec-packages-heads-next`) has no `ec2:RunInstances` policy attached, so spot requests time out and the on-demand fallback fails with `UnauthorizedOperation`. The job has been failing on every v4 nightly since #22930; `continue-on-error` for `-nightly.` tags has masked it. - Mirrors c3c1371 (#23167), which applied the same workaround to `ci-release-publish` on this branch. Example of the failure being patched: https://github.com/AztecProtocol/aztec-packages/actions/runs/25737242295/job/75580441745 ## Test plan - [ ] CI3 on this PR runs to green (regular `ci` job is unaffected — it already uses static keys). - [ ] Apply the `ci-compat-e2e` label here to exercise the compat-e2e job end-to-end and confirm the EC2 spot/on-demand request succeeds with the static credentials. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 tasks
ludamad
pushed a commit
that referenced
this pull request
Jun 1, 2026
PR #22930 split ci-release into a build-only ci-release plus a separate ci-release-publish, with publishing gated on a standalone ci-compat-e2e GitHub job. A side effect: the release dry-run leg of merge-queue-ci (ci-release v0.0.1-commit.<sha> under DRY_RUN=1) degraded to build-only, so the publish flow had zero dry-run coverage — the first time publish code ran was a live tag push. Reunify: - bootstrap.sh: ci-release once again does semver check -> backwards-compat e2e -> private-fork handling -> build release -> release (publish), all DRY_RUN-aware. Removed ci-release-publish and the standalone ci-compat-e2e case; the compat suite is now a release_compat_e2e bash check called inline (blocking for stable/RC, non-blocking + Slack-notify for nightlies, honors SKIP_COMPAT_E2E, amd64-only). - ci.sh: dropped release-publish and compat-e2e; release is a single multi_job_run of ci-release on amd64 + arm64, with a roomier AWS_SHUTDOWN_TIME since the amd64 leg now does compat + build + publish. - ci3.yml: removed the ci-release-publish and ci-compat-e2e jobs (publishing happens in the main ci job on a tag, as before the split); dropped the ci-release-publish dependency from ci-network-scenario.
ludamad
pushed a commit
that referenced
this pull request
Jun 1, 2026
PR #22930 split ci-release into a build-only ci-release plus a separate ci-release-publish, with publishing gated on a standalone ci-compat-e2e GitHub job. A side effect: the release dry-run leg of merge-queue-ci (ci-release v0.0.1-commit.<sha> under DRY_RUN=1) degraded to build-only, so the publish flow had zero dry-run coverage — the first time publish code ran was a live tag push. Reunify: - bootstrap.sh: ci-release once again does semver check -> backwards-compat e2e -> private-fork handling -> build release -> release (publish), all DRY_RUN-aware. Removed ci-release-publish and the standalone ci-compat-e2e case; the compat suite is now a release_compat_e2e bash check called inline (blocking for stable/RC, non-blocking + Slack-notify for nightlies, honors SKIP_COMPAT_E2E, amd64-only). - ci.sh: dropped release-publish and compat-e2e; release is a single multi_job_run of ci-release on amd64 + arm64, with a roomier AWS_SHUTDOWN_TIME since the amd64 leg now does compat + build + publish. - ci3.yml: removed the ci-release-publish and ci-compat-e2e jobs (publishing happens in the main ci job on a tag, as before the split); dropped the ci-release-publish dependency from ci-network-scenario.
danielntmd
pushed a commit
to danielntmd/aztec-packages
that referenced
this pull request
Jun 4, 2026
…tecProtocol#23780) ## What 1. Reunifies the release flow into a single `ci-release` command (undoes the `ci-release` / `ci-release-publish` split from AztecProtocol#22930). This had introduced several bugs and caused the command to span multiple machines. 2. Fixes runner-side `git push` auth, broken by the checkout-v6 repin.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes https://linear.app/aztec-labs/issue/F-605/forward-port-backwards-compatibility-workflows-to-v5next
Summary
Forward-ports the v4 backwards-compatibility CI workflows onto
merge-train/fairies(F-605). Picks up two PRs that landed onbackport-to-v4-next-staging:feat(ci): add backwards compatibility e2e tests to CI pipelinefix: e2e compat should not fail for contracts added after legacy stablesThe code is dormant on v5 today (no v5 stable tags >= cutoff yet), but keeping the wiring in place avoids divergence between v4 and v5 release pipelines as called out in the original PR's "Future work".