[FLINK-40248][ci] Add nightly snapshot publishing to GHA pipeline by dalelane · Pull Request #28831 · apache/flink

dalelane · 2026-07-28T10:56:15Z

What is the purpose of the change

The nightly Azure pipeline publishes two things that the GitHub Actions nightly
workflow currently doesn't: release binaries uploaded to S3, and snapshot jars
deployed to repository.apache.org. That gap needs closing before Flink can move off
Azure Pipelines (FLINK-27075).

Which pipeline publishes a given branch will be configuration held in the GHA nightly
workflow itself.

Because that workflow is versioned per branch, each branch can be switched over
independently, and a branch that hasn't been switched over yet still builds its
snapshots on GHA (keeping them as build artifacts of the workflow run rather than
publishing them).

The aim of this is to make sure that the new GHA jobs get exercised on every branch
before they become the canonical source of anything we publish.

master is deliberately listed in both of the lists described below, which is the
first phase of the rollout proposed on FLINK-40248: GHA publishes master's binaries
under a gha-trial/ prefix while Azure continues to publish to the bucket root, so the
two can be compared over a series of nightly runs before anything depends on GHA's
output.

Removing master from the shadow list and removing Azure's cron_snapshot_deployment
job will be a follow-up, once the outputs have been confirmed to match.

Brief change log

Adds snapshot_binary and snapshot_maven jobs to .github/workflows/nightly.yml,
mirroring the cron_snapshot_deployment_binary and cron_snapshot_deployment_maven
jobs in tools/azure-pipelines/build-nightly-dist.yml. Both run on JDK 11, the
minimum supported version, matching the Azure jobs.
PUBLISH_SNAPSHOT_BRANCHES lists branches whose snapshots this pipeline publishes.
Because nightly.yml is versioned per branch, this is per-branch configuration,
and a branch that has the change backported without being added to the list defaults
closed: it builds, but publishes nothing.
SHADOW_SNAPSHOT_BRANCHES lists branches that GHA and Azure are both publishing
during comparison. Binaries for those branches are uploaded to a gha-trial/ prefix
in the S3 bucket rather than the bucket root, so the two pipelines can't overwrite
each other. Snapshot jars uploaded to Maven need no equivalent renaming, as they
already have timestamped names.
The binary release is uploaded as a GHA build artifact on every branch, whether or
not it publishes, so its contents can be compared against what Azure produces.
When a branch isn't publishing, the Maven job deploys into a local repository via
-DaltDeploymentRepository and uploads that as a tarball artifact, instead of
deploying to repository.apache.org.
tools/ci/deploy_nightly_to_s3.sh: upload_to_s3 takes an optional destination
prefix, defaulting to the existing behaviour of writing to the bucket root.
tools/releasing/create_binary_release.sh: adds SKIP_PYTHON_WHEELS, defaulting to
false so the manual release process is unaffected. Nightly builds set it to
true, so a binary release can be produced without the 12 platform wheels that a
release manager downloads by hand beforehand. Also enables nullglob around the
wheel loop, so it doesn't iterate over a literal *.whl when no wheels are present.
tools/azure-pipelines/build-nightly-dist.yml: sets SKIP_PYTHON_WHEELS=true, and
now checks the exit code of create_binary_release.sh. Previously the nightly Azure
build depended on the missing-wheels check failing, with that failure masked by the
echo/find commands following it in the same inline script — which meant any
other error in the script was silently ignored too.

Verifying this change

This change is a CI configuration change without test coverage. I'll verify by
running affected pipelines on my fork.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components),
Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude Sonnet 5

flinkbot · 2026-07-28T11:03:11Z

CI report:

5e497a5 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

dalelane · 2026-07-28T11:50:36Z

+          name: snapshot-maven-jars-${{ github.run_number }}
+          path: ${{ env.STAGING_JARS_TARBALL }}
+          if-no-files-found: error
+          retention-days: 5


re: dalelane#1 (comment)

I'm wondering whether it's feasible to have all the generated jars in a tar.gz build artifact or whether we hit some size limits here.

@XComp setting a retention time is a nod towards your comment on the storage requirements here. We could make this even shorter, I largely picked 5 days out of the air.

FWIW gzip'ping doesn't seem to make much difference.

I tried building a sample of modules - which suggests that the total size will be in the 1gb - 1.5gb ballpark. That is bigger than the snapshot binaries (which seem to be in the 600mb ballpark)

If this is too big, we could just tar the primary jars, and drop the sources/javadoc versions? Is that worth doing?

can we hide that behind a flag as well which, when set to 0 doesn't run the artifact upload at all?

maybe, we should double-check with INFRA whether uploading to s3 on a prefix or build artifacts are a preferred approach. Or whether there's a 3rd option. 🤔

XComp

Thanks for coming up with this PR, @dalelane . Do we already have a test CI run with build artifacts?

XComp · 2026-08-03T07:03:31Z

+  # that it starts publishing to the bucket root as the Azure pipeline stops.
+  # Leaving a branch here after AZP has stopped means nothing is publishing its
+  # snapshots to the location that consumers read from.
+  SHADOW_SNAPSHOT_BRANCHES: '["master"]'


I'm wondering whether it would be sufficient to compare binary artifacts via the build artifacts which means we would not need to push them under a different name at all. 🤔

XComp · 2026-08-03T07:15:30Z

+          export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
+          echo "Determined RELEASE_VERSION as '$RELEASE_VERSION'"
+          cd tools
+          MVN_RUN_VERBOSE=true SKIP_GPG=true SKIP_PYTHON_WHEELS=true ./releasing/create_binary_release.sh


Why don't we need an explicit echo here in case the script fails? And would it be worth adding the echo to the script instead of changing the azure pipeline?

github actions fails fast by default, so I didn't think this was needed - but in hindsight, relying on a default that could change is probably not ideal, and it is clearer to have it in the script - so I did that in 336e964

XComp · 2026-08-03T07:34:34Z

+      - name: "Initialize job"
+        uses: "./.github/actions/job_init"
+        with:
+          jdk_version: 11


shouldn't we rely on the default version (which is 17 now) instead of hard-coding 11 here?

I feel like I'm too often pointing at "because-thats-how-the-Azure-pipeline-does-it" as an answer, but that is the reason here. I was copying what build-apache-repo.yml does

added a comment to explain in 5e497a5

XComp · 2026-08-03T08:14:38Z

+          name: snapshot-maven-jars-${{ github.run_number }}
+          path: ${{ env.STAGING_JARS_TARBALL }}
+          if-no-files-found: error
+          retention-days: 5


can we hide that behind a flag as well which, when set to 0 doesn't run the artifact upload at all?

XComp · 2026-08-03T08:16:09Z

+          name: snapshot-maven-jars-${{ github.run_number }}
+          path: ${{ env.STAGING_JARS_TARBALL }}
+          if-no-files-found: error
+          retention-days: 5


maybe, we should double-check with INFRA whether uploading to s3 on a prefix or build artifacts are a preferred approach. Or whether there's a 3rd option. 🤔

XComp

@snuyanzin can you also have a look if time permits.

Adding two jobs to the nightly GHA workflow that matches the cron_snapshot_deploment_binary and cron_snapshot_deployment_maven jobs that are in tools/azure-pipelines/build-nightly-dist.yml Both of these are using JDK 11 as the minimum supported version described in build-apache-repo.yml Signed-off-by: Dale Lane <dlane@confluent.io>

The create_binary_release.sh script is used in two ways: 1) manually, as part of creating a release 2) automated, as part of a nightly pipeline When run manually, there is a documented expectation (in the wiki) that a release manager will put the wheels in place ahead of running the script. When run as part of the pipeline, the wheels won't be there, and this causes the script to exit with an error. The wheels aren't needed for nightly builds, so this error is ignored. This commit changes that behaviour to be more intentional. It explicitly stops the automated build from requiring wheels, and starts checking exit codes so any other errors in future will not be ignored. Signed-off-by: Dale Lane <dlane@confluent.io>

If the deploy_nightly_to_s3.sh script is used by both overnight pipelines, then they need to write to separate locations to avoid them clobbering each other. I've left the AZP destination as-is, and added a prefix to the new GHA pipeline. If/when the AZP pipeline is retired, that prefix can be removed from the GHA call. Signed-off-by: Dale Lane <dlane@confluent.io>

Signed-off-by: Dale Lane <dale.lane@uk.ibm.com>

Reworded the PUBLISH_SNAPSHOT_BRANCHES comment to clarify the purpose of the guard Emptied the PUBLISH_SNAPSHOT_BRANCHES and SHADOW_SNAPSHOT_BRANCHES lists until there is agreement that this should go live Remvoed the redundant RELEASE_DIR environment variable and inlined the path instead Added an ERR trap to create_binary_release.sh Added a secrets-missing warn (and skip) guard to the S3 upload step Removed my indented comment style Extracted the Maven snapshot deploy logic into it's own script Signed-off-by: Dale Lane <dlane@confluent.io>

Signed-off-by: Dale Lane <dlane@confluent.io>

dalelane mentioned this pull request Jul 28, 2026

feat: add nightly snapshot releases to GHA pipeline dalelane/flink#1

Closed

dalelane commented Jul 28, 2026

View reviewed changes

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Jul 28, 2026

XComp reviewed Aug 3, 2026

View reviewed changes

dalelane and others added 5 commits August 4, 2026 10:48

feat: support for migrating one branch at a time

70fe811

Signed-off-by: Dale Lane <dale.lane@uk.ibm.com>

dalelane force-pushed the FLINK-40248 branch from 336e964 to ce1719a Compare August 4, 2026 09:51

docs: copy comment from AZP pipeline to explain JDK 11

5e497a5

Signed-off-by: Dale Lane <dlane@confluent.io>

Uh oh!

Conversation

dalelane commented Jul 28, 2026

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

XComp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

XComp left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

flinkbot commented Jul 28, 2026 •

edited

Loading