ci: publish npm packages over OIDC instead of a static token

[jasonhernandez]

Motivation

NPM_TOKEN is a long-lived npm write token injected into the environment of every build on the high-risk Buildkite agent queues. Since npm capped granular access tokens at 90 days, it also has to be rotated on a timer, which is the cost we actually want to eliminate.

npm's trusted publishing replaces tokens with short-lived OIDC credentials, but it accepts them from GitHub Actions, GitLab, and CircleCI only — not from Buildkite.

Companion PR in MaterializeInc/i2 provisions the staging bucket and the IAM role.

Description

Rather than move the release off Buildkite, this keeps Buildkite as the pipeline and moves only the npm publish call:

1. deploy-npm builds the wasm packages as before, then npm packs them and writes a manifest.json describing what to publish and with which dist-tags.
2. It uploads that staging directory to S3 (agent instance role) and fires a repository_dispatch.
3. .github/workflows/publish-npm.yml assumes an AWS role via GitHub OIDC, downloads the tarballs, and publishes each one with an npm OIDC credential.
4. The Buildkite step polls the run, annotates the build with a link to it, and exits nonzero if the workflow failed, so a broken publish still turns the pipeline red where people are looking.

No npm token anywhere in the path — including for the latest-X.Y dist-tags, which is the part that took some doing. The npm CLI only performs the OIDC exchange from npm publish and npm stage publish; npm dist-tag never does, so it would ordinarily need a token. The workflow therefore performs the exchange by hand — POST /-/npm/v1/oidc/token/exchange/package/<pkg> with the Actions ID token, exactly as the CLI does internally — and passes the resulting short-lived, package-scoped registry token to npm dist-tag add.

This is the one thing that needs an empirical check. npm documents the exchanged token as covering publish, and the trusted-publisher UI's "allowed actions" list offers only npm publish and npm stage publish, so the registry may reject a dist-tag write with it. The CLI does use the same token for a non-publish endpoint (libaccess.getVisibility), which is why this is worth trying. One workflow_dispatch run against a dev version settles it. If the registry says no, the fallback that still gets us to zero tokens is to retire the latest-X.Y tags in favor of ~X.Y.0 ranges in consumers — ~26.37.0 resolves to the newest 26.37.x, which is what the tag means.

Other non-obvious decisions:

• Provenance is explicitly disabled. Trusted publishing generates attestations automatically, but they would claim the Actions run built tarballs that Buildkite actually built. A knowingly false signed attestation is worse than the none we publish today.
• No registry-url on setup-node. It writes an _authToken line into .npmrc, and an empty token there convinces npm it is already authenticated so it never attempts OIDC, failing with ENEEDAUTH (actions/setup-node#1551). A guard step fails the job if anything else configures one.
• repository_dispatch, not a tag trigger, because the dev channel publishes a prerelease on every main build. Dispatched workflows always run the definition from the default branch, so the payload can't run workflow code from another ref.
• The publish step skips versions already on npm, which makes the workflow re-runnable from the same staging URI. That is the remedy if the dist-tag step fails; staged tarballs are kept for 14 days.

Worth being explicit about the security boundary: this retires the long-lived token, but the published bytes still come off a Buildkite agent, and S3-write plus dispatch from that environment are jointly equivalent to publish rights. The hardening win of building where you publish is not part of this change.

Required setup before merging

• Configure a trusted publisher for @materializeinc/sql-lexer, sql-parser, and sql-pretty: this repository, workflow filename publish-npm.yml, environment npm-publish. npm does not validate any of this at save time — mistakes surface as ENEEDAUTH at publish.
• Create the npm-publish environment and set the NPM_PUBLISH_ROLE_ARN repository variable to the GithubActionsNpmPublish role ARN from the i2 PR.
• Deploy the i2 PR first, so NPM_STAGING_BUCKET reaches the agents.

Once a release has gone out through this path: set each package's publishing access to "require 2FA and disallow tokens", revoke the automation token, and drop npm_token from i2. At that point there is no npm credential left to rotate.

Verification

Not yet verified end to end — the handoff can't run until the trusted publishers and the AWS role exist, which is why this is a draft. Locally: bin/pyactivate -m ci.deploy.npm --no-release still exercises the build path unchanged, and the publish path is guarded by version_exists_in_npm, so a retried step re-publishes nothing.

The first real test should be a workflow_dispatch run against a dev build, which exercises the publish path but not the dist-tag path (dev versions carry no latest-X.Y). A release build is what settles the dist-tag question above.