Repository navigation
Conversation
…eline checks scripts/release/verify.sh compares the release-please PR with the last release tag. It builds an inventory of the commits, runs the baseline checks (most of them from scripts/dependency-pr), runs the agent checks of each unit, and calculates one verdict. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…LI surface Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
A check that starts a server and exits without stopping it left the server running. The server kept its port and the lock file that the release runner holds for the worktree, so the next check waited forever. timeout leads its own process group, so the runner now stops that group after the check. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
A finding that explains a blocking check now replaces the block of that check, so the report lists each bug one time, with the explanation of the agent. A claim that a blocking check disproves is 'disproved', not incomplete. The report no longer cuts the text of blocks and questions. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…that it does not start Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requirements
scripts/release/test/run.sh: 21 unit tests.scripts/dependency-pr/test/run.shstill passes.)Related issues
This PR is stacked on #849, the Dependabot verifier. Merge #849 first. This PR reuses the check protocol, the hermetic environment, the secret scrubbing, and most baseline checks of #849.
Describe the solution you've provided
This PR adds a verifier for a full release. It compares the release-please PR with the last release tag. A run has a fixed part and a part that an agent decides.
The fixed part is
scripts/release/verify.sh:--phase preparemakes two worktrees (the last tag and the candidate), reads CI, and writesinventory.md. The inventory lists each commit with its PR, its changelog section, and its product areas fromscripts/release/risk-map.json. It proposes units (groups of commits), and it flags a commit and its revert, and shipped changes that the changelog leaves out.--phase baselineruns 22 checks. 15 come from feat(deps): agent-driven Dependabot PR verifier (phase 1) #849. The new ones check the version files and the bump, the changelog, CI on the release head, the CLI surface (commands, flags, and defaults, 4 levels deep), and the npm package contents.--phase unitsruns the agent checks on the last release, on the candidate, and on the candidate without given commits (revert). A check counts as proof only if it tells the versions apart.--phase renderwritesresult.jsonandreport.md. The verdict issafe-to-release,needs-human,block, orincomplete.The agent part is the
verify-releaseskill (.agents/skills/verify-release/). The agent groups the commits into units and writes claims from the PRs and the code. It writes a check for each claim. Parallel subagents can each own units (agent/units/<unit>.json). A separate judge agent then reviews every check and finding, and names gaps. The verdict staysincompletewithout a judge review, without an adversarial check in each high-risk unit, or if a commit is in no unit.Checks can use a cached binary build, a local fake LaunchDarkly API that records requests, and a read-only proxy to a live account. The proxy keeps the token in the parent process and refuses each write with 405.
Result on #826 (3.13.0)
The verdict is
block. 46 commits in 11 units, 22 baseline checks, and 71 agent checks. 68 agent checks count as proof, and 3 found confirmed bugs. The judge reviewed all of them. All 194 live requests were reads.Blocks:
sync prompts: after a 409 on the manifest save, the output isnull(JSON) orunknown error occurred. The message "run sync again" is lost.sync prompts linkwrites the variation file before it asks for confirmation, and keeps it after a "no". The next sync creates the variation that the user canceled.aws-devops-agent setup: after a failed token replacement, the rerun that setup advises restores only one of the agent spaces.dev-server startwith a published port no longer answers, because fix(dev-server): listen on 127.0.0.1 by default #845 listens on 127.0.0.1. 3.12.0 worked.Questions for a person: 15 agent files in the npm package; the new listen address as a fix in a minor release; the move to API version 20240415 (#832) without a changelog entry; the dev server stops if a flag lacks a field that v24 requires; the Alpine 3.24 image removes 45 root certificates; and the release path of release-please-action v5, which no local check can run.
Describe alternatives you've considered
sync prompts, so the agent writes checks for the claims of each release.GITHUB_TOKENstarts no workflows on the release-please PR, and the agent part needs a VM.reference/automation.mdgives a Cursor Automation prompt instead.Additional context
run_onenow stops processes that a check leaves running: they held ports and the worktree lock. Thedocker-imagemessage now says that the dev server does not answer, not that it does not start..verify-out/release-pr-<N>/, which git ignores. This PR does not post a comment on chore(main): release 3.13.0 #826.