ci: validate all reproducible codegen is up to date#5553
Merged
Conversation
Replace the piecemeal generate-schema / generate-validation steps and the
separate validate-python-codegen job with a single drift gate that runs the
reproducible subset of `task generate` and asserts a clean tree. This is now
possible because CLI command-stub generation is hermetic (driven by the
checked-in .codegen/cli.json), so generate-cligen, generate-refschema and
generate-direct can be checked alongside the existing schema/validation/pydabs
generators.
Adds a `generate-check` Taskfile target listing the generators that reproduce
byte-for-byte on a clean checkout: cligen, refschema, schema, validation,
direct, pydabs. It omits the three generators in `generate` whose output is not
reproducible per-PR:
- generate-tf-schema downloads the Terraform provider (no CI runner sets it up)
- generate-schema-docs stamps x-since-version from git tags (published to the
docgen branch by update-schema-docs.yml)
- generate-docs stamps the current date via time.Now
The job gains Python + uv (needed by generate-direct and pydabs-codegen) and
uses `git add --intent-to-add` before `git diff` so newly generated files count,
not just modifications.
Co-authored-by: Isaac
pietern
changed the title
The runner's pre-installed python3 already covers the python invocations generate-check needs (EMBED_SOURCES' list_embeds.py and the stdlib-only generate-direct-apitypes); the existing job relied on it too. uv covers the remaining uv-run generators (pydabs-codegen, generate-direct-resources). EMBED_SOURCES keeps a python3 dependency repo-wide, so unifying everything on uv isn't possible here. Co-authored-by: Isaac
Rename validate-generated-is-up-to-date -> validate-generated. Switch to setup-build-environment (with a dedicated cache-key so it doesn't evict the primary test caches) for Go + Python + uv, and add `go mod download` before generate-check. generate-cligen formats the regenerated stubs with goimports, which adds imports it can only resolve from the module cache (e.g. cobra in cmd/workspace/cmd.go, whose template doesn't emit it). On a cold cache goimports silently omits the import and the regenerated tree fails to build. Co-authored-by: Isaac
janniklasrose
approved these changes
Jun 11, 2026
janniklasrose
left a comment
janniklasrose
left a comment
Contributor
There was a problem hiding this comment.
Finally validate-generated-is-up-to-date pass is complete/no longer giving false sense of security
setup-jfrog (via setup-build-environment) points uv at the JFrog pypi proxy, so pydabs-codegen's `uv run` rewrites python/uv.lock and python/codegen/uv.lock with proxy URLs (and drops some size fields). The committed locks use pypi.org, so that is environment noise rather than codegen drift; restore the files before the git diff check. Co-authored-by: Isaac
Per review (janniklasrose): the gate also checks non-code artifacts like jsonschema.json and the *.yml files, so "files" is more accurate than "code". Co-authored-by: Isaac
Collaborator
|
Commit: 2d3088c
24 interesting tests: 15 SKIP, 7 KNOWN, 2 flaky
Top 28 slowest tests (at least 2 minutes):
|
Collaborator
|
Commit: f63abcd
475 interesting tests: 394 MISS, 37 FAIL, 25 RECOVERED, 13 KNOWN, 4 PANIC, 2 SKIP
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Now that CLI command-stub generation is hermetic, fold the per-generator
generate-schema/generate-validationsteps and thevalidate-python-codegenjob into one gate that runs the reproducible subset oftask generateand asserts a clean tree.The new
generate-checktask runs the generators that reproduce byte-for-byte on a clean checkout (cligen, refschema, schema, validation, direct, pydabs). It omits the three ingeneratethat can't reproduce per-PR, documented inline:generate-tf-schema(downloads the TF provider),generate-schema-docs(git-tagx-since-version, published todocgen), andgenerate-docs(stampstime.Now).This pull request and its description were written by Isaac.