Add version and upstream license links to THIRD_PARTY_NOTICES.md - #2794
Draft
abrarshivani wants to merge 18 commits into
Draft
Add version and upstream license links to THIRD_PARTY_NOTICES.md#2794abrarshivani wants to merge 18 commits into
abrarshivani wants to merge 18 commits into
Conversation
The notices generator had no tests. Guard main so the script can be sourced, and add a minimal assertion harness for the URL derivation added next. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
The original tautology assertion would silently pass in CI environments where go-licenses is installed, allowing a regressed guard to corrupt the tracked THIRD_PARTY_NOTICES.md file. Add a deterministic structural check for BASH_SOURCE[0] in the generator, and redirect OUTPUT to a throwaway path to prevent file corruption if the guard ever regresses. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
Proxy case-encoding is done in awk: the obvious sed form silently corrupts every module path containing a capital, which the proxy then rejects. Also covers version and ref normalisation, the gopkg.in convention, GitHub submodule subdirectories, and the blob and raw URL templates for GitHub and Gerrit. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
annotate_modules keeps the version it already parsed instead of discarding it, and each package's license files are looked up in the verified URL map. A missing entry is fatal. Each file's URL is printed under its own heading, which is the only unambiguous place for it when a package carries several, and the Dependency header no longer disagrees with the Module bullet. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
location_cell was called nested inside printf's argument list, so its die/exit only killed the command substitution, not the script; under set -e a package with no verified URL rendered a blank Location cell and the generator still exited 0. Hoist the call to its own statement so the failure propagates, and add a regression test that a missing map entry aborts emit_index_table. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
Uses the module proxy Origin, then the go-import meta tag, then the github.com path shape. Upstream publishes these mappings, so nothing is hand-written: dario.cat/mergo resolves to imdario/mergo and go.yaml.in to yaml/go-yaml. The EXIT trap that removes the temp file referenced a variable local to main(); once main returned normally rather than exiting, the process's implicit exit fired that trap after the variable had gone out of scope, and set -u turned the cleanup itself into a failure that overwrote a successful run's exit status. main() now exits explicitly so the trap runs while the variable is still in scope. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
A URL is recorded only when the file it serves hashes identically to the copy under vendor/. Probing for a 200 cannot tell a correct link from one that serves the wrong licence; hashing can, and it also removes the need to guess whether a submodule inherits its licence from the repository root or ships its own. Scope is the shipped set: the verifier reuses the generator's collection, so build- and test-only dependencies are not resolved or documented. Ref candidates for go.googlesource.com repos are split into tag names and commit hashes so only tag names get the refs/tags/ qualification; a raw commit hash under refs/tags/ 404s, which otherwise breaks pseudo-versioned modules such as google.golang.org/protobuf. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
curl piped into sha256sum hashes zero bytes to a fixed, valid-looking constant on any transport failure -- a 404, a DNS error, a timeout, a rate-limit. The match at the candidate loop compared only that string against the vendored file's hash, so a failed fetch against a 0-byte licence file would have read as a genuine match and written an URL nothing ever verified. Gate the comparison on remote_sha's exit status so a failed fetch is correctly treated as no match. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
The generator fails when a license file has no verified URL, so the existing freshness gate also catches a bump that skipped the network step. A scheduled job re-checks every link against the vendored bytes to catch upstream retagging after the fact. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
- location_cell and verify-license-urls.sh's per-package loop can each find zero license files without dying, producing a blank Location cell or a silent pass; both now fail closed. - location_for treats an empty URL field as a match instead of a miss. - emit_sections swallowed a license_dir_within_module failure and could publish a link resolved against the wrong directory; it now dies like location_cell. - fence_for propagated grep's no-match exit status through pipefail, which is harmless today only because every caller is inside a command substitution. - resolve-module-repos.sh died on any unresolved module, including the build/test-only ones out of scope for the notices document; downgraded to a warning since verify-license-urls.sh already enforces fail-closed for in-scope modules. - test-tools wasn't wired into any CI-run target, so the regression tests guarding these paths never ran automatically; added it to CHECK_TARGETS. - Three regression tests sourced the generator by a repo-root-relative path and passed vacuously outside the repo root; they now take the absolute path. - Reworded the notices header sentence that contradicted the adjacent gpuop-cfg text to state the rule the tooling actually implements, and regenerated THIRD_PARTY_NOTICES.md (prose only; all 124 rows and every license text are unchanged). Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
Add shellcheck disable=SC1091 alongside the existing source= directives so shellcheck's static-only run (no -x) stays quiet about sourced sibling scripts. Discard the unused license CSV field in verify-license-urls.sh with _ instead of a named unused variable, and mark the deliberately single-quoted bash -c bodies in the notices test with disable=SC2016. No behavior changes. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
abrarshivani
requested review from
cdesiniotis,
karthikvetrivel,
rahulait,
rajathagasthya,
shivamerla and
tariq1890
as code owners
August 24, 2026 23:58
license_files_for enumerated licenses from the go-licenses save cache, which keeps only the single file go-licenses classifies as the license per package. Sibling license files in the same vendor directory - a second license, a PATENTS grant, a docs license - were silently dropped, and tools/verify-license-urls.sh enumerated the same cache, so the URL map agreed with the omission and nothing caught it. Switch all three enumeration call sites to the governing directory in vendor/ that license_dir_within_module already computes, rather than the cache. Enumerating straight from vendor/ can now match a genuine source file that starts with a license-shaped header (kube-openapi's pkg/validation/spec/license.go), so license_files_for gets an extension exclusion for common source file types. Recovers 15 previously unattributed files: PATENTS grants for every golang.org/x/* module and google.golang.org/protobuf, LICENSE.libyaml for the embedded libyaml port in go.yaml.in/yaml/v2, LICENSE.docs in opencontainers/go-digest, and PATENTS/AUTHORS files nested under third_party subpackages of several k8s.io modules. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
Replace bare-adjective and abbreviated local variable names in tools/generate-third-party-notices.sh, tools/verify-license-urls.sh, and tools/resolve-module-repos.sh with names that carry their meaning (e.g. lf -> license_file, tmp -> verified_urls_tmp_file / repos_tmp_file, rsha -> remote_sha_value). Pure rename, no behavior change. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
The three function headers described what the code below already shows. Keep only the rationale a reader cannot recover from the code: why the name filter exists, why the nearest enclosing directory wins, and what the link separator mirrors. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
abrarshivani
marked this pull request as draft
August 25, 2026 00:44
go-licenses classifies a license file as whichever license it scores highest, so gopkg.in/yaml.v3, go.yaml.in/yaml/v3 and go.yaml.in/yaml/v2 each report only one identifier even though their license file(s) bundle both Apache-2.0 and MIT grants. The license text was already reproduced in full; only the reported identifier understated it. Add tools/license-overrides.tsv, a hand-curated map of package to the correct joined identifier, and wire it into both the index table and the license sections so they agree. A stale-entry guard fails the build if an override names a package that no longer appears in the generated index. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
The index table repeats the module path next to the package path even when they match, which is true for most rows. Package and Location already identify the dependency, and the Module bullet in each package's detail section still explains the cases where the package path and module diverge, so the index no longer needs the column. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
Follows the removal of the Module column from the index table: the module a package belongs to is no longer shown anywhere in the generated notices, so drop the remaining per-section bullet and reword the header prose that described it. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
The rewording after the Module bullet was removed left a short line, and the em dashes did not match the surrounding paragraphs, which use semicolons. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
VersionandLocationcolumns toTHIRD_PARTY_NOTICES.md.Locationlinks to the licensefile in the dependency's own upstream repository, pinned to the version we redistribute:
github.com/klauspost/compress/zstd/internal/xxhashVersion was dropped twice before (
ab62fd0ac) because a bump with no license change dirtied thefile. That is reversed here: a notices file that does not say which version it describes cannot be
matched to a release. The churn is real but small, one index row and two bullets per bump.
Every URL in the file was verified by fetching it and comparing its sha256 against the copy under
vendor/. A URL that does not match is never written, so there are no dead links and none pointat the wrong license. 153 URLs across 114 modules.
That includes the secondary license files a module ships alongside its main one, which are easy to
lose:
go.yaml.in/yaml/v2carriesLICENSE.libyaml(MIT, for the embedded libyaml port) on top ofits Apache-2.0
LICENSE,github.com/opencontainers/go-digestcarriesLICENSE.docs, and thegolang.org/x/*modules andgoogle.golang.org/protobufeach carryPATENTS.What to review
Hand-written, 917 lines:
tools/verify-license-urls.shtools/license-overrides.tsvtools/resolve-module-repos.shtools/license-url-lib.shtools/generate-third-party-notices_test.shtools/generate-third-party-notices.shtools/license-url-lib_test.sh.github/workflows/third-party-notices-links.yamltools/test-helpers.shMakefile.github/workflows/third-party-notices-check.yamlGenerated, do not read:
THIRD_PARTY_NOTICES.md(1,570),tools/license-urls.tsv(158),tools/module-repos.tsv(128).tools/verify-license-urls.shis the one to read. Everything else supports it.How it works
vendor/gives the module and version for free, and the license file names, but it does notrecord the upstream repository.
dario.cat/mergoactually lives atgithub.com/imdario/mergo,k8s.io/apiatgithub.com/kubernetes/api. Scraping that out of vendored sources is wrong moreoften than right, so two committed maps carry it instead, both machine-generated:
tools/module-repos.tsvmaps module to repository. Resolution is the Go module proxy'sOrigin(101 of 124), then the
go-importmeta tag thatgo getitself uses (21), then thegithub.com/<org>/<repo>path shape (2). Nothing is hand-written. It is keyed by module and notby version, so a bump does not invalidate it.
tools/license-urls.tsvmaps module, version and license path to a verified URL. A row is writtenonly when the bytes at that URL hash identically to the vendored copy. Probing for a 200 is not
enough: it cannot tell a correct link from one that returns 200 for the wrong license.
Both are produced out of band by
make third-party-notices-reposandmake third-party-notices-urls, which need network.make third-party-noticesreads them offline, somake check-third-party-noticesstays hermetic and cannot flap on a proxy that withholdsOrigin.Scope is what we ship. The verifier reuses the generator's own collection, so it covers the
packages
go-licensesattributes to./cmd/....vendor/holds 168 license files; the 15 thatbelong to build and test only dependencies (ginkgo, gomega, go-cmp, x/tools and friends) are not
redistributed and are not listed.
License files are enumerated from
vendor/rather than from thego-licenses saveoutput, becausethat output keeps only the one file it classifies as the license per package and drops the rest.
tools/license-overrides.tsvcorrects the License column where go-licenses under-reports it. Threemodules ship one file holding two licenses:
gopkg.in/yaml.v3andgo.yaml.in/yaml/v3each carry afull-text MIT section plus a short-form Apache-2.0 grant, and
go.yaml.in/yaml/v2carriesLICENSE(Apache-2.0) alongsideLICENSE.libyaml(MIT). go-licenses classifies each as a singlelicense, so all three read
Apache-2.0 / MITfrom the override instead. It is curated by handrather than detected, because scanning license text cannot tell BSD-2-Clause from BSD-3-Clause and
a wrong addition to this file is worse than an omission. Generation fails if an override names a
package that is no longer in the index, so the file cannot rot unnoticed.
.github/workflows/third-party-notices-links.yamlre-verifies every URL weekly. Links are provencorrect when written, but upstream can retag or archive a repository afterwards and no offline gate
can see that.
Note for dependency bumps
A version change now needs two commands, because a verified URL contains the version:
Renovate and Dependabot cannot do the first on their own. Their bump job needs wiring, or a human
runs it. This is the direct cost of requiring every link to be verified rather than derived.
Testing
make test-toolsruns the new bash suites, 48 assertions across two files, and is now part ofCHECK_TARGETSso it runs in CI.Verified by hand:
HEADor branch refsvendor/modules.txtklauspost/compressrows fromtools/license-urls.tsvmakes
make third-party-noticesexit non-zero naming that module