Fix the four outstanding CI-investigation defects, and port reviewBaselines - #66
Open
chschan wants to merge 7 commits into
Open
Fix the four outstanding CI-investigation defects, and port reviewBaselines#66chschan wants to merge 7 commits into
chschan wants to merge 7 commits into
Conversation
…lines
Closes out the seven defects catalogued in the "rhtmlCombinedScatter CI will
stop working from 30 Sep 2024" investigation. Three were already fixed on
master, independently rather than from that investigation's branch: the Windows
path corruption in compileRenderContentPage and the page.waitFor removal both
came in with the esbuild work, and the jest ceiling was lifted by the jest 29 /
jest-image-snapshot 6 / puppeteer 24 upgrade. The remaining four are here.
Defect 3 -- a mismatching snapshot reported PASS. The catch around
toMatchImageSnapshot swallowed the failure entirely, so the job only went red
via jest's aggregate snapshotState.unmatched count and the per-test list said
everything passed. Failures are now collected through the loop and rethrown
once at the end: collected rather than thrown immediately because throwing
inside the loop would abort it and lose the new_snapshots diagnostic image for
every widget after the first. The diagnostic write also becomes synchronous;
the old fs.writeFile callback form was fire-and-forget, so the process could
exit before the image reached disk. This is the most important one of the four,
because it makes the upcoming baseline review trustworthy.
Defect 5 -- acceptNewSnapshots defaulted to true, so a snapshot with no
baseline was written and PASSED. A newly added test could look green forever
while never being regression-tested. Now defaults false.
Defect 4 -- the --env whitelist allowed only 'local' and 'travis', so a CI
environment could not be named after the system running it. rhtmlCombinedScatter
had to bypass the flag entirely via snapshotTesting.env in its own config.
Whitelist dropped.
Defect 1 -- clean deleted man/, which holds TRACKED roxygen output that only
makeDocs can regenerate, and makeDocs swallows its own failure so a missing R
install is not fatal. `build` therefore silently deleted tracked R documentation
on every machine without R on PATH, including all CI runners. man/ is out of the
delete list. makeDocs also now runs `Rscript -e` rather than `r --no-save` with
a bash herestring and POSIX redirects, none of which cmd.exe can parse -- so it
could never have succeeded on Windows even with R installed.
Also ports the reviewBaselines task from that investigation's branch, which
existed nowhere else. It builds a local side-by-side review page because
GitHub's diff renderer gives up on a few hundred binary files, which is exactly
the size of a regenerated baseline set. It needed no conversion for the gulp
removal -- it never took a gulp argument -- only its usage strings and the
dropping of its `-t` alias for --to, since yargs is a shared singleton across
task modules and -t already means testNamePattern in two other tasks.
NB the investigation's branch cc-ci-fixes is NOT merged here and should not be.
It branched before 8.0.0, so merging it would revert esbuild, the puppeteer and
jest upgrade, jest.config.js and every .jest.test.js suite, and restore the old
mocha processTestPlans.test.js. These fixes were re-applied onto current master
instead. cc-ci-fixes can now be deleted.
Two things found while doing this:
* A real bug in the CLI added by the gulp removal: task names were taken as
every argument not starting with '-', so a SPACE separated flag value was
read as a task name. `rhtml reviewBaselines --from HEAD` failed with
"unknown task 'HEAD'", and `rhtml testVisual -t someFilter` would have
failed the same way -- which is the form rhtmlCombinedScatter's CI uses.
Task names are now the leading positional arguments only, stopping at the
first flag, which is what gulp did. Extracted to src/lib/parseTaskNames.js
with tests, since it shipped silently once already.
* A fidelity gap in the eslint config from the eslint 10 upgrade:
@Stylistic's customize() defaults operator-linebreak to 'before' but
eslint-config-standard used 'after'. Nothing on master happened to wrap an
operator, so the mismatch was invisible until a file written under the old
config was added, which reported 13 errors for previously-correct style.
Set to standard's value.
Verified: eslint . clean, 84/84 jest tests pass across 9 suites, bin/prepush
exits 0, and the fixture byte-diff from the gulp removal is still empty, so none
of this changes build output. The six new tests for defect 3 were confirmed to
FAIL against the old swallow-the-error behaviour (4 of 6 red) rather than merely
passing against the new one. reviewBaselines was exercised end to end against a
git fixture with a changed, an added, a removed and 18 identical baselines, plus
both of its guard paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The base config declared CommonJS globally, via eslint-plugin-n's flat/recommended-script. That is correct for this package's own source and for most of a widget repo, but WRONG for the one part that matters most to a widget: theSrc/scripts, the standardised home of the widget's own code, which is browser ES modules bundled by esbuild. Found while migrating rhtmlCombinedScatter: adopting the shared config gave a "'import' and 'export' may appear only with sourceType: module" parsing error on every one of its 36 source files. It parsed under the old .eslintrc only because eslint-config-standard set sourceType: 'module' for the whole project. So the existing browser-ESM override now also covers theSrc/scripts/**/*.js and theSrc/internal_www/js/**/*.js, which brings browser globals and the disabling of the node-oriented n/* rules with it. Globs that do not apply in a given repo match nothing, so one list serves both this package and its consumers. Verified: eslint . clean and 84/84 tests still pass here, and the config now parses rhtmlCombinedScatter's source. The ESM boundary there is exactly theSrc/scripts -- 36 files, nothing else in the repo uses import/export. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The alias map shimmed `crypto` to crypto-browserify because browserify used to provide node builtins implicitly and esbuild does not. That shim was incomplete: crypto-browserify's own dependency tree -- asn1.js, browserify-sign, browserify-rsa, safe-buffer -- requires 'buffer' in 19 places, so aliasing crypto without also aliasing buffer fails the bundle outright with "Could not resolve buffer". Found while migrating rhtmlCombinedScatter, whose dependency graph reaches crypto and which therefore could not build at all against 9.0.0: `rhtml core compileWidgetEntryPoint` exited 1 with 19 unresolved-buffer errors. Verified: eslint . clean and 84/84 tests still pass here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The alias map shimmed `crypto` to crypto-browserify because browserify used to provide node builtins implicitly and esbuild does not. That shim was incomplete. crypto-browserify's own dependency tree -- asn1.js, browserify-sign, browserify-rsa, safe-buffer, cipher-base, hash-base, readable-stream -- requires buffer, stream and events, so aliasing crypto alone fails the bundle outright, first with 19 "Could not resolve buffer" errors and then with 4 more for stream and events. Found while migrating rhtmlCombinedScatter, which could not build at all against 9.0.0: `rhtml core compileWidgetEntryPoint` exited 1. Its chain is bignumber.js@2.4.0, whose minified build requires crypto for BigNumber.random, then the tree above. Any widget depending on bignumber.js v2 hits exactly this, which is why it belongs in the shared alias map rather than in one widget's esbuildOptions escape hatch. Verified: eslint clean here, and the rhtmlCombinedScatter bundle now builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The esbuild alias map inherited `crypto: 'crypto-browserify'` from the
browserify->esbuild migration, on the reasoning that browserify shimmed node
builtins implicitly. For crypto that reasoning does not hold, because browserify
never actually shipped it for these widgets.
bignumber.js@2 -- rhtmlCombinedScatter and rhtmlLabeledScatter both depend on it --
reaches for crypto with
if ( !cryptoObj ) try { cryptoObj = require('cry' + 'pto'); } catch (e) {}
The concatenation and the try/catch are a deliberate bundler-evasion idiom: they
stop static resolution, and the catch makes runtime absence harmless. browserify's
scanner only matches literal string arguments, so it saw no dependency and emitted
no crypto. esbuild constant-folds the concatenation, so it DOES resolve, and the
alias then pulled 616 KiB across 180 files (elliptic, four separate copies of
bn.js, asn1.js, browserify-sign, diffie-hellman) into rhtmlCombinedScatter's bundle
-- 1651 KiB to 2341 KiB -- for BigNumber.random, which nothing calls.
So the default is now src/lib/cryptoStub.js, which reproduces what browserify
shipped. A widget that genuinely needs crypto opts back in from its own config:
esbuildOptions: { alias: { crypto: 'crypto-browserify' } }
NB rhtmlPictographs IS such a widget: CacheService.js and SvgDefinitionManager.js
call crypto.createHash for cache keys. It is pinned to 7.2.3 so nothing breaks
today, but it must add the opt-in when it migrates. Checked the other ten
dependent repos; none references crypto in its own source. Documented in the README
upgrade notes, naming Pictographs.
The stub's members throw rather than being absent, so that case fails with a
message naming the fix rather than "crypto.createHash is not a function".
Everything else is left undefined, which is what bignumber's own feature detection
expects.
buffer/stream/events stay aliased even though the stub no longer needs them: they
are what makes the crypto-browserify opt-in resolve at all, they are the right
answer for widget code that requires them directly, and they cost nothing when
nothing does.
Verified: eslint clean, 84/84 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jest-image-snapshot changed how it names baseline files between v3 and v6. Its
createSnapshotIdentifier now reads
let snapshotIdentifier = customSnapshotIdentifier || `${defaultIdentifier}-snap`
so '-snap' is appended ONLY when no custom identifier is supplied, and the baseline
is written as `${snapshotIdentifier}.png`. v3 appended it either way, which is why
all 877 committed baselines in rhtmlCombinedScatter -- and every other widget repo
-- are named <name>-snap.png.
testSnapshots passes a custom identifier, so under v6 it was producing <name>.png.
Nothing matched the existing baselines, every test looked new, and the first
regeneration run on rhtmlCombinedScatter wrote a parallel un-suffixed set while
orphaning the entire committed one. The suffix is now applied at the call site.
Added two tests that pin the identifier, single- and multi-widget. The stub matcher
now records what it was asked for. This failure mode is silent -- the suite goes
green, the file names are simply wrong -- so it needs a test rather than a comment.
Verified: 86/86 tests pass, eslint clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
.tmp/snapshot_dynamic_config.json exists only to carry command line values across a
process boundary: takeSnapshotsForEachTestDefinition shells out to jest, and jest
propagates no arguments to the workers that read widgetConfig. But widgetConfig merges
that file at HIGHER precedence than the widget's own build/config/widget.config.js, and
nothing ever removed it -- so it became ambient state that silently reconfigured every
LATER task.
Found in rhtmlCombinedScatter. A filtered probe run,
rhtml testVisual --env=local --branch=probe --snapshotDirectory=.tmp/probe2
left `rhtml reviewBaselines` reading .tmp/probe2/local/probe instead of the real
snapshot tree, and it failed with "Could not read .tmp/probe2/local/probe at HEAD".
That is exactly the moment reviewBaselines has to be right: straight after a test run,
when you are about to accept 430 regenerated baselines.
Now removed in the shell.exec callback, with force: true so a run that never got as far
as writing it does not fail on the way out.
NB takeExperimentSnapshots writes its own separate pass-through file, which widgetConfig
does not read, so it does not leak this way.
Verified: eslint clean, 86/86 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #65 (which is stacked on #64). Retarget down the chain as each merges.
Closes out the seven defects catalogued in rhtmlCombinedScatter CI will stop working from 30 Sep 2024.
Where the seven stood
cleandeletes trackedman/;makeDocsswallows its failurecompileRenderContentPagetestSnapshotsswallows the matcher throw → failing tests report PASS--envwhitelisted tolocal/travisacceptNewSnapshotsdefaults true → new snapshots never failpage.waitForblocks puppeteer upgrades2, 6 and 7 were fixed independently, not from the investigation's branch — 2 and 6 came in with the esbuild work, 7 with the jest 29 / jest-image-snapshot 6 / puppeteer 24 upgrade.
The four fixes
Defect 3 — the important one. The catch around
toMatchImageSnapshotswallowed the failure entirely, so a test whose images did not match reported PASS; the job only went red via jest's aggregatesnapshotState.unmatchedcount, and the per-test list said everything passed. Failures are now collected through the loop and rethrown once at the end — collected rather than thrown immediately, because throwing inside the loop would abort it and lose thenew_snapshotsdiagnostic image for every widget after the first. The diagnostic write also becomes synchronous; the oldfs.writeFilecallback form was fire-and-forget, so the process could exit before the image hit disk.This one matters most right now: it's what makes the upcoming CombinedScatter baseline review trustworthy.
Defect 5.
acceptNewSnapshotsnow defaultsfalse. Attrueit appended--ci=0, making jest computeupdateSnapshot: 'new', so a missing baseline was written and passed — a newly added test could look green forever while testing nothing.Defect 4. The
--envwhitelist allowed onlylocalandtravis, so a CI environment couldn't be named after the system running it. CombinedScatter had to bypass the flag entirely viasnapshotTesting.envin its own config. Whitelist dropped.Defect 1.
cleandeletedman/, which holds tracked roxygen output that onlymakeDocsregenerates — andmakeDocsswallows its own failure so a missing R install isn't fatal. Sobuildsilently deleted tracked R documentation on every machine without R on PATH, including all CI runners.man/is out of the list.makeDocsnow runsRscript -einstead ofr --no-savewith a bash herestring and POSIX redirects, none of which cmd.exe can parse — so it could never have succeeded on Windows even with R installed.reviewBaselinesported307 lines that existed only on the unmerged branch. Builds a local side-by-side review page, because GitHub's diff renderer gives up on a few hundred binary files — exactly the size of a regenerated baseline set.
It needed no conversion for the gulp removal (it never took a
gulpargument); only its usage strings, and dropping its-talias for--to, sinceyargsis a shared singleton across task modules and-talready meanstestNamePatternin two other tasks.Exercised end to end against a git fixture — 1 changed, 1 added, 1 removed, 18 identical, all four categories rendered — plus both guard paths (missing
--from, and--from/--toresolving to the same sha).cc-ci-fixesis not merged here, and should not beIt branched before 8.0.0. Merging it would revert esbuild, the puppeteer/jest upgrade,
jest.config.jsand every.jest.test.jssuite, and restore the old mochaprocessTestPlans.test.js— its diff against master removes 12,341 lines. These fixes were re-applied onto current master instead.cc-ci-fixescan now be deleted.Two bugs found while doing this
A real bug in the CLI I added in #65. Task names were taken as every argument not starting with
-, so a space-separated flag value was read as a task name.rhtml reviewBaselines --from HEADfailed withunknown task 'HEAD'— andrhtml testVisual -t someFilterwould have failed identically, which is the form CombinedScatter's CI uses. Task names are now the leading positional arguments only, stopping at the first flag (what gulp did). Extracted tosrc/lib/parseTaskNames.jswith 9 tests, since it shipped silently once already.A fidelity gap in the eslint config from #64.
@stylistic'scustomize()defaultsoperator-linebreakto'before', buteslint-config-standardused'after'. Nothing on master happened to wrap an operator, so the mismatch was invisible until a file written under the old config was added — which then reported 13 errors for previously-correct style. Set to standard's value.Behaviour changes for widget repos
No edit needed, but results change, so sequence a bump deliberately — documented in the README's upgrade section:
--acceptNewSnapshotsto opt back in when bootstrapping.buildno longer deletesman/.After this lands, CombinedScatter's CI can drop its
--acceptNewSnapshots=falseworkaround and set--env=cidirectly instead of viawidget.config.js.Verification
eslint .npm testnode ./bin/prepushreviewBaselinesend to end🤖 Generated with Claude Code