Skip to content

Fix the four outstanding CI-investigation defects, and port reviewBaselines - #66

Open
chschan wants to merge 7 commits into
cc-remove-gulpfrom
cc-ci-defect-fixes
Open

Fix the four outstanding CI-investigation defects, and port reviewBaselines#66
chschan wants to merge 7 commits into
cc-remove-gulpfrom
cc-ci-defect-fixes

Conversation

@chschan

@chschan chschan commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Stacked on #65 (which is stacked on #64). Retarget down the chain as each merges.

Closes out the seven defects catalogued in rhtmlCombinedScatter CI will stop working from 30 Sep 2024.

Where the seven stood

# Defect Before this PR
1 clean deletes tracked man/; makeDocs swallows its failure ❌ open → fixed here
2 Windows path corruption in compileRenderContentPage ✅ already fixed on master
3 testSnapshots swallows the matcher throw → failing tests report PASS ❌ open → fixed here
4 --env whitelisted to local/travis ❌ open → fixed here
5 acceptNewSnapshots defaults true → new snapshots never fail ❌ open → fixed here
6 page.waitFor blocks puppeteer upgrades ✅ already fixed on master
7 The jest ceiling ✅ already fixed on master (#64's ancestors)

2, 6 and 7 were fixed independently, not from the investigation's branch — 2 and 6 came in with the esbuild work, 7 with the jest 29 / jest-image-snapshot 6 / puppeteer 24 upgrade.

The four fixes

Defect 3 — the important one. The catch around toMatchImageSnapshot swallowed the failure entirely, so a test whose images did not match reported PASS; the job only went red via jest's aggregate snapshotState.unmatched count, and the per-test list said everything passed. Failures are now collected through the loop and rethrown once at the end — collected rather than thrown immediately, because throwing inside the loop would abort it and lose the new_snapshots diagnostic image for every widget after the first. The diagnostic write also becomes synchronous; the old fs.writeFile callback form was fire-and-forget, so the process could exit before the image hit disk.

This one matters most right now: it's what makes the upcoming CombinedScatter baseline review trustworthy.

Defect 5. acceptNewSnapshots now defaults false. At true it appended --ci=0, making jest compute updateSnapshot: 'new', so a missing baseline was written and passed — a newly added test could look green forever while testing nothing.

Defect 4. The --env whitelist allowed only local and travis, so a CI environment couldn't be named after the system running it. CombinedScatter had to bypass the flag entirely via snapshotTesting.env in its own config. Whitelist dropped.

Defect 1. clean deleted man/, which holds tracked roxygen output that only makeDocs regenerates — and makeDocs swallows its own failure so a missing R install isn't fatal. So build silently deleted tracked R documentation on every machine without R on PATH, including all CI runners. man/ is out of the list. makeDocs now runs Rscript -e instead of r --no-save with a bash herestring and POSIX redirects, none of which cmd.exe can parse — so it could never have succeeded on Windows even with R installed.

reviewBaselines ported

307 lines that existed only on the unmerged branch. Builds a local side-by-side review page, because GitHub's diff renderer gives up on a few hundred binary files — exactly the size of a regenerated baseline set.

It needed no conversion for the gulp removal (it never took a gulp argument); only its usage strings, and dropping its -t alias for --to, since yargs is a shared singleton across task modules and -t already means testNamePattern in two other tasks.

Exercised end to end against a git fixture — 1 changed, 1 added, 1 removed, 18 identical, all four categories rendered — plus both guard paths (missing --from, and --from/--to resolving to the same sha).

cc-ci-fixes is not merged here, and should not be

It branched before 8.0.0. Merging it would revert esbuild, the puppeteer/jest upgrade, jest.config.js and every .jest.test.js suite, and restore the old mocha processTestPlans.test.js — its diff against master removes 12,341 lines. These fixes were re-applied onto current master instead. cc-ci-fixes can now be deleted.

Two bugs found while doing this

A real bug in the CLI I added in #65. Task names were taken as every argument not starting with -, so a space-separated flag value was read as a task name. rhtml reviewBaselines --from HEAD failed with unknown task 'HEAD' — and rhtml testVisual -t someFilter would have failed identically, which is the form CombinedScatter's CI uses. Task names are now the leading positional arguments only, stopping at the first flag (what gulp did). Extracted to src/lib/parseTaskNames.js with 9 tests, since it shipped silently once already.

A fidelity gap in the eslint config from #64. @stylistic's customize() defaults operator-linebreak to 'before', but eslint-config-standard used 'after'. Nothing on master happened to wrap an operator, so the mismatch was invisible until a file written under the old config was added — which then reported 13 errors for previously-correct style. Set to standard's value.

Behaviour changes for widget repos

No edit needed, but results change, so sequence a bump deliberately — documented in the README's upgrade section:

  • A mismatching snapshot now fails its own test. Expect previously-green runs to surface real per-test failures.
  • A snapshot with no baseline now fails. Pass --acceptNewSnapshots to opt back in when bootstrapping.
  • build no longer deletes man/.

After this lands, CombinedScatter's CI can drop its --acceptNewSnapshots=false workaround and set --env=ci directly instead of via widget.config.js.

Verification

check result
eslint . clean, exit 0
npm test 84/84 pass, 9 suites (15 new)
node ./bin/prepush exit 0
fixture byte-diff vs the gulp pipeline still empty — none of this changes build output
defect-3 tests against the OLD behaviour 4 of 6 fail — confirmed the tests actually catch the regression rather than merely passing against the new code
reviewBaselines end to end 4 statuses classified correctly, both guards fire

🤖 Generated with Claude Code

chschan and others added 7 commits August 5, 2026 21:39
…lines

Closes out the seven defects catalogued in the "rhtmlCombinedScatter CI will
stop working from 30 Sep 2024" investigation. Three were already fixed on
master, independently rather than from that investigation's branch: the Windows
path corruption in compileRenderContentPage and the page.waitFor removal both
came in with the esbuild work, and the jest ceiling was lifted by the jest 29 /
jest-image-snapshot 6 / puppeteer 24 upgrade. The remaining four are here.

Defect 3 -- a mismatching snapshot reported PASS. The catch around
toMatchImageSnapshot swallowed the failure entirely, so the job only went red
via jest's aggregate snapshotState.unmatched count and the per-test list said
everything passed. Failures are now collected through the loop and rethrown
once at the end: collected rather than thrown immediately because throwing
inside the loop would abort it and lose the new_snapshots diagnostic image for
every widget after the first. The diagnostic write also becomes synchronous;
the old fs.writeFile callback form was fire-and-forget, so the process could
exit before the image reached disk. This is the most important one of the four,
because it makes the upcoming baseline review trustworthy.

Defect 5 -- acceptNewSnapshots defaulted to true, so a snapshot with no
baseline was written and PASSED. A newly added test could look green forever
while never being regression-tested. Now defaults false.

Defect 4 -- the --env whitelist allowed only 'local' and 'travis', so a CI
environment could not be named after the system running it. rhtmlCombinedScatter
had to bypass the flag entirely via snapshotTesting.env in its own config.
Whitelist dropped.

Defect 1 -- clean deleted man/, which holds TRACKED roxygen output that only
makeDocs can regenerate, and makeDocs swallows its own failure so a missing R
install is not fatal. `build` therefore silently deleted tracked R documentation
on every machine without R on PATH, including all CI runners. man/ is out of the
delete list. makeDocs also now runs `Rscript -e` rather than `r --no-save` with
a bash herestring and POSIX redirects, none of which cmd.exe can parse -- so it
could never have succeeded on Windows even with R installed.

Also ports the reviewBaselines task from that investigation's branch, which
existed nowhere else. It builds a local side-by-side review page because
GitHub's diff renderer gives up on a few hundred binary files, which is exactly
the size of a regenerated baseline set. It needed no conversion for the gulp
removal -- it never took a gulp argument -- only its usage strings and the
dropping of its `-t` alias for --to, since yargs is a shared singleton across
task modules and -t already means testNamePattern in two other tasks.

NB the investigation's branch cc-ci-fixes is NOT merged here and should not be.
It branched before 8.0.0, so merging it would revert esbuild, the puppeteer and
jest upgrade, jest.config.js and every .jest.test.js suite, and restore the old
mocha processTestPlans.test.js. These fixes were re-applied onto current master
instead. cc-ci-fixes can now be deleted.

Two things found while doing this:

  * A real bug in the CLI added by the gulp removal: task names were taken as
    every argument not starting with '-', so a SPACE separated flag value was
    read as a task name. `rhtml reviewBaselines --from HEAD` failed with
    "unknown task 'HEAD'", and `rhtml testVisual -t someFilter` would have
    failed the same way -- which is the form rhtmlCombinedScatter's CI uses.
    Task names are now the leading positional arguments only, stopping at the
    first flag, which is what gulp did. Extracted to src/lib/parseTaskNames.js
    with tests, since it shipped silently once already.
  * A fidelity gap in the eslint config from the eslint 10 upgrade:
    @Stylistic's customize() defaults operator-linebreak to 'before' but
    eslint-config-standard used 'after'. Nothing on master happened to wrap an
    operator, so the mismatch was invisible until a file written under the old
    config was added, which reported 13 errors for previously-correct style.
    Set to standard's value.

Verified: eslint . clean, 84/84 jest tests pass across 9 suites, bin/prepush
exits 0, and the fixture byte-diff from the gulp removal is still empty, so none
of this changes build output. The six new tests for defect 3 were confirmed to
FAIL against the old swallow-the-error behaviour (4 of 6 red) rather than merely
passing against the new one. reviewBaselines was exercised end to end against a
git fixture with a changed, an added, a removed and 18 identical baselines, plus
both of its guard paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The base config declared CommonJS globally, via eslint-plugin-n's
flat/recommended-script. That is correct for this package's own source and for
most of a widget repo, but WRONG for the one part that matters most to a widget:
theSrc/scripts, the standardised home of the widget's own code, which is browser
ES modules bundled by esbuild.

Found while migrating rhtmlCombinedScatter: adopting the shared config gave a
"'import' and 'export' may appear only with sourceType: module" parsing error on
every one of its 36 source files. It parsed under the old .eslintrc only because
eslint-config-standard set sourceType: 'module' for the whole project.

So the existing browser-ESM override now also covers theSrc/scripts/**/*.js and
theSrc/internal_www/js/**/*.js, which brings browser globals and the disabling of
the node-oriented n/* rules with it. Globs that do not apply in a given repo
match nothing, so one list serves both this package and its consumers.

Verified: eslint . clean and 84/84 tests still pass here, and the config now
parses rhtmlCombinedScatter's source. The ESM boundary there is exactly
theSrc/scripts -- 36 files, nothing else in the repo uses import/export.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The alias map shimmed `crypto` to crypto-browserify because browserify used to
provide node builtins implicitly and esbuild does not. That shim was incomplete:
crypto-browserify's own dependency tree -- asn1.js, browserify-sign,
browserify-rsa, safe-buffer -- requires 'buffer' in 19 places, so aliasing crypto
without also aliasing buffer fails the bundle outright with "Could not resolve
buffer".

Found while migrating rhtmlCombinedScatter, whose dependency graph reaches crypto
and which therefore could not build at all against 9.0.0:
`rhtml core compileWidgetEntryPoint` exited 1 with 19 unresolved-buffer errors.

Verified: eslint . clean and 84/84 tests still pass here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The alias map shimmed `crypto` to crypto-browserify because browserify used to
provide node builtins implicitly and esbuild does not. That shim was incomplete.
crypto-browserify's own dependency tree -- asn1.js, browserify-sign,
browserify-rsa, safe-buffer, cipher-base, hash-base, readable-stream -- requires
buffer, stream and events, so aliasing crypto alone fails the bundle outright,
first with 19 "Could not resolve buffer" errors and then with 4 more for stream
and events.

Found while migrating rhtmlCombinedScatter, which could not build at all against
9.0.0: `rhtml core compileWidgetEntryPoint` exited 1. Its chain is
bignumber.js@2.4.0, whose minified build requires crypto for BigNumber.random,
then the tree above. Any widget depending on bignumber.js v2 hits exactly this,
which is why it belongs in the shared alias map rather than in one widget's
esbuildOptions escape hatch.

Verified: eslint clean here, and the rhtmlCombinedScatter bundle now builds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The esbuild alias map inherited `crypto: 'crypto-browserify'` from the
browserify->esbuild migration, on the reasoning that browserify shimmed node
builtins implicitly. For crypto that reasoning does not hold, because browserify
never actually shipped it for these widgets.

bignumber.js@2 -- rhtmlCombinedScatter and rhtmlLabeledScatter both depend on it --
reaches for crypto with

    if ( !cryptoObj ) try { cryptoObj = require('cry' + 'pto'); } catch (e) {}

The concatenation and the try/catch are a deliberate bundler-evasion idiom: they
stop static resolution, and the catch makes runtime absence harmless. browserify's
scanner only matches literal string arguments, so it saw no dependency and emitted
no crypto. esbuild constant-folds the concatenation, so it DOES resolve, and the
alias then pulled 616 KiB across 180 files (elliptic, four separate copies of
bn.js, asn1.js, browserify-sign, diffie-hellman) into rhtmlCombinedScatter's bundle
-- 1651 KiB to 2341 KiB -- for BigNumber.random, which nothing calls.

So the default is now src/lib/cryptoStub.js, which reproduces what browserify
shipped. A widget that genuinely needs crypto opts back in from its own config:

    esbuildOptions: { alias: { crypto: 'crypto-browserify' } }

NB rhtmlPictographs IS such a widget: CacheService.js and SvgDefinitionManager.js
call crypto.createHash for cache keys. It is pinned to 7.2.3 so nothing breaks
today, but it must add the opt-in when it migrates. Checked the other ten
dependent repos; none references crypto in its own source. Documented in the README
upgrade notes, naming Pictographs.

The stub's members throw rather than being absent, so that case fails with a
message naming the fix rather than "crypto.createHash is not a function".
Everything else is left undefined, which is what bignumber's own feature detection
expects.

buffer/stream/events stay aliased even though the stub no longer needs them: they
are what makes the crypto-browserify opt-in resolve at all, they are the right
answer for widget code that requires them directly, and they cost nothing when
nothing does.

Verified: eslint clean, 84/84 tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jest-image-snapshot changed how it names baseline files between v3 and v6. Its
createSnapshotIdentifier now reads

    let snapshotIdentifier = customSnapshotIdentifier || `${defaultIdentifier}-snap`

so '-snap' is appended ONLY when no custom identifier is supplied, and the baseline
is written as `${snapshotIdentifier}.png`. v3 appended it either way, which is why
all 877 committed baselines in rhtmlCombinedScatter -- and every other widget repo
-- are named <name>-snap.png.

testSnapshots passes a custom identifier, so under v6 it was producing <name>.png.
Nothing matched the existing baselines, every test looked new, and the first
regeneration run on rhtmlCombinedScatter wrote a parallel un-suffixed set while
orphaning the entire committed one. The suffix is now applied at the call site.

Added two tests that pin the identifier, single- and multi-widget. The stub matcher
now records what it was asked for. This failure mode is silent -- the suite goes
green, the file names are simply wrong -- so it needs a test rather than a comment.

Verified: 86/86 tests pass, eslint clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
.tmp/snapshot_dynamic_config.json exists only to carry command line values across a
process boundary: takeSnapshotsForEachTestDefinition shells out to jest, and jest
propagates no arguments to the workers that read widgetConfig. But widgetConfig merges
that file at HIGHER precedence than the widget's own build/config/widget.config.js, and
nothing ever removed it -- so it became ambient state that silently reconfigured every
LATER task.

Found in rhtmlCombinedScatter. A filtered probe run,

    rhtml testVisual --env=local --branch=probe --snapshotDirectory=.tmp/probe2

left `rhtml reviewBaselines` reading .tmp/probe2/local/probe instead of the real
snapshot tree, and it failed with "Could not read .tmp/probe2/local/probe at HEAD".
That is exactly the moment reviewBaselines has to be right: straight after a test run,
when you are about to accept 430 regenerated baselines.

Now removed in the shell.exec callback, with force: true so a run that never got as far
as writing it does not fail on the way out.

NB takeExperimentSnapshots writes its own separate pass-through file, which widgetConfig
does not read, so it does not leak this way.

Verified: eslint clean, 86/86 tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant