doc: recommend node-stress-single-test for flaky tests

[trivikr]

Recommend node-stress-single-test in the collaborator guide and onboarding
docs for validating fixes to flaky tests.

Refs: https://openjs-foundation.slack.com/archives/C03BJP63CH0/p1782716150521499

[nodejs-github-bot]

Review requested:

• @nodejs/tsc

[sxa]

Suggest adding the hyperlink in, but approving regardless 👍🏻

[sxa]

Suggested change

	fixing a flaky test, it is recommended to run [`node-stress-single-test`][]
	fixing a flaky test, it is recommended to run [`node-stress-single-test`][https://ci.nodejs.org/job/node-stress-single-test]

[trivikr]

Mentioning the hyperlink in the bottom improves the source code readability, right?

It's done twice on line 289 for example.

[reliability repository][] and [guide on how to deal with CI flakes][].

[lpinca]

It is not very useful if the flaky test is in test/parallel because afaik the job runs the test sequentially.

[trivikr]

It is not very useful if the flaky test is in test/parallel because afaik the job runs the test sequentially.

The node-stress-single-test is still helpful for sequential tests.
Do you have a suggestion on how to phrase it in documentation?

[lpinca]

How about something like "When fixing a flaky test in test/sequentual, it is recommended to run node-stress-single-test. It is also good for tests in in test/parallel, but not very reliable because the job runs the test sequentially".

[sxa]

How about something like "When fixing a flaky test in test/sequentual, it is recommended to run node-stress-single-test. It is also good for tests in in test/parallel, but not very reliable because the job runs the test sequentially".

I'll be honest I've found it to be good for parallel tests as well. Even though it's theoretically possible that we'd get issues with bad interactions/conflicts when running alongside other tests in the parallel suite I don't think I've personally come across a situation where the fact that it's a parallel test has caused it not to show up in the stress job so I would dispute the "not very reliable" suggestion here and definitely would not discourage people from using the stress job for parallel tests. Curious as to whether others people's experiences are different from mine. And if a test is flaky because it's in the parallel suite, then the solution is probably to move it to sequential.

[lpinca]

I've run into a lot of cases where node-stress-single-test never failed and the flakiness was reproducible only by using -j 16 --repeat=1000.

And if a test is flaky because it's in the parallel suite, then the solution is probably to move it to sequential.

I disagree. In most cases it would only hide the underlying issue.

[sxa]

I've run into a lot of cases where node-stress-single-test never failed and the flakiness was reproducible only by using -j 16 --repeat=1000.

Thanks - good to know that I've just been lucky!

[joyeecheung]

I think there are multiple uses for the stress test:

1. For parallel testing, to put it under load
2. For sequential testing, to simply sample many many runs

It is not inherently less reliable when you run it sequentially, it just means in that case you are not putting it under load and are merely sampling many runs, which is also a valid, reliable way to use it, just be aware that it's for a different purpose.

One solution is to make -j configurable in Jenkins, I just added a workflow that allows stress testing with a configurable -j in #64118 for the same reason.

[trivikr]

FWIW I started using the stress test on main to verify flaky tests mentioned on nodejs/reliability after it was called out in #64223 (comment), and 4/5 flaky tests have failed in their stress tests:

• test: fix watch cwd argv test flake #64084
• test: limit young generation in heap snapshot tests #64107
• test: make blob desiredSize assertion robust #64106 (even though this didn't fail in main, the justification was clear and historical context was provided by a TSC member)
• debugger: preserve probe failure attribution #64102

I'm not sure how reliable the stress tests are, but I'll follow what the existing more experienced maintainers recommend.
If these tests can be run automatically, that would make it easy. The solution I would love to see is similar to what request-ci does, i.e. run CI for PRs.

Maybe flaky-test label can run CI on main as well as PR branch and post a comment?
The GitHub workflow which @joyeecheung mentioned in the above comment goes in that direction.

---

To unblock this PR, I've re-requested approval from @sxa who'd raised concerns about effectivity of stress tests for parallel tests. If they provide their approval, I'll request merging the PR.

We can update the documentation later when -j gets configurable in jenkins. WDYT?

Alternative is to remove sections of sequential and parallel from this PR.