Intentional usage of `continue-on-error`?

[saulecabrera]

I was a bit surprised to learn that when continue-on-error is used at the step level, everything appears green, which, creates a discrepancy between local runs of the test suite and what's reported on CI. I was
expecting some sort of signal, indicating that something unexpected happened in the case of non-zero exit code.

My assumption is that the usage of this feature is intentional given that tests are still in flux and that some failures are still expected, namely, there are several open issues to track expected failures for several combinations of (wasi-target, os, runtime) namely:

• http-response wasip3 test fails after toolchain, wit-bindgen, wasmtime update #186
• Newly added WASIp3 tests failing #173
• WASIp1 tests failing on Windows #175
• Tests failing on Wasmtime #101

I'm wondering if defining the test expectations in the suite itself is a viable alternative to using continue-on-error: true; hoping that it would allow a more accurate view of which tests are expected to fail vs which are expected to pass, accurate behavior between CI runs and local runs and also allowing us to catch unexpected regressions, after dependency updates like the one reported in #186

cc/ @wingo

[saulecabrera]

I have to note though, that I just started poking at this code recently, so it's totally possible that I'm missing something.

[wingo]

Hey! Yes indeed things are still in flux. The issue is that the tests aren't reliable enough at this point to use for proper CI (in the sense that they don't all pass on wasmtime on ubuntu, and fewer pass on mac, fewer still on windows, and there are no other wasip3 implementations besides jco which didn't do async at all last I checked but probably does now). Basically the CI runs are just advisory at this point while we manage to get wasmtime working on all of them on all platforms.

Is there a way to make these failing tests visible while not preventing a PR from landing?

A further question is whether we should require tests to pass on all platforms and runtimes before landing -- a test can be correct and one or more runtime/platform combinations might fail it. But we can answer that once tests are working somewhere.