tools: add more options to track flaky tests #43954

aduh95 · 2022-07-23T11:43:46Z

For tests that are listed as flaky, try running them 100 times until we get a passing one. If it keeps failing after 100 attempts, it's not flaky, it's broken.
For test failures from tests that are not marked as flaky, try to run them 10 times to get an idea if the test is flaky or broken.

Those changes only apply to GHA, Jenkins config is not on this repo AFAIK, and local runs are not affected as it doesn't change the defaults.

My hope is with such changes, we would be lenient on marking tests as flaky because the CI would check that are indeed flaky (currently, it would simply ignore its result).

Refs: #43929 (comment)
/cc @tniessen

Refs: nodejs#43929 (comment)

nodejs-github-bot · 2022-07-23T11:43:49Z

Review requested:

@nodejs/actions

tniessen · 2022-07-23T12:01:22Z

Oh wow! You were quick to act on my suggestion!

tniessen · 2022-07-23T12:02:13Z

cc @nodejs/build

mcollina

lgtm

tniessen

The approach here looks like what I suggested, but I'd prefer if someone who's more familiar with make run-ci and tools/test.py takes a look, too.

lpinca · 2022-07-23T13:13:35Z

The problem I see here is that it does not run tests on platforms where tests are flaky, for example arm or smartOS.

aduh95 · 2022-07-23T13:15:49Z

The problem I see here is that it does not run tests on platforms where tests are flaky, for example arm or smartOS.

If this PR lands, the config of the Jenkins runners can be updated to take advantage of these changes, and can be applied to arm and/or SmartOS. Or do you mean something else?

lpinca · 2022-07-23T13:22:01Z

If a test is flaky only a particular platform and we don't measure its flakiness on that particular platform how can we determine if it is flaky or broken?

tniessen · 2022-07-23T13:30:56Z

I might be missing something; my suggestion was to measure flakiness on any platform where tests are marked as flaky.

lpinca · 2022-07-23T13:36:54Z

That makes sense but from my understanding this uses only GHA platforms which is a small subset of the supported platforms.

tniessen · 2022-07-23T13:37:57Z

My understanding is that this is limited to GHA only because Jenkins config lives outside of the repo, and once this lands, Jenkins could be adapted accordingly. Is that correct @aduh95?

lpinca · 2022-07-23T13:40:06Z

Ah ok, I guess that's what @aduh95 wrote in #43954 (comment).

tniessen

This needs to be adapted:

node/tools/test.py

Lines 1436 to 1438 in d29e78a

    
           if options.flaky_tests not in [RUN, SKIP, DONTCARE]: 
        
             print("Unknown flaky-tests mode %s" % options.flaky_tests) 
        
             return False

tools/test.py

nodejs-github-bot · 2022-07-25T12:01:43Z

Landed in 2e4bcfc

Refs: #43929 (comment) PR-URL: #43954 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Feng Yu <F3n67u@outlook.com>

##### [\`v18.7.0\`](https://github.com/nodejs/node/releases/tag/v18.7.0) ##### Notable changes - **doc**: - add F3n67u to collaborators (Feng Yu) [#43953](nodejs/node#43953) - deprecate coercion to integer in process.exit (Daeyeon Jeong) [#43738](nodejs/node#43738) - **(SEMVER-MINOR)** deprecate diagnostics_channel object subscribe method (Stephen Belanger) [#42714](nodejs/node#42714) - **events**: - **(SEMVER-MINOR)** expose CustomEvent on global with CLI flag (Daeyeon Jeong) [#43885](nodejs/node#43885) - **(SEMVER-MINOR)** add `CustomEvent` (Daeyeon Jeong) [#43514](nodejs/node#43514) - **http**: - **(SEMVER-MINOR)** add drop request event for http server (theanarkh) [#43806](nodejs/node#43806) - **lib**: - **(SEMVER-MINOR)** improved diagnostics_channel subscribe/unsubscribe (Stephen Belanger) [#42714](nodejs/node#42714) - **util**: - **(SEMVER-MINOR)** add tokens to parseArgs (John Gee) [#43459](nodejs/node#43459) ##### Commits - \[[`0aa255ab72`](nodejs/node@0aa255ab72)] - **bootstrap**: handle snapshot errors gracefully (Joyee Cheung) [#43531](nodejs/node#43531) - \[[`0783ddf57e`](nodejs/node@0783ddf57e)] - **buffer**: do not leak memory if buffer is too big (Keyhan Vakil) [#43938](nodejs/node#43938) - \[[`12657accdd`](nodejs/node@12657accdd)] - **build**: add .gitattributes for npm and other shims (Hrishikesh Kadam) [#43879](nodejs/node#43879) - \[[`c2db4f4581`](nodejs/node@c2db4f4581)] - **build**: make GitPod less noisy (Rich Trott) [#43829](nodejs/node#43829) - \[[`364deeadcd`](nodejs/node@364deeadcd)] - **build**: add GitHub token permissions for workflows (Varun Sharma) [#43743](nodejs/node#43743) - \[[`8b83b4d5be`](nodejs/node@8b83b4d5be)] - **child_process**: do not need to count length when maxBuffer is Infinity (theanarkh) [#43822](nodejs/node#43822) - \[[`c1893b7a7c`](nodejs/node@c1893b7a7c)] - **child_process**: avoid repeated calls to `normalizeSpawnArguments` (木杉) [#43345](nodejs/node#43345) - \[[`7b276b89b9`](nodejs/node@7b276b89b9)] - **cluster**: send connection to other server when worker drop it (theanarkh) [#43747](nodejs/node#43747) - \[[`e8c66f92a5`](nodejs/node@e8c66f92a5)] - **crypto**: remove unneeded guard (Rich Trott) [#43856](nodejs/node#43856) - \[[`c95132e9ea`](nodejs/node@c95132e9ea)] - **deps**: cherry-pick [libuv/libuv@`3a7b955`](libuv/libuv@3a7b955) (Ben Noordhuis) [#43950](nodejs/node#43950) - \[[`cc8d5426d2`](nodejs/node@cc8d5426d2)] - **deps**: cherry-pick [libuv/libuv@`abb109f`](libuv/libuv@abb109f) (Ben Noordhuis) [#43950](nodejs/node#43950) - \[[`7762e463d6`](nodejs/node@7762e463d6)] - **deps**: update corepack to 0.12.1 (Node.js GitHub Bot) [#43965](nodejs/node#43965) - \[[`1256c4dad5`](nodejs/node@1256c4dad5)] - **deps**: update hast-util-raw (Moshe Atlow) [#43927](nodejs/node#43927) - \[[`aac97c2d2a`](nodejs/node@aac97c2d2a)] - **deps**: update undici to 5.8.0 (Node.js GitHub Bot) [#43886](nodejs/node#43886) - \[[`cdff61917d`](nodejs/node@cdff61917d)] - **deps**: clean archs files for OpenSSL (RafaelGSS) [#43735](nodejs/node#43735) - \[[`fc936a84e4`](nodejs/node@fc936a84e4)] - **deps**: remove not used architectures (RafaelGSS) [#43735](nodejs/node#43735) - \[[`361a643d8b`](nodejs/node@361a643d8b)] - **deps**: V8: backport [`f3cad8c`](nodejs/node@f3cad8cec656) (Joyee Cheung) [#43531](nodejs/node#43531) - \[[`2e1732ebd0`](nodejs/node@2e1732ebd0)] - **deps**: V8: backport [`22698d2`](nodejs/node@22698d267667) (Chengzhong Wu) [#43751](nodejs/node#43751) - \[[`979f469d3a`](nodejs/node@979f469d3a)] - **deps**: upgrade npm to 8.15.0 (npm team) [#43917](nodejs/node#43917) - \[[`4096d81988`](nodejs/node@4096d81988)] - **deps**: upgrade npm to 8.14.0 (npm team) [#43826](nodejs/node#43826) - \[[`2ec8092e2c`](nodejs/node@2ec8092e2c)] - **deps,src**: use SIMD for normal base64 encoding (Brian White) [#39775](nodejs/node#39775) - \[[`67b4edde37`](nodejs/node@67b4edde37)] - **dns**: fix getServers return undefined (jiahao.si) [#43922](nodejs/node#43922) - \[[`7c75539a88`](nodejs/node@7c75539a88)] - **dns**: fix cares memory leak (theanarkh) [#43912](nodejs/node#43912) - \[[`1f80b88da5`](nodejs/node@1f80b88da5)] - **doc**: update email and mailmap for BethGriggs (Beth Griggs) [#43985](nodejs/node#43985) - \[[`8a2a6e16eb`](nodejs/node@8a2a6e16eb)] - **doc**: add 15.x - 18.x to Other Versions section (shhh7612) [#43940](nodejs/node#43940) - \[[`51cb0d42ca`](nodejs/node@51cb0d42ca)] - **doc**: inspector.close undefined in worker threads (Keyhan Vakil) [#43867](nodejs/node#43867) - \[[`c789c0f5f7`](nodejs/node@c789c0f5f7)] - **doc**: improve documentation for safe `Promise` statics alternatives (Antoine du Hamel) [#43759](nodejs/node#43759) - \[[`cb9b0e0011`](nodejs/node@cb9b0e0011)] - **doc**: recommend git-node-v8 (Keyhan Vakil) [#43934](nodejs/node#43934) - \[[`d7e9bd1830`](nodejs/node@d7e9bd1830)] - **doc**: clarify subprocess.stdout/in/err property (Kohei Ueno) [#43910](nodejs/node#43910) - \[[`808793ebb5`](nodejs/node@808793ebb5)] - **doc**: fix typo in `src/crypto/README.md` (Jianru Lin) [#43968](nodejs/node#43968) - \[[`bbc455c4f9`](nodejs/node@bbc455c4f9)] - **doc**: remind backporter about v8\_embedder_string (Keyhan Vakil) [#43924](nodejs/node#43924) - \[[`a86b66c8b4`](nodejs/node@a86b66c8b4)] - **doc**: fix typo in http.md (Airing) [#43933](nodejs/node#43933) - \[[`a96af37233`](nodejs/node@a96af37233)] - **doc**: add F3n67u to collaborators (Feng Yu) [#43953](nodejs/node#43953) - \[[`aa7d4e59f7`](nodejs/node@aa7d4e59f7)] - **doc**: improve test runner timeout docs (Tobias Nießen) [#43836](nodejs/node#43836) - \[[`80c2fa8212`](nodejs/node@80c2fa8212)] - **doc**: mention Win 32-bit openssl build issue (RafaelGSS) [#43853](nodejs/node#43853) - \[[`8b8c55df7e`](nodejs/node@8b8c55df7e)] - **doc**: add security release specifics to releases.md (Beth Griggs) [#43835](nodejs/node#43835) - \[[`42693aaf9f`](nodejs/node@42693aaf9f)] - **doc**: add history info for `global.performance` (Antoine du Hamel) [#43841](nodejs/node#43841) - \[[`140d6af572`](nodejs/node@140d6af572)] - **doc**: add platform-windows-arm to who to CC (Michael Dawson) [#43808](nodejs/node#43808) - \[[`976093efe3`](nodejs/node@976093efe3)] - **doc**: document ES2022's Error "cause" property (James Ide) [#43830](nodejs/node#43830) - \[[`ec7e45e4a2`](nodejs/node@ec7e45e4a2)] - **doc**: include make clean to openssl arch (RafaelGSS) [#43735](nodejs/node#43735) - \[[`d64dfd53c9`](nodejs/node@d64dfd53c9)] - **doc**: add link to diagnostic tools (Rafael Gonzaga) [#43736](nodejs/node#43736) - \[[`2910136920`](nodejs/node@2910136920)] - **doc**: update links to MDN page about dynamic imports (Jannis R) [#43847](nodejs/node#43847) - \[[`d88a9fae79`](nodejs/node@d88a9fae79)] - **doc**: deprecate coercion to integer in process.exit (Daeyeon Jeong) [#43738](nodejs/node#43738) - \[[`fc843e103d`](nodejs/node@fc843e103d)] - **doc**: add MoLow to triagers (Moshe Atlow) [#43799](nodejs/node#43799) - \[[`8c8c97da61`](nodejs/node@8c8c97da61)] - **(SEMVER-MINOR)** **doc**: deprecate diagnostics_channel object subscribe method (Stephen Belanger) [#42714](nodejs/node#42714) - \[[`9b53a694b5`](nodejs/node@9b53a694b5)] - **doc**: revert anachronistic 'node:' module require()s in API history notes (DeeDeeG) [#43768](nodejs/node#43768) - \[[`2815bd3002`](nodejs/node@2815bd3002)] - **doc**: clarify release process for new releasers (Rafael Gonzaga) [#43739](nodejs/node#43739) - \[[`50b3750e67`](nodejs/node@50b3750e67)] - **doc**: fix typo in ngtcp2 readme (Dan Castillo) [#43767](nodejs/node#43767) - \[[`6bcd40dd85`](nodejs/node@6bcd40dd85)] - **domain**: fix vm promise tracking while keeping isolation (Stephen Belanger) [#43556](nodejs/node#43556) - \[[`e89e0b470b`](nodejs/node@e89e0b470b)] - **esm**: remove superfluous argument (Rich Trott) [#43884](nodejs/node#43884) - \[[`0d2921f396`](nodejs/node@0d2921f396)] - **esm**: fix erroneous re-initialization of ESMLoader (Jacob Smith) [#43763](nodejs/node#43763) - \[[`9b5b8d78c3`](nodejs/node@9b5b8d78c3)] - **esm**: throw on any non-2xx response (LiviaMedeiros) [#43742](nodejs/node#43742) - \[[`dfc4832ef1`](nodejs/node@dfc4832ef1)] - **(SEMVER-MINOR)** **events**: expose CustomEvent on global with CLI flag (Daeyeon Jeong) [#43885](nodejs/node#43885) - \[[`e4473952ae`](nodejs/node@e4473952ae)] - **(SEMVER-MINOR)** **events**: add `CustomEvent` (Daeyeon Jeong) [#43514](nodejs/node#43514) - \[[`100f6deb09`](nodejs/node@100f6deb09)] - **fs**: use signed types for stat data (LiviaMedeiros) [#43714](nodejs/node#43714) - \[[`25ec71db63`](nodejs/node@25ec71db63)] - **http**: fix http server connection list when close (theanarkh) [#43949](nodejs/node#43949) - \[[`ca658c8afe`](nodejs/node@ca658c8afe)] - **(SEMVER-MINOR)** **http**: add drop request event for http server (theanarkh) [#43806](nodejs/node#43806) - \[[`9c699bd8a8`](nodejs/node@9c699bd8a8)] - **http**: wait for pending responses in closeIdleConnections (Paolo Insogna) [#43890](nodejs/node#43890) - \[[`781d5e54e3`](nodejs/node@781d5e54e3)] - **inspector**: set sampling interval before start (Shelley Vohr) [#43779](nodejs/node#43779) - \[[`0b5dbb2a56`](nodejs/node@0b5dbb2a56)] - **lib**: refactor PriorityQueue to use private field (Finn Yu) [#43889](nodejs/node#43889) - \[[`324473ca32`](nodejs/node@324473ca32)] - **(SEMVER-MINOR)** **lib**: improved diagnostics_channel subscribe/unsubscribe (Stephen Belanger) [#42714](nodejs/node#42714) - \[[`5aa3b213ac`](nodejs/node@5aa3b213ac)] - **meta**: update AUTHORS (Node.js GitHub Bot) [#43966](nodejs/node#43966) - \[[`e707552357`](nodejs/node@e707552357)] - **meta**: update `node-api` in label-pr-config (Daeyeon Jeong) [#43794](nodejs/node#43794) - \[[`8a8de94034`](nodejs/node@8a8de94034)] - **meta**: update AUTHORS (Node.js GitHub Bot) [#43872](nodejs/node#43872) - \[[`7d49fc766c`](nodejs/node@7d49fc766c)] - **meta**: use platform dropdown on flaky template (Rafael Gonzaga) [#43855](nodejs/node#43855) - \[[`e4aa50fc3f`](nodejs/node@e4aa50fc3f)] - **meta**: enable blank issues (Matteo Collina) [#43775](nodejs/node#43775) - \[[`ceb7c150ec`](nodejs/node@ceb7c150ec)] - **meta**: move one or more collaborators to emeritus (Node.js GitHub Bot) [#43770](nodejs/node#43770) - \[[`29bcd47738`](nodejs/node@29bcd47738)] - **net**: fix socket.\_getpeername (Daeyeon Jeong) [#43010](nodejs/node#43010) - \[[`380659daf1`](nodejs/node@380659daf1)] - **process**: use `defineProperty` instead of assignment (Mark S. Miller) [#43907](nodejs/node#43907) - \[[`aba9c8ebea`](nodejs/node@aba9c8ebea)] - **repl**: fix overzealous top-level await (Tobias Nießen) [#43827](nodejs/node#43827) - \[[`1deb6b73b7`](nodejs/node@1deb6b73b7)] - **repl**: use `SafePromiseAll` and `SafePromiseRace` (Antoine du Hamel) [#43758](nodejs/node#43758) - \[[`bf8f2e23ff`](nodejs/node@bf8f2e23ff)] - **src**: refactor DH groups to delete crypto_groups.h (Tobias Nießen) [#43896](nodejs/node#43896) - \[[`9435fbf8cd`](nodejs/node@9435fbf8cd)] - **src**: remove dead code in base64\_encode (Tobias Nießen) [#43979](nodejs/node#43979) - \[[`2c47e58ea0`](nodejs/node@2c47e58ea0)] - **src**: fix regression that a source marker is lost (cola119) [#43086](nodejs/node#43086) - \[[`d084150320`](nodejs/node@d084150320)] - **src**: per-isolate eternal template properties (Chengzhong Wu) [#43802](nodejs/node#43802) - \[[`9f9d00ccbb`](nodejs/node@9f9d00ccbb)] - **src**: merge NativeModuleEnv into NativeModuleLoader (Joyee Cheung) [#43824](nodejs/node#43824) - \[[`bb512904e9`](nodejs/node@bb512904e9)] - **src**: use named struct instead of typedef (Tobias Nießen) [#43881](nodejs/node#43881) - \[[`bb5511e8cc`](nodejs/node@bb5511e8cc)] - **src**: use named enum instead of typedef (Tobias Nießen) [#43880](nodejs/node#43880) - \[[`5db0c8f667`](nodejs/node@5db0c8f667)] - **src**: pass only Isolate\* and env_vars to EnabledDebugList::Parse() (Darshan Sen) [#43668](nodejs/node#43668) - \[[`249365524e`](nodejs/node@249365524e)] - **src**: fix node watchdog race condition (theanarkh) [#43780](nodejs/node#43780) - \[[`17cb27237d`](nodejs/node@17cb27237d)] - **src**: deduplicate `SetALPN` implementations (Tobias Nießen) [#43756](nodejs/node#43756) - \[[`b4c75a96be`](nodejs/node@b4c75a96be)] - **src**: fix `napi_check_object_type_tag()` (Daeyeon Jeong) [#43788](nodejs/node#43788) - \[[`8432d6596f`](nodejs/node@8432d6596f)] - **src**: slim down env-inl.h (Ben Noordhuis) [#43745](nodejs/node#43745) - \[[`2266a4b6d6`](nodejs/node@2266a4b6d6)] - **stream**: improve `respondWithNewView()` (Daeyeon Jeong) [#43866](nodejs/node#43866) - \[[`bf3991b406`](nodejs/node@bf3991b406)] - **stream**: fix 0 transform hwm backpressure (Robert Nagy) [#43685](nodejs/node#43685) - \[[`a057510037`](nodejs/node@a057510037)] - **stream**: initial approach to include strategy options on Readable.toWeb() (txxnano) [#43515](nodejs/node#43515) - \[[`198cf59d2c`](nodejs/node@198cf59d2c)] - **test**: update WPT encoding tests (Kohei Ueno) [#43958](nodejs/node#43958) - \[[`f0ed1aed8d`](nodejs/node@f0ed1aed8d)] - **test**: remove test-whatwg-events-add-event-listener-options-once.js (Feng Yu) [#43877](nodejs/node#43877) - \[[`88505556fe`](nodejs/node@88505556fe)] - **test**: work scheduled in process.nextTick can keep the event loop alive (Andreu Botella) [#43787](nodejs/node#43787) - \[[`81a21946eb`](nodejs/node@81a21946eb)] - **test**: simplify test-tls-set-secure-context (Tobias Nießen) [#43878](nodejs/node#43878) - \[[`61cd11a8a7`](nodejs/node@61cd11a8a7)] - **test**: use `common.mustNotMutateObjectDeep()` in fs tests (LiviaMedeiros) [#43819](nodejs/node#43819) - \[[`b1081dbe12`](nodejs/node@b1081dbe12)] - **test**: fix test http upload timeout (theanarkh) [#43935](nodejs/node#43935) - \[[`efd5e0e925`](nodejs/node@efd5e0e925)] - **test**: simplify ReplStream.wait() (Tobias Nießen) [#43857](nodejs/node#43857) - \[[`ef21ad2996`](nodejs/node@ef21ad2996)] - **test**: merge test-crypto-dh-hash with modp18 test (Tobias Nießen) [#43891](nodejs/node#43891) - \[[`e502c50a90`](nodejs/node@e502c50a90)] - **test**: refactor `test/es-module/test-esm-resolve-type` (Antoine du Hamel) [#43178](nodejs/node#43178) - \[[`c782c3dc69`](nodejs/node@c782c3dc69)] - **test**: ensure NODE_EXTRA_CA_CERTS not set before tests (KrayzeeKev) [#43858](nodejs/node#43858) - \[[`bb6787cb57`](nodejs/node@bb6787cb57)] - **test**: add check to test-fs-readfile-tostring-fail (Richard Lau) [#43850](nodejs/node#43850) - \[[`7571704186`](nodejs/node@7571704186)] - **test**: complete TODO in test/wpt/test-url.js (Kohei Ueno) [#43797](nodejs/node#43797) - \[[`6f1d2dfb9d`](nodejs/node@6f1d2dfb9d)] - **test**: add test on worker process.exit in async modules (Chengzhong Wu) [#43751](nodejs/node#43751) - \[[`776cc3abbd`](nodejs/node@776cc3abbd)] - **test**: use `common.mustNotMutateObjectDeep()` in immutability tests (LiviaMedeiros) [#43196](nodejs/node#43196) - \[[`42f2deb3a0`](nodejs/node@42f2deb3a0)] - **test**: add `common.mustNotMutateObjectDeep()` (LiviaMedeiros) [#43196](nodejs/node#43196) - \[[`f3fc51c508`](nodejs/node@f3fc51c508)] - **test**: fix coverity warning in test (Michael Dawson) [#43631](nodejs/node#43631) - \[[`a9ecba2fa8`](nodejs/node@a9ecba2fa8)] - **test**: mark test-http-client-response-timeout flaky (Tobias Nießen) [#43792](nodejs/node#43792) - \[[`cd0d9ddb7c`](nodejs/node@cd0d9ddb7c)] - **test_runner**: add support for boolean values for `concurrency` option (Lenvin Gonsalves) [#43887](nodejs/node#43887) - \[[`f98020138a`](nodejs/node@f98020138a)] - **test_runner**: validate `timeout` option (Antoine du Hamel) [#43843](nodejs/node#43843) - \[[`58d15b3687`](nodejs/node@58d15b3687)] - **test_runner**: pass signal on timeout (Moshe Atlow) [#43911](nodejs/node#43911) - \[[`8b0248506f`](nodejs/node@8b0248506f)] - **test_runner**: do not report an error when tests are passing (Antoine du Hamel) [#43919](nodejs/node#43919) - \[[`aa8053e1fa`](nodejs/node@aa8053e1fa)] - **test_runner**: recieve and pass AbortSignal (Moshe Atlow) [#43554](nodejs/node#43554) - \[[`f13e4c1be9`](nodejs/node@f13e4c1be9)] - **test_runner**: fix `it` concurrency (Moshe Atlow) [#43757](nodejs/node#43757) - \[[`e404a3ef6d`](nodejs/node@e404a3ef6d)] - **test_runner**: support timeout for tests (Moshe Atlow) [#43505](nodejs/node#43505) - \[[`f28198cc05`](nodejs/node@f28198cc05)] - **test_runner**: catch errors thrown within `describe` (Moshe Atlow) [#43729](nodejs/node#43729) - \[[`bfe0ac6cd0`](nodejs/node@bfe0ac6cd0)] - **tools**: add more options to track flaky tests (Antoine du Hamel) [#43954](nodejs/node#43954) - \[[`17a4e5e775`](nodejs/node@17a4e5e775)] - **tools**: add verbose flag to inactive TSC finder (Rich Trott) [#43913](nodejs/node#43913) - \[[`373304b0c7`](nodejs/node@373304b0c7)] - **tools**: add support for using API key to vuln checking script (Facundo Tuesca) [#43909](nodejs/node#43909) - \[[`ed45088c14`](nodejs/node@ed45088c14)] - **tools**: support versioned node shared libs on z/OS (alexcfyung) [#42256](nodejs/node#42256) - \[[`c9ecd6d21f`](nodejs/node@c9ecd6d21f)] - **tools**: update doc to highlight.js@11.6.0 (Node.js GitHub Bot) [#43870](nodejs/node#43870) - \[[`c92135aa0f`](nodejs/node@c92135aa0f)] - **tools**: update lint-md-dependencies to rollup@2.77.0 (Node.js GitHub Bot) [#43871](nodejs/node#43871) - \[[`e12bf40fd1`](nodejs/node@e12bf40fd1)] - **tools**: update eslint to 8.20.0 (Node.js GitHub Bot) [#43873](nodejs/node#43873) - \[[`09fe9b30a9`](nodejs/node@09fe9b30a9)] - **tools**: add script for vulnerability checking (Facundo Tuesca) [#43362](nodejs/node#43362) - \[[`19e8876877`](nodejs/node@19e8876877)] - **trace_events**: trace net connect event (theanarkh) [#43903](nodejs/node#43903) - \[[`1af7f24143`](nodejs/node@1af7f24143)] - **util**: remove unicode support todo for perf implications (Rhys) [#43762](nodejs/node#43762) - \[[`acfc33ca8c`](nodejs/node@acfc33ca8c)] - **(SEMVER-MINOR)** **util**: add tokens to parseArgs (John Gee) [#43459](nodejs/node#43459) - \[[`f32aec8a6d`](nodejs/node@f32aec8a6d)] - **util**: refactor to use validateObject (Kohei Ueno) [#43769](nodejs/node#43769) - \[[`d7cfd0c5ba`](nodejs/node@d7cfd0c5ba)] - **v8**: serialize BigInt64Array and BigUint64Array (Ben Noordhuis) [#43571](nodejs/node#43571)

PR nodejs#43954 introduced the use of f-strings to `test.py` making the script only work with python versions >= 3.6. Remove that use to make it compatible again with older python versions.

PR #43954 introduced the use of f-strings to `test.py` making the script only work with python versions >= 3.6. Remove that use to make it compatible again with older python versions. PR-URL: #44407 Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Refs: nodejs#43929 (comment) PR-URL: nodejs#43954 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Feng Yu <F3n67u@outlook.com>

PR nodejs#43954 introduced the use of f-strings to `test.py` making the script only work with python versions >= 3.6. Remove that use to make it compatible again with older python versions. PR-URL: nodejs#44407 Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

PR #43954 introduced the use of f-strings to `test.py` making the script only work with python versions >= 3.6. Remove that use to make it compatible again with older python versions. PR-URL: #44407 Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Refs: nodejs/node#43929 (comment) PR-URL: nodejs/node#43954 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Feng Yu <F3n67u@outlook.com>

PR #43954 introduced the use of f-strings to `test.py` making the script only work with python versions >= 3.6. Remove that use to make it compatible again with older python versions. PR-URL: #44407 Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Refs: nodejs#43929 (comment) PR-URL: nodejs#43954 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Feng Yu <F3n67u@outlook.com>

Refs: #43929 (comment) PR-URL: #43954 Backport-PR-URL: #45126 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Feng Yu <F3n67u@outlook.com>

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [node](https://github.com/nodejs/node) | stage | minor | `14.20.1-bullseye-slim` -> `14.21.0-bullseye-slim` | --- ### Release Notes <details> <summary>nodejs/node</summary> ### [`v14.21.0`](https://github.com/nodejs/node/releases/tag/v14.21.0) [Compare Source](nodejs/node@v14.20.1...v14.21.0) ##### Notable changes - **deps**: - update corepack to 0.14.2 (Node.js GitHub Bot) [#44775](nodejs/node#44775) - **src**: - add --openssl-shared-config option (Daniel Bevenius) [#43124](nodejs/node#43124) ##### Commits - \[[`773f587912`](nodejs/node@773f587912)] - **deps**: cherry-pick [libuv/libuv@`3a7b955`](libuv/libuv@3a7b955) (Ben Noordhuis) [#43950](nodejs/node#43950) - \[[`a1dea66956`](nodejs/node@a1dea66956)] - **deps**: cherry-pick [libuv/libuv@`abb109f`](libuv/libuv@abb109f) (Ben Noordhuis) [#43950](nodejs/node#43950) - \[[`98c49d81f5`](nodejs/node@98c49d81f5)] - **deps**: update corepack to 0.14.2 (Node.js GitHub Bot) [#44775](nodejs/node#44775) - \[[`18c43c8518`](nodejs/node@18c43c8518)] - **deps**: update timezone to tz2022e (Richard Lau) [#45094](nodejs/node#45094) - \[[`a1f8e4db48`](nodejs/node@a1f8e4db48)] - **deps**: update corepack to 0.14.1 (Node.js GitHub Bot) [#44704](nodejs/node#44704) - \[[`e55389ca86`](nodejs/node@e55389ca86)] - **deps**: update corepack to 0.14.0 (Node.js GitHub Bot) [#44509](nodejs/node#44509) - \[[`0227462418`](nodejs/node@0227462418)] - **deps**: update corepack to 0.13.0 (Node.js GitHub Bot) [#44318](nodejs/node#44318) - \[[`ee24c320ea`](nodejs/node@ee24c320ea)] - **deps**: update corepack to 0.12.3 (Node.js GitHub Bot) [#44229](nodejs/node#44229) - \[[`28e9891449`](nodejs/node@28e9891449)] - **deps**: update corepack to 0.12.2 (Node.js GitHub Bot) [#44159](nodejs/node#44159) - \[[`b6972c9df2`](nodejs/node@b6972c9df2)] - **deps**: update corepack to 0.12.1 (Node.js GitHub Bot) [#43965](nodejs/node#43965) - \[[`9d6cb3b5f1`](nodejs/node@9d6cb3b5f1)] - **deps**: update corepack to 0.12.0 (Node.js GitHub Bot) [#43748](nodejs/node#43748) - \[[`fa6c276b4f`](nodejs/node@fa6c276b4f)] - **deps**: update Corepack to 0.11.2 (Maël Nison) [#43402](nodejs/node#43402) - \[[`4f83d75626`](nodejs/node@4f83d75626)] - **(SEMVER-MAJOR)** **src,doc,test**: add --openssl-shared-config option (Daniel Bevenius) [#43124](nodejs/node#43124) - \[[`9487028043`](nodejs/node@9487028043)] - **test**: fix intl tests on small-icu builds (Antoine du Hamel) [#41939](nodejs/node#41939) - \[[`a1d52097f8`](nodejs/node@a1d52097f8)] - **tools**: add more options to track flaky tests (Antoine du Hamel) [#43954](nodejs/node#43954) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.walbeck.it/mwalbeck/docker-jellyfin-livestream/pulls/195 Co-authored-by: renovate-bot <bot@walbeck.it> Co-committed-by: renovate-bot <bot@walbeck.it>

Refs: nodejs/node#43954

tools: add more options to track flaky tests

895db5a

Refs: nodejs#43929 (comment)

nodejs-github-bot added meta Issues and PRs related to the general management of the project. test Issues and PRs related to the tests. tools Issues and PRs related to the tools directory. labels Jul 23, 2022

mcollina approved these changes Jul 23, 2022

View reviewed changes

tniessen approved these changes Jul 23, 2022

View reviewed changes

tniessen reviewed Jul 23, 2022

View reviewed changes

tools/test.py Outdated Show resolved Hide resolved

kvakil reviewed Jul 23, 2022

View reviewed changes

tools/test.py Outdated Show resolved Hide resolved

fixup! tools: add more options to track flaky tests

6a816a2

aduh95 added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jul 23, 2022

fixup! tools: add more options to track flaky tests

a189bff

F3n67u approved these changes Jul 25, 2022

View reviewed changes

F3n67u added commit-queue Add this label to land a pull request using GitHub Actions. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Jul 25, 2022

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 25, 2022

nodejs-github-bot merged commit 2e4bcfc into nodejs:main Jul 25, 2022

aduh95 deleted the track-flaky-tests branch July 25, 2022 19:52

danielleadams mentioned this pull request Jul 26, 2022

v18.7.0 proposal #43993

Merged

targos mentioned this pull request Aug 3, 2022

v16.17.0 release proposal #44098

Merged

MoLow mentioned this pull request Aug 21, 2022

python tools/test.py -J --mode=release parallel/test-* #44330

Closed

santigimeno mentioned this pull request Aug 26, 2022

tools: don't use f-strings in test.py #44407

Merged

aduh95 mentioned this pull request Oct 22, 2022

[v14.x backport] tools: add more options to track flaky tests #45126

Closed

danielleadams added the backported-to-v14.x label Oct 26, 2022

danielleadams mentioned this pull request Nov 1, 2022

v14.21.0 proposal #44889

Merged

richardlau pushed a commit to nodejs/build that referenced this pull request Sep 28, 2023

windows: measure flakiness of failed tests (#3056)

726a7ca

Refs: nodejs/node#43954

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools: add more options to track flaky tests #43954

tools: add more options to track flaky tests #43954

aduh95 commented Jul 23, 2022 •

edited

nodejs-github-bot commented Jul 23, 2022

tniessen commented Jul 23, 2022

tniessen commented Jul 23, 2022

mcollina left a comment

tniessen left a comment

lpinca commented Jul 23, 2022 •

edited

aduh95 commented Jul 23, 2022

lpinca commented Jul 23, 2022

tniessen commented Jul 23, 2022

lpinca commented Jul 23, 2022 •

edited

tniessen commented Jul 23, 2022

lpinca commented Jul 23, 2022 •

edited

tniessen left a comment

nodejs-github-bot commented Jul 25, 2022

	if options.flaky_tests not in [RUN, SKIP, DONTCARE]:
	print("Unknown flaky-tests mode %s" % options.flaky_tests)
	return False

tools: add more options to track flaky tests #43954

tools: add more options to track flaky tests #43954

Conversation

aduh95 commented Jul 23, 2022 • edited

nodejs-github-bot commented Jul 23, 2022

tniessen commented Jul 23, 2022

tniessen commented Jul 23, 2022

mcollina left a comment

Choose a reason for hiding this comment

tniessen left a comment

Choose a reason for hiding this comment

lpinca commented Jul 23, 2022 • edited

aduh95 commented Jul 23, 2022

lpinca commented Jul 23, 2022

tniessen commented Jul 23, 2022

lpinca commented Jul 23, 2022 • edited

tniessen commented Jul 23, 2022

lpinca commented Jul 23, 2022 • edited

tniessen left a comment

Choose a reason for hiding this comment

nodejs-github-bot commented Jul 25, 2022

aduh95 commented Jul 23, 2022 •

edited

lpinca commented Jul 23, 2022 •

edited

lpinca commented Jul 23, 2022 •

edited

lpinca commented Jul 23, 2022 •

edited