Mocha's own CI tests appear to be unstable #4103

broofa · 2019-11-24T15:48:14Z

Mocha's own internal tests appear to be unstable, both in CI and in local development.

On Appveyor:
While working on #3979 (CC @craigtaub @boneskull) I have had at least two occasions where Mocha's CI tests (unrelated to my work) fail, only to succeed when I force the tests to rerun by adding an empty commit to the PR. You can see the most recent example of this here:

Initial test failure [Appveyor]
Subsquent success [Appveyor]

(empty) comparison of the two git refs

In local development:
On master (i.e. not my code) I ran the following to repeatedly run the watch.spec.js tests:
while [ 1 ]; do node bin/mocha --timeout 10000 --slow 3750

In 5 test runs (see log below), it succeeded twice and failed three times, all for different reasons. :-/

$ while [ 1 ]; do node bin/mocha --timeout 10000 --slow 3750 "test/integration/options/watch.spec.js" ; done


  --watch
    when enabled
      ✓ reruns test when watched test file is touched (2031ms)
      ✓ reruns test when file matching --watch-files changes (2017ms)
      ✓ reruns test when file matching --watch-files is added (2018ms)
      ✓ reruns test when file matching --watch-files is removed (2022ms)
      ✓ does not rerun test when file not matching --watch-files is changed (2026ms)
      ✓ picks up new test files when they are added (2019ms)
      ✓ reruns test when file matching --extension is changed (2014ms)
      ✓ reruns test when file starting with . and matching --extension is changed (2023ms)
      ✓ ignores files in "node_modules" and ".git" by default (2027ms)
      ✓ ignores files matching --watch-ignore (2015ms)
      ✓ reloads test files when they change (2024ms)
      ✓ reloads test dependencies when they change (2021ms)
      ✓ respects --fgrep on re-runs (2022ms)


  13 passing (26s)



  --watch
    when enabled
      ✓ reruns test when watched test file is touched (2029ms)
      ✓ reruns test when file matching --watch-files changes (2025ms)
      ✓ reruns test when file matching --watch-files is added (2019ms)
      ✓ reruns test when file matching --watch-files is removed (2028ms)
      ✓ does not rerun test when file not matching --watch-files is changed (2017ms)
      ✓ picks up new test files when they are added (2024ms)
      ✓ reruns test when file matching --extension is changed (2021ms)
      ✓ reruns test when file starting with . and matching --extension is changed (2027ms)
      ✓ ignores files in "node_modules" and ".git" by default (2017ms)
      ✓ ignores files matching --watch-ignore (2017ms)
      1) reloads test files when they change
      ✓ reloads test dependencies when they change (2019ms)
      ✓ respects --fgrep on re-runs (2026ms)


  12 passing (26s)
  1 failing

  1) --watch
       when enabled
         reloads test files when they change:
     SyntaxError: Unexpected end of JSON input
      at JSON.parse (<anonymous>)
      at /Users/kieffer/repos/mocha/test/integration/options/watch.spec.js:300:24
      at Array.map (<anonymous>)
      at /Users/kieffer/repos/mocha/test/integration/options/watch.spec.js:300:10
      at processTicksAndRejections (internal/process/task_queues.js:93:5)





  --watch
    when enabled
      ✓ reruns test when watched test file is touched (2036ms)
      ✓ reruns test when file matching --watch-files changes (2027ms)
      1) reruns test when file matching --watch-files is added
      ✓ reruns test when file matching --watch-files is removed (2024ms)
      ✓ does not rerun test when file not matching --watch-files is changed (2026ms)
      ✓ picks up new test files when they are added (2016ms)
      ✓ reruns test when file matching --extension is changed (2022ms)
      ✓ reruns test when file starting with . and matching --extension is changed (2022ms)
      ✓ ignores files in "node_modules" and ".git" by default (2030ms)
      ✓ ignores files matching --watch-ignore (2021ms)
      ✓ reloads test files when they change (2028ms)
      ✓ reloads test dependencies when they change (2023ms)
      ✓ respects --fgrep on re-runs (2025ms)


  12 passing (26s)
  1 failing

  1) --watch
       when enabled
         reruns test when file matching --watch-files is added:
     SyntaxError: Unexpected end of JSON input
      at JSON.parse (<anonymous>)
      at /Users/kieffer/repos/mocha/test/integration/options/watch.spec.js:300:24
      at Array.map (<anonymous>)
      at /Users/kieffer/repos/mocha/test/integration/options/watch.spec.js:300:10
      at processTicksAndRejections (internal/process/task_queues.js:93:5)





  --watch
    when enabled
      ✓ reruns test when watched test file is touched (2068ms)
      ✓ reruns test when file matching --watch-files changes (2026ms)
      ✓ reruns test when file matching --watch-files is added (2061ms)
      1) reruns test when file matching --watch-files is removed
      ✓ does not rerun test when file not matching --watch-files is changed (2027ms)
      ✓ picks up new test files when they are added (2027ms)
      ✓ reruns test when file matching --extension is changed (2020ms)
      ✓ reruns test when file starting with . and matching --extension is changed (2023ms)
      ✓ ignores files in "node_modules" and ".git" by default (2025ms)
      ✓ ignores files matching --watch-ignore (2021ms)
      ✓ reloads test files when they change (2019ms)
      ✓ reloads test dependencies when they change (2019ms)
      ✓ respects --fgrep on re-runs (2017ms)


  12 passing (27s)
  1 failing

  1) --watch
       when enabled
         reruns test when file matching --watch-files is removed:
     SyntaxError: Unexpected end of JSON input
      at JSON.parse (<anonymous>)
      at /Users/kieffer/repos/mocha/test/integration/options/watch.spec.js:300:24
      at Array.map (<anonymous>)
      at /Users/kieffer/repos/mocha/test/integration/options/watch.spec.js:300:10
      at processTicksAndRejections (internal/process/task_queues.js:93:5)





  --watch
    when enabled
      ✓ reruns test when watched test file is touched (2038ms)
      ✓ reruns test when file matching --watch-files changes (2022ms)
      ✓ reruns test when file matching --watch-files is added (2019ms)
      ✓ reruns test when file matching --watch-files is removed (2018ms)
      ✓ does not rerun test when file not matching --watch-files is changed (2022ms)
      ✓ picks up new test files when they are added (2025ms)
      ✓ reruns test when file matching --extension is changed (2022ms)
      ✓ reruns test when file starting with . and matching --extension is changed (2025ms)
      ✓ ignores files in "node_modules" and ".git" by default (2023ms)
      ✓ ignores files matching --watch-ignore (2020ms)
      ✓ reloads test files when they change (2021ms)
      ✓ reloads test dependencies when they change (2032ms)
      ✓ respects --fgrep on re-runs (2015ms)


  13 passing (26s)

^C

$

The text was updated successfully, but these errors were encountered:

juergba · 2019-11-24T17:27:15Z

The watch tests have been stable since weeks/months, locally on my Windows10 and with our CI tests.
I did have problems with them, when I didn't keep my package-lock up to date. So try npm install and they (hopefully) should work.

There is one test with uncaught exceptions, which is unstable once upon a time. When the test servers are busy and our test run slow, then it throws errorCode 7 instead of 2. I will have a look at it next week.

broofa · 2019-11-24T20:50:00Z

Reinstalled from scratch (rm -fr node_modules package-lock.json && npm install). The install error'ed out (which I've seen before), but I'm able to run the while-loop test, which now passes consistently. Weird. I'm not sure what's going on.

kieffer@TRS-80$ npm i
npm WARN deprecated coffee-script@1.12.7: CoffeeScript on NPM has moved to "coffeescript" (no hyphen)
npm WARN deprecated sw-precache@5.2.1: Please migrate to Workbox: https://developers.google.com/web/tools/workbox/guides/migrations/migrate-from-sw
npm WARN deprecated sw-toolbox@3.6.0: Please migrate to Workbox: https://developers.google.com/web/tools/workbox/guides/migrations/migrate-from-sw
npm WARN deprecated gulp-header@1.8.12: Removed event-stream from gulp-header
npm WARN deprecated @types/vfile-message@2.0.0: This is a stub types definition. vfile-message provides its own type definitions, so you do not need this installed.

> fsevents@1.2.9 install /Users/kieffer/repos/mocha/node_modules/@11ty/eleventy/node_modules/fsevents
> node install

node-pre-gyp WARN Using request for node-pre-gyp https download
[fsevents] Success: "/Users/kieffer/repos/mocha/node_modules/@11ty/eleventy/node_modules/fsevents/lib/binding/Release/node-v72-darwin-x64/fse.node" is installed via remote

> fsevents@1.2.9 install /Users/kieffer/repos/mocha/node_modules/browser-sync/node_modules/fsevents
> node install

node-pre-gyp WARN Using request for node-pre-gyp https download
[fsevents] Success: "/Users/kieffer/repos/mocha/node_modules/browser-sync/node_modules/fsevents/lib/binding/Release/node-v72-darwin-x64/fse.node" is installed via remote

> fsevents@1.2.9 install /Users/kieffer/repos/mocha/node_modules/nunjucks/node_modules/fsevents
> node install

node-pre-gyp WARN Using request for node-pre-gyp https download
[fsevents] Success: "/Users/kieffer/repos/mocha/node_modules/nunjucks/node_modules/fsevents/lib/binding/Release/node-v72-darwin-x64/fse.node" is installed via remote

> fsevents@1.2.9 install /Users/kieffer/repos/mocha/node_modules/watchify/node_modules/fsevents
> node install

node-pre-gyp WARN Using request for node-pre-gyp https download
[fsevents] Success: "/Users/kieffer/repos/mocha/node_modules/watchify/node_modules/fsevents/lib/binding/Release/node-v72-darwin-x64/fse.node" is installed via remote

> canvas@2.6.0 install /Users/kieffer/repos/mocha/node_modules/canvas
> node-pre-gyp install --fallback-to-build

node-pre-gyp WARN Using request for node-pre-gyp https download
[canvas] Success: "/Users/kieffer/repos/mocha/node_modules/canvas/build/Release/canvas.node" is installed via remote

> sharp@0.21.3 install /Users/kieffer/repos/mocha/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

info sharp Using cached /Users/kieffer/.npm/_libvips/libvips-8.7.0-darwin-x64.tar.gz
prebuild-install WARN install No prebuilt binaries found (target=12.10.0 runtime=node arch=x64 libc= platform=darwin)
  TOUCH Release/obj.target/libvips-cpp.stamp
  CXX(target) Release/obj.target/sharp/src/common.o
In file included from ../src/common.cc:27:
../src/common.h:78:20: error: no member named 'Handle' in namespace 'v8'
  bool HasAttr(v8::Handle<v8::Object> obj, std::string attr);
               ~~~~^
../src/common.h:78:37: error: expected '(' for function-style cast or type construction
  bool HasAttr(v8::Handle<v8::Object> obj, std::string attr);
                          ~~~~~~~~~~^
../src/common.h:78:39: error: use of undeclared identifier 'obj'
  bool HasAttr(v8::Handle<v8::Object> obj, std::string attr);
                                      ^
../src/common.h:78:56: error: expected '(' for function-style cast or type construction
  bool HasAttr(v8::Handle<v8::Object> obj, std::string attr);
                                           ~~~~~~~~~~~ ^
../src/common.h:79:29: error: no member named 'Handle' in namespace 'v8'
  std::string AttrAsStr(v8::Handle<v8::Object> obj, std::string attr);
                        ~~~~^
../src/common.h:79:46: error: expected '(' for function-style cast or type construction
  std::string AttrAsStr(v8::Handle<v8::Object> obj, std::string attr);
                                   ~~~~~~~~~~^
../src/common.h:79:48: error: use of undeclared identifier 'obj'
  std::string AttrAsStr(v8::Handle<v8::Object> obj, std::string attr);
                                               ^
../src/common.h:79:65: error: expected '(' for function-style cast or type construction
  std::string AttrAsStr(v8::Handle<v8::Object> obj, std::string attr);
                                                    ~~~~~~~~~~~ ^
../src/common.h:80:38: error: no member named 'Handle' in namespace 'v8'
  std::vector<double> AttrAsRgba(v8::Handle<v8::Object> obj, std::string attr);
                                 ~~~~^
../src/common.h:80:55: error: expected '(' for function-style cast or type construction
  std::vector<double> AttrAsRgba(v8::Handle<v8::Object> obj, std::string attr);
                                            ~~~~~~~~~~^
../src/common.h:80:57: error: use of undeclared identifier 'obj'
  std::vector<double> AttrAsRgba(v8::Handle<v8::Object> obj, std::string attr);
                                                        ^
../src/common.h:80:74: error: expected '(' for function-style cast or type construction
  std::vector<double> AttrAsRgba(v8::Handle<v8::Object> obj, std::string attr);
                                                             ~~~~~~~~~~~ ^
../src/common.h:81:37: warning: variable templates are a C++14 extension [-Wc++14-extensions]
  template<typename T> v8::Local<T> AttrAs(v8::Handle<v8::Object> obj, std::string attr) {
                                    ^
../src/common.h:81:48: error: no member named 'Handle' in namespace 'v8'
  template<typename T> v8::Local<T> AttrAs(v8::Handle<v8::Object> obj, std::string attr) {
                                           ~~~~^
../src/common.h:81:65: error: expected '(' for function-style cast or type construction
  template<typename T> v8::Local<T> AttrAs(v8::Handle<v8::Object> obj, std::string attr) {
                                                      ~~~~~~~~~~^
../src/common.h:81:67: error: use of undeclared identifier 'obj'
  template<typename T> v8::Local<T> AttrAs(v8::Handle<v8::Object> obj, std::string attr) {
                                                                  ^
../src/common.h:81:84: error: expected '(' for function-style cast or type construction
  template<typename T> v8::Local<T> AttrAs(v8::Handle<v8::Object> obj, std::string attr) {
                                                                       ~~~~~~~~~~~ ^
../src/common.h:81:89: error: expected ';' at end of declaration
  template<typename T> v8::Local<T> AttrAs(v8::Handle<v8::Object> obj, std::string attr) {
                                                                                        ^
                                                                                        ;
../src/common.h:81:90: error: expected unqualified-id
  template<typename T> v8::Local<T> AttrAs(v8::Handle<v8::Object> obj, std::string attr) {
                                                                                         ^
../src/common.h:84:26: warning: variable templates are a C++14 extension [-Wc++14-extensions]
  template<typename T> T AttrTo(v8::Handle<v8::Object> obj, std::string attr) {
                         ^
../src/common.h:84:37: error: no member named 'Handle' in namespace 'v8'
  template<typename T> T AttrTo(v8::Handle<v8::Object> obj, std::string attr) {
                                ~~~~^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
2 warnings and 20 errors generated.
make: *** [Release/obj.target/sharp/src/common.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/kieffer/.nvm/versions/node/v12.10.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:209:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
gyp ERR! System Darwin 18.7.0
gyp ERR! command "/Users/kieffer/.nvm/versions/node/v12.10.0/bin/node" "/Users/kieffer/.nvm/versions/node/v12.10.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/kieffer/repos/mocha/node_modules/sharp
gyp ERR! node -v v12.10.0
gyp ERR! node-gyp -v v5.0.5
gyp ERR! not ok
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! sharp@0.21.3 install: `(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the sharp@0.21.3 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/kieffer/.npm/_logs/2019-11-24T20_31_19_055Z-debug.log

boneskull · 2019-11-25T23:45:48Z

sharp doesn't seem to work in Node.js v12. you only need it to build the website for production. you can probably get away with npm install --ignore-scripts

boneskull · 2019-11-25T23:46:04Z

(or nvm use 10)

broofa added the unconfirmed-bug label Nov 24, 2019

broofa changed the title ~~Mocha's own CI tests appear are unstable~~ Mocha's own CI tests appear to be unstable Nov 24, 2019

juergba mentioned this issue Nov 25, 2019

Fix unstable CI uncaughtException test #4107

Merged

juergba added area: async related to asynchronous use of Mocha type: bug a defect, confirmed by a maintainer qa and removed unconfirmed-bug labels Jan 13, 2020

juergba closed this as completed Jan 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mocha's own CI tests appear to be unstable #4103

Mocha's own CI tests appear to be unstable #4103

broofa commented Nov 24, 2019

juergba commented Nov 24, 2019

broofa commented Nov 24, 2019

boneskull commented Nov 25, 2019

boneskull commented Nov 25, 2019

Mocha's own CI tests appear to be unstable #4103

Mocha's own CI tests appear to be unstable #4103

Comments

broofa commented Nov 24, 2019

juergba commented Nov 24, 2019

broofa commented Nov 24, 2019

boneskull commented Nov 25, 2019

boneskull commented Nov 25, 2019