Browser test is failing

[mdjermanovic]

ESLint version

main

What problem do you want to solve?

Browser test fails on CI, on Jenkins, and for me locally.

CI (today, 2023-09-16): https://github.com/eslint/eslint/actions/runs/6206203980/job/16855618328

Running unit tests on browsers
@babel/preset-env: `DEBUG` option

Using targets:
{
  "chrome": "109",
  "edge": "114",
  "firefox": "115",
  "ios": "15.6",
  "opera": "100",
  "safari": "15.6"
}

Using modules transform: auto

Using plugins:
  transform-unicode-sets-regex { chrome < 112, firefox < 116, ios, safari < tp }
  proposal-class-static-block { ios < 16.4, safari < 16.4 }
  syntax-private-property-in-object
  syntax-class-properties
  syntax-numeric-separator
  syntax-nullish-coalescing-operator
  syntax-optional-chaining
  syntax-json-strings
  syntax-optional-catch-binding
  transform-parameters { ios < 16.3, safari < 16.3 }
  syntax-async-generators
  syntax-object-rest-spread
  proposal-export-namespace-from { }
  syntax-dynamic-import
  syntax-top-level-await
  syntax-import-meta

Using polyfills: No polyfills were added, since the `useBuiltIns` option was not set.
[BABEL] Note: The code generator has deoptimised the styling of /home/runner/work/eslint/eslint/node_modules/graphemer/lib/Graphemer.js as it exceeds the max of 500KB.

Execution of 1 workers started at 

[0-0] RUNNING in chrome - file:///tests/lib/linter/linter.js
[0-0]  Error:  Test failed due to following error(s):
  - index.js?v=6d81e957: Uncaught SyntaxError: The requested module '/node_modules/split2/index.js?v=6d81e957' does not provide an export named 'default'

[0-0] FAILED in chrome - file:///tests/lib/linter/linter.js

 "concise" Reporter:
========= Your concise report ==========
chrome (v116.0.5845.187) on linux
❌ Failed to setup tests, no tests found


Spec Files:	 0 passed, 1 failed, 1 total (100% completed) in 00:00:20

/home/runner/work/eslint/eslint/node_modules/shelljs/src/common.js:401
      if (config.fatal) throw e;
                        ^

Error: exec:
    at Object.error (/home/runner/work/eslint/eslint/node_modules/shelljs/src/common.js:110:27)
    at execSync (/home/runner/work/eslint/eslint/node_modules/shelljs/src/exec.js:120:12)
    at _exec (/home/runner/work/eslint/eslint/node_modules/shelljs/src/exec.js:223:12)
    at /home/runner/work/eslint/eslint/node_modules/shelljs/src/common.js:335:23
    at Function.target.wdio (/home/runner/work/eslint/eslint/Makefile.js:634:24)
    at Object.global.target.<computed> [as wdio] (/home/runner/work/eslint/eslint/node_modules/shelljs/make.js:36:40)
    at /home/runner/work/eslint/eslint/node_modules/shelljs/make.js:48:27
    at Array.forEach (<anonymous>)
    at Timeout._onTimeout (/home/runner/work/eslint/eslint/node_modules/shelljs/make.js:46:10)
    at listOnTimeout (node:internal/timers:559:17)
##[error]Process completed with exit code 1.

Jenkins (last release, 2023-09-08)

Running unit tests on browsers
@babel/preset-env: `DEBUG` option

Using targets:
{
  "chrome": "109",
  "edge": "114",
  "firefox": "115",
  "ios": "15.6",
  "opera": "100",
  "safari": "15.6"
}

Using modules transform: auto

Using plugins:
  transform-unicode-sets-regex { chrome < 112, firefox < 116, ios, safari }
  proposal-class-static-block { ios < 16.4, safari < 16.4 }
  syntax-private-property-in-object
  syntax-class-properties
  syntax-numeric-separator
  syntax-nullish-coalescing-operator
  syntax-optional-chaining
  syntax-json-strings
  syntax-optional-catch-binding
  transform-parameters { ios < 16.3, safari < 16.3 }
  syntax-async-generators
  syntax-object-rest-spread
  proposal-export-namespace-from { }
  syntax-dynamic-import
  syntax-top-level-await
  syntax-import-meta

Using polyfills: No polyfills were added, since the `useBuiltIns` option was not set.

Execution of 1 workers started at 2023-09-08T17:57:54.977Z

Error: Error: Command failed: which google-chrome

    at ChildProcess.exithandler (node:child_process:402:12)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess.emit (node:domain:489:12)
    at maybeClose (node:internal/child_process:1100:16)
    at Socket.<anonymous> (node:internal/child_process:458:11)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at Pipe.<anonymous> (node:net:301:12)

Locally (today, 2023-09-16; Windows; I deleted node_modules, run npm install and then node Makefile.js wdio):

Running unit tests on browsers
@babel/preset-env: `DEBUG` option

Using targets:
{
  "chrome": "109",
  "edge": "114",
  "firefox": "115",
  "ios": "15.6",
  "opera": "100",
  "safari": "15.6"
}

Using modules transform: auto

Using plugins:
  transform-unicode-sets-regex { chrome < 112, firefox < 116, ios, safari < tp }
  proposal-class-static-block { ios < 16.4, safari < 16.4 }
  syntax-private-property-in-object
  syntax-class-properties
  syntax-numeric-separator
  syntax-nullish-coalescing-operator
  syntax-optional-chaining
  syntax-json-strings
  syntax-optional-catch-binding
  transform-parameters { ios < 16.3, safari < 16.3 }
  syntax-async-generators
  syntax-object-rest-spread
  proposal-export-namespace-from { }
  syntax-dynamic-import
  syntax-top-level-await
  syntax-import-meta

Using polyfills: No polyfills were added, since the `useBuiltIns` option was not set.
[BABEL] Note: The code generator has deoptimised the styling of C:\milos\eslint\node_modules\graphemer\lib\Graphemer.js as it exceeds the max of 500KB.

Execution of 1 workers started at 2023-09-16T20:24:10.102Z

[0-0] RUNNING in chrome - file:///C:/milos/eslint/tests/lib/linter/linter.js
[0-0] node:events:498
[0-0]       throw er; // Unhandled 'error' event
[0-0]       ^
[0-0]
[0-0] Error: spawn C:\Users\golde\AppData\Local\Temp\chromedriver\win64-116.0.5845.96\chromedriver-win64\chromedriver.exe ENOENT
[0-0]     at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
[0-0]     at onErrorNT (node:internal/child_process:478:16)
[0-0]     at processTicksAndRejections (node:internal/process/task_queues:83:21)
[0-0] Emitted 'error' event on ChildProcess instance at:
[0-0]     at Process.ChildProcess._handle.onexit (node:internal/child_process:289:12)
[0-0]     at onErrorNT (node:internal/child_process:478:16)
[0-0]     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
[0-0]   errno: -4058,
[0-0]   code: 'ENOENT',
[0-0]   syscall: 'spawn C:\\Users\\golde\\AppData\\Local\\Temp\\chromedriver\\win64-116.0.5845.96\\chromedriver-win64\\chromedriver.exe',
[0-0]   path: 'C:\\Users\\golde\\AppData\\Local\\Temp\\chromedriver\\win64-116.0.5845.96\\chromedriver-win64\\chromedriver.exe',
[0-0]   spawnargs: [ '--port=52378', '--allowed-origins=*', '--allowed-ips=0.0.0.0' ]
[0-0] }
[0-0] FAILED in chrome - file:///C:/milos/eslint/tests/lib/linter/linter.js

Spec Files:      0 passed, 1 failed, 1 total (100% completed) in 00:00:03

C:\milos\eslint\node_modules\shelljs\src\common.js:401
      if (config.fatal) throw e;
                        ^

Error: exec: [0-0] node:events:498
[0-0]       throw er; // Unhandled 'error' event
[0-0]       ^
[0-0]
[0-0] Error: spawn C:/Users/golde/AppData/Local/Temp/chromedriver/win64-116.0.5845.96/chromedriver-win64/chromedriver.exe ENOENT
[0-0]     at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
[0-0]     at onErrorNT (node:internal/child_process:478:16)
[0-0]     at processTicksAndRejections (node:internal/process/task_queues:83:21)
[0-0] Emitted 'error' event on ChildProcess instance at:
[0-0]     at Process.ChildProcess._handle.onexit (node:internal/child_process:289:12)
[0-0]     at onErrorNT (node:internal/child_process:478:16)
[0-0]     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
[0-0]   errno: -4058,
[0-0]   code: 'ENOENT',
[0-0]   syscall: 'spawn C://Users//golde//AppData//Local//Temp//chromedriver//win64-116.0.5845.96//chromedriver-win64//chromedriver.exe',
[0-0]   path: 'C://Users//golde//AppData//Local//Temp//chromedriver//win64-116.0.5845.96//chromedriver-win64//chromedriver.exe',
[0-0]   spawnargs: [ '--port=52378', '--allowed-origins=*', '--allowed-ips=0.0.0.0' ]
[0-0] }

    at Object.error (C:\milos\eslint\node_modules\shelljs\src\common.js:110:27)
    at execSync (C:\milos\eslint\node_modules\shelljs\src\exec.js:120:12)
    at _exec (C:\milos\eslint\node_modules\shelljs\src\exec.js:223:12)
    at C:\milos\eslint\node_modules\shelljs\src\common.js:335:23
    at Function.target.wdio (C:\milos\eslint\Makefile.js:634:24)
    at Object.global.target.<computed> [as wdio] (C:\milos\eslint\node_modules\shelljs\make.js:36:40)
    at C:\milos\eslint\node_modules\shelljs\make.js:48:27
    at Array.forEach (<anonymous>)
    at Timeout._onTimeout (C:\milos\eslint\node_modules\shelljs\make.js:46:10)
    at listOnTimeout (node:internal/timers:559:17)

What do you think is the correct solution?

To fix the test.

Participation

• I am willing to submit a pull request for this change.

Additional comments

All three problems seem different. The CI problem might be related to webdriverio/webdriverio#11175

cc @christian-bromann

[christian-bromann]

Taking a look 👀

[christian-bromann]

I've fixed the issue and builds should pass now. Unfortunately this kind if bug is hard to detect in the project pipeline so we have a component testing example repository which we haven't updated in time to catch the problem. I will make sure to create some sort of automatism behind this so we can better act on this next time.

[mdjermanovic]

Seems to work on CI now, thanks! Should we maybe update some dependencies (@wdio/browser-runner?) in package.json to make sure it will work for everyone?

[christian-bromann]

Should we maybe update some dependencies (@wdio/browser-runner?) in package.json to make sure it will work for everyone?

I would have made a PR if necessary but the versions are defined with a ^ so you will get all bug fixes for issues that come in. We use semver so no breaking changes expected.

[nzakas]

It looks like this is resolved now? Can we close this issue?

[nzakas]

We've got another failure: https://github.com/eslint/eslint/actions/runs/6353292043?pr=17510

[christian-bromann]

Thanks for letting me know, will push a fix in a minute. I will have to work out a better system to not have this going on forever. Basically we run a bunch of e2e tests on the WebdriverIO side but they don't seem to catch these dependency issues. ESM vs. CJS is fun!

Edit: should be fixed now.

[nzakas]

Looks like it, thanks!

[mdjermanovic]

Now It works on CI, and for me locally. Thanks for fixing this!

The Command failed: which google-chrome error we were getting on Jenkins looks like a different problem.

We didn't try running the browser test on Jenkins again as it's still removed from npm test (#17550). I'm not sure if it's important to run the browser test on Jenkins since we're already running it on CI, so perhaps we could keep it omitted from npm test. @nzakas what do you think?

[nzakas]

I'm fine with not running the browser test during a release. 👍

[mdjermanovic]

Alright, then we can close this. I'll prepare a PR to update the comment in Makefile.