Issues with Weather Tests. - Tests currently disabled.

[MichMich]

For some unknown reason most of the weather module tests seem to fail (both on Travis as well as local) due to timeout issues. To be able to continue the development I have disabled the weather tests by adding a skip statement: https://github.com/MichMich/MagicMirror/blob/develop/tests/e2e/modules/weather_spec.js#L18

This means the weather tests are currently not used. This could impact the quality of the code. I can't seem to find any reason why these tests time out.

If anyone is able to solve this issue and get travis to successfully complete the tests, please send a PR. Your contribution would be highly appreciated.

Thanks!

[roramirez]

After merge #1849 I could check it

[MichMich]

Fixed by reverting electron. See #1800

[sdetweil]

still failing today
see pr #1858

[MichMich]

They run ok now. They timeout after 10sec. We might need to stretch that 10 sec.

[sdetweil]

The code says if under CI then timeout is 30 seconds

[MichMich]

This line (for example) uses 10000ms:
https://github.com/MichMich/MagicMirror/blob/69a887fb05af79071eb20875c1c4a856627f7811/tests/e2e/modules/weather_spec.js#L48

[sdetweil]

Ok. Other tests call a function in global_setup to set timeouts. Figured they all would

[roramirez]

I've try catch the failed of this and apparently is because is too slow to run these tests. There a other line
https://github.com/MichMich/MagicMirror/blob/69a887fb05af79071eb20875c1c4a856627f7811/tests/e2e/modules/weather_spec.js#L11
for a wait. Maybe @fewieden can check a look of this.

[fewieden]

@roramirez @MichMich I tried this morning the newest version of develop and run the tests locally, they all passed after ~7s.

What is the setup I need to reproduce the issue?

Since this seems to be still relevant, we should reopen the issue.

[roramirez]

Hi @fewieden

You should try into a similar environment where is running like Travis. Maybe you can using the follow guide (I'm use this one) https://jonlabelle.com/snippets/view/markdown/run-travis-ci-builds-locally-with-docker.

Or also requests the debug mode for your Fork in Travis
https://docs.travis-ci.com/user/running-build-in-debug-mode/#enabling-debug-mode

Since this seems to be still relevant, we should reopen the issue.

Yes. Should reopen this issue

[roramirez]

@fewieden I forget other thing. Also should run the test against the PR #1840

[fewieden]

@roramirez the PR that you tried to reference is a link to this issue.

Therefore I tried https://jonlabelle.com/snippets/view/markdown/run-travis-ci-builds-locally-with-docker to debug this commit which had the original build log.

Executing the tests inside the docker container resulted in a different error

What I found is https://stackoverflow.com/questions/50642308/webdriverexception-unknown-error-devtoolsactiveport-file-doesnt-exist-while-t but even with adding all those argumens to the chrome driver the issue still appeared. Maybe I passed them wrong?

Running the tests locally gives a third result. Sometimes they pass, sometimes the same tests fail. I could see that they made actual requests to the api server instead of using the mock. Maybe we have a timing iue in there.

Updating electron and spectron to their latest stable versions is passing all the tests. What is failing, is the npm list command afterwards.

CC @MichMich

[roramirez]

Hi @fewieden You're right. I've reference wrong the PR. This should be #1892. Sorry for that. This change is already merge in develop you could try against this branch. The test for weather is disabled in this branch.

The problem you are getting is because the sandbox mode is enable by default. More information about this is #1892

Setting the enviroment variable ELECTRON_DISABLE_SANDBOX=1 before run the test should be enough.

[fewieden]

@roramirez looks like I found the issue

[roramirez]

@fewieden Nice work!. Seems there this was little trickle but you found it.

@MichMich You should merge the #1927

[MichMich]

Merged!