build: test Python 3.7 on Travis CI #29451

cclauss · 2019-09-05T13:59:30Z

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

sam-github · 2019-09-05T17:36:59Z

Requires #25878 or py3 will fail at configure, so I cherry-picked it in.

Took me a bit to get pyenv working locally on Ubuntu, but while waiting for Travis, I'm in process of reproducing the buildsteps travis should do.

Note, for the pyenv statements to work, anyone else doing this (and me) will need to do export TRAVIS=yes to cause pyenv's python to be chosen over a specific version.

Once I've done this, current state is:

Compile V8

./configure: OK
make -j2 -C out V=1 v8: failing due to Failed to parse config file: Type names and field names cannot be a keyword: 'async', fix in tools: python3 compat for inspector code generator #29340, but after cherry-picking, I get the same issue. I guess the fix is still WIP by @bnoordhuis and @targos , so I'm moving on.

Compile Node.js

make fails with same keyword problem as above
moving on

Test JS Suites

obviously impossible until node.js Compile works

Test C++ Suites

make cctest also has the async keyword problem, so this step is impossible as well

At this point, I think futher progress is blocked until #29340 completes and lands.

cclauss · 2019-09-06T04:33:47Z

As noted in #29340, both the tools and deps version of that file need to be modified.

sam-github · 2019-09-06T04:45:32Z

Thanks, I'll take another shot at this tomorrow.

cclauss · 2019-09-06T10:42:12Z

I added the async change to the deps version of code_generator.py

sam-github · 2019-09-06T19:34:21Z

With the fixes merged in, I can now build and test locally without errors, which is unfortunate, because now I can't troubleshoot locally.

Travis error is:

Building addon in /home/travis/build/nodejs/node/test/addons/01_function_arguments
465Building addon in /home/travis/build/nodejs/node/test/addons/02_callbacks
466/home/travis/build/nodejs/node/tools/build-addons.js:58
467main(process.argv[3]).catch((err) => setImmediate(() => { throw err; }));
468                                                          ^
469
470Error: Command failed: /home/travis/build/nodejs/node/out/Release/node /home/travis/build/nodejs/node/deps/npm/node_modules/node-gyp/bin/node-gyp.js rebuild --directory=/home/travis/build/nodejs/node/test/addons/01_function_arguments
471
472    at ChildProcess.exithandler (child_process.js:295:12)
473    at ChildProcess.emit (events.js:209:13)
474    at maybeClose (internal/child_process.js:1028:16)
475    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5) {
476  killed: false,
477  code: 1,
478  signal: null,
479  cmd: '/home/travis/build/nodejs/node/out/Release/node /home/travis/build/nodejs/node/deps/npm/node_modules/node-gyp/bin/node-gyp.js rebuild --directory=/home/travis/build/nodejs/node/test/addons/01_function_arguments',
480  stdout: '',
481  stderr: ''
482}
483Makefile:412: recipe for target 'test/addons/.buildstamp' failed

I'm trying to directly run the failing node-gyp command, and interestingly, I see that it uses python2... So, I removed /usr/bin/python2, and see something more interesting:

% PYTHON=/usr/bin/python3 ./out/Release/node deps/npm/node_modules/node-gyp/bin/node-gyp.js rebuild --directory=/home/travis/build/no
dejs/node/test/addons/01_function_arguments --nodedir=/usr/local/stow/lts/include/
gyp info it worked if it ends with ok                                     
gyp info using node-gyp@5.0.3                             
gyp info using node@13.0.0-pre | linux | x64              
gyp WARN chdir /home/travis/build/nodejs/node/test/addons/01_function_arguments is not a directory
gyp ERR! find Python                                                    
gyp ERR! find Python Python is not set from command line or npm configuration
gyp ERR! find Python checking Python explicitly set from environment variable PYTHON
gyp ERR! find Python - process.env.PYTHON is "/usr/bin/python3"                
gyp ERR! find Python - executable path is "/usr/bin/python3"
gyp ERR! find Python - version is "3.7.3"
gyp ERR! find Python - version is 3.7.3 - should be >=2.6.0 <3.0.0 
gyp ERR! find Python - THIS VERSION OF PYTHON IS NOT SUPPORTED                                                                                                               
gyp ERR! find Python checking if "python" can be used                                                                                                                        
gyp ERR! find Python - executable path is "/home/sam/.pyenv/versions/3.7.3/bin/python"                                                                                       
gyp ERR! find Python - version is "3.7.3"                                                                                                                                    
gyp ERR! find Python - version is 3.7.3 - should be >=2.6.0 <3.0.0
gyp ERR! find Python - THIS VERSION OF PYTHON IS NOT SUPPORTED          
gyp ERR! find Python checking if "python2" can be used    
gyp ERR! find Python - "python2" is not in PATH or produced an error

This sparked a theory that Travis doesn't have python2 installed when pyenv global 3.7.1 is used, which is why its failing, and by destroying my local python2, I can now see the same error Travis gets:

% make -j1 V=1 test/addons/.buildstamp
Building addon in /home/sam/w/core/lts/test/addons/01_function_arguments
Building addon in /home/sam/w/core/lts/test/addons/02_callbacks
Building addon in /home/sam/w/core/lts/test/addons/03_object_factory
Building addon in /home/sam/w/core/lts/test/addons/04_function_factory
/home/sam/w/core/lts/tools/build-addons.js:58
main(process.argv[3]).catch((err) => setImmediate(() => { throw err; }));
                                                          ^

Error: Command failed: /home/sam/w/core/lts/out/Release/node /home/sam/w/core/lts/deps/npm/node_modules/node-gyp/bin/node-gyp.js rebuild --directory=/home/sam/w/core/lts/test/addons/01_function_arguments

    at ChildProcess.exithandler (child_process.js:295:12)
    at ChildProcess.emit (events.js:209:13)
    at maybeClose (internal/child_process.js:1028:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5) {
  killed: false,
  code: 1,
  signal: null,
  cmd: '/home/sam/w/core/lts/out/Release/node /home/sam/w/core/lts/deps/npm/node_modules/node-gyp/bin/node-gyp.js rebuild --directory=/home/sam/w/core/lts/test/addons/01_function_arguments',
  stdout: '',
  stderr: ''
}
make: *** [Makefile:412: test/addons/.buildstamp] Error 1

cclauss · 2019-09-07T05:55:53Z

cp tools/gyp/pylib/gyp/input.py deps/npm/node_modules/node-gyp/gyp/pylib/gyp/
cp tools/gyp/pylib/gyp/generator/make.py deps/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/

cclauss · 2019-09-07T06:54:05Z

https://travis-ci.com/nodejs/node/jobs/232041903#L1008

python tools/test.py -j 2 -p dots --report --mode=release --flaky-tests=dontcare addons js-native-api node-api
tools/test.py:32: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
Total: 115 tests
 *    0 tests will be skipped
 *  115 tests are expected to pass
 *    0 tests are expected to fail that we won't fix
 *    0 tests are expected to fail that we should fix
Running 115 tests
...................C..............................
..................................................
...............
=== release test ===
Path: addons/dlopen-ping-pong/test
--- stderr ---
node: ../binding.cc:29: void {anonymous}::LoadLibrary(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `handle != nullptr' failed.
Command: out/Release/node /home/travis/build/nodejs/node/test/addons/dlopen-ping-pong/test.js
--- CRASHED (Signal: 6) ---
===
=== 1 tests failed
=== 1 tests CRASHED
===
The command "python tools/test.py -j 2 -p dots --report --mode=release --flaky-tests=dontcare addons js-native-api node-api" exited with 1.

sam-github · 2019-09-07T17:38:17Z

That's interesting, I'll take a look Monday unless @bnoordhuis gets to it first.

Trott · 2019-09-09T17:59:04Z

I guess the deps changes should be a separate commit from the other ones so that it can be re-cherry-picked onto things if the deps get updated but still don't have these changes?

Trott · 2019-09-09T17:59:52Z

.travis.yml

@@ -6,9 +6,15 @@ x-ccache-setup-steps: &ccache-setup-steps

 os: linux
 language: cpp
+# Currently this file can only support one PYTHON_VERSION.
+# To experiment with Python 3, comment out Python 2.7 and uncomment one of the Python 3 versions.


IIUC, with this change, Python 2.7 is already commented out, so this comment needs to be updated?

Trott · 2019-09-09T18:01:40Z

@cclauss Should the work-in-progress label be removed at this point? Or is it still work-in-progress?

sam-github · 2019-09-09T19:29:24Z

Still WIP, it isn't green, and has local copies of changes that are pending in other PRs, or other repos (v8, node-gyp, npm). Its very useful to show what problems will remain even after the other in-progress work completes.

cclauss · 2019-09-09T19:31:20Z

This is still a WIP. It is designed to pass the Travis CI tests on Python 3 but it still fails.

cclauss · 2019-09-09T20:29:47Z

Does anyone know how to resolve the last failing test?

sam-github · 2019-09-09T21:37:34Z

@bnoordhuis any ideas?

bnoordhuis · 2019-09-10T08:39:43Z

I looked into it yesterday but it wasn't immediately obvious to me what went wrong. binding.node tries to dlopen ping.so but somehow it fails.

Time for some printf-style debugging, I suppose. Christian, can you try this?

diff --git a/test/addons/dlopen-ping-pong/binding.cc b/test/addons/dlopen-ping-pong/binding.cc
index 6a6c297b52..02ade757c0 100644
--- a/test/addons/dlopen-ping-pong/binding.cc
+++ b/test/addons/dlopen-ping-pong/binding.cc
@@ -26,6 +26,7 @@ static ping ping_func;
 void LoadLibrary(const FunctionCallbackInfo<Value>& args) {
   const String::Utf8Value filename(args.GetIsolate(), args[0]);
   void* handle = dlopen(*filename, RTLD_LAZY);
+  if (handle == nullptr) fprintf(stderr, "dlopen(%s): %s\n", *filename, dlerror());
   assert(handle != nullptr);
   ping_func = reinterpret_cast<ping>(dlsym(handle, "dlopen_ping"));
   assert(ping_func != nullptr);

(I tried to push it to your branch but you have collaborator edits disabled, it seems?)

cclauss · 2019-09-10T10:51:01Z

dlopen(/home/travis/build/nodejs/node/test/addons/dlopen-ping-pong/build/Release/ping.so): /home/travis/build/nodejs/node/test/addons/dlopen-ping-pong/build/Release/ping.so: cannot open shared object file: No such file or directory

sam-github · 2019-10-16T01:16:20Z

Rebased onto master now that #29885 has landed.

.travis.yml

richardlau

Copying my review comments from #29985 (duplicate?).

.travis.yml

sam-github · 2019-10-16T04:00:02Z

I'll clean up the comments and up the python version, but would like to see travis pass. It seems really, really slow... is ccache not shared? Something else? Maybe ccache is reset when .travis.yml changes?

Anyhow, I'll give it overnight to run some more before touching the branch again.

sam-github · 2019-10-16T04:04:36Z

@cclauss I don't think #29985 was needed, this just needed rebasing and pushing. Also, to pass. I can hope 🤞

richardlau · 2019-10-16T04:07:19Z

I'll clean up the comments and up the python version, but would like to see travis pass. It seems really, really slow... is ccache not shared? Something else? Maybe ccache is reset when .travis.yml changes?

https://github.com/nodejs/node/compare/253137b5502a6732eac9f0f899c6577c04338023..2ff431b7a08fab7938a73fb6f10ed76f151035aa#diff-42f385dff7890b3c213d9699bcc350cc suggests v8.h was updated in the force push which is going to force a recompilation of most of V8.

nodejs-github-bot · 2019-10-17T18:23:25Z

CI: https://ci.nodejs.org/job/node-test-pull-request/26085/

sam-github · 2019-10-17T18:23:38Z

@nodejs/python @nodejs/node-gyp PTAL

sam-github · 2019-10-17T20:02:17Z

smartos out of memory:

tools/doc/generate.js --node-version=v13.0.0 --apilinks=out/doc/apilinks.json doc/api/url.md --output-directory=out/doc/api\""; exit 1; fi;
11:33:46 
11:33:46 
11:33:46 #
11:33:46 # Fatal error in , line 0
11:33:46 # Fatal process out of memory: Zone
11:33:46 #
11:33:46 #
11:33:46 #
11:33:46 #FailureMessage Object: fffffbffede7a500

Windows is an instance of #30011

Resuming the build.

nodejs-github-bot · 2019-10-17T20:02:50Z

CI: https://ci.nodejs.org/job/node-test-pull-request/26087/

sam-github · 2019-10-17T20:04:21Z

@nodejs/platform-smartos FYI - #29451 (comment) I don't know how frequent that is, and hope it will be ephemral, but just to let you know.

Request Python 3 with pyenv and ensure that python3 is used by Makefile to run Python scripts. PR-URL: #29451 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rod Vagg <rod@vagg.org>

sam-github · 2019-10-18T18:41:40Z

Landed in d594a9a 🎉

Request Python 3 with pyenv and ensure that python3 is used by Makefile to run Python scripts. PR-URL: #29451 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rod Vagg <rod@vagg.org>

nodejs-github-bot added the build Issues and PRs related to build files or the CI. label Sep 5, 2019

cclauss mentioned this pull request Sep 5, 2019

tools/test.py is not yet Python 3 compatible #29246

Closed

cclauss added the python PRs and issues that require attention from people who are familiar with Python. label Sep 5, 2019

sam-github added the wip Issues and PRs that are still a work in progress. label Sep 5, 2019

sam-github force-pushed the Travis-test-Py36-or-Py37 branch from cfe37bd to 0d60361 Compare September 5, 2019 16:57

cclauss force-pushed the Travis-test-Py36-or-Py37 branch from 567ca38 to b5fd477 Compare September 6, 2019 10:37

cclauss mentioned this pull request Sep 6, 2019

build: add comment to .travis.yml on how to test Py3 #29473

Closed

4 tasks

cclauss changed the title ~~build: explain how to test Py3.6 & Py3.7 on Travis~~ build: test Python 3.7 on Travis CI Sep 6, 2019

cclauss force-pushed the Travis-test-Py36-or-Py37 branch from 7d70614 to 014f468 Compare September 7, 2019 05:55

Trott reviewed Sep 9, 2019

View reviewed changes

sam-github force-pushed the Travis-test-Py36-or-Py37 branch from 2ba9e62 to b535e9c Compare September 9, 2019 21:34

cclauss force-pushed the Travis-test-Py36-or-Py37 branch from b535e9c to 23ce029 Compare September 9, 2019 21:35

cclauss force-pushed the Travis-test-Py36-or-Py37 branch from 23ce029 to 5a6e21d Compare September 9, 2019 21:48

cclauss force-pushed the Travis-test-Py36-or-Py37 branch from 5a6e21d to 9ab7107 Compare September 10, 2019 10:28

sam-github force-pushed the Travis-test-Py36-or-Py37 branch from 253137b to 2ff431b Compare October 16, 2019 01:13

rvagg reviewed Oct 16, 2019

View reviewed changes

.travis.yml Show resolved Hide resolved

richardlau reviewed Oct 16, 2019

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

.travis.yml Outdated Show resolved Hide resolved

fixup: remove EXPERIMENTAL_NODE_GYP_PYTHON3="yes"

77fb266

sam-github removed the wip Issues and PRs that are still a work in progress. label Oct 17, 2019

richardlau approved these changes Oct 17, 2019

View reviewed changes

sam-github approved these changes Oct 17, 2019

View reviewed changes

sam-github mentioned this pull request Oct 17, 2019

flaky test: test-http-dump-req-when-res-ends on windows #30011

Closed

rvagg approved these changes Oct 17, 2019

View reviewed changes

sam-github closed this Oct 18, 2019

MylesBorins mentioned this pull request Oct 23, 2019

v13.0.1 proposal #30081

Merged

targos mentioned this pull request Nov 10, 2019

v12.13.1 release proposal #30352

Merged

targos added the dont-land-on-v12.x label Nov 11, 2019

devsnek deleted the Travis-test-Py36-or-Py37 branch May 21, 2020 17:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: test Python 3.7 on Travis CI #29451

build: test Python 3.7 on Travis CI #29451

cclauss commented Sep 5, 2019 •

edited by sam-github

sam-github commented Sep 5, 2019

cclauss commented Sep 6, 2019

sam-github commented Sep 6, 2019

cclauss commented Sep 6, 2019

sam-github commented Sep 6, 2019

cclauss commented Sep 7, 2019 •

edited

cclauss commented Sep 7, 2019

sam-github commented Sep 7, 2019

Trott commented Sep 9, 2019

Trott Sep 9, 2019

Trott commented Sep 9, 2019

sam-github commented Sep 9, 2019

cclauss commented Sep 9, 2019

cclauss commented Sep 9, 2019

sam-github commented Sep 9, 2019

bnoordhuis commented Sep 10, 2019

cclauss commented Sep 10, 2019

sam-github commented Oct 16, 2019

richardlau left a comment

sam-github commented Oct 16, 2019

sam-github commented Oct 16, 2019

richardlau commented Oct 16, 2019

nodejs-github-bot commented Oct 17, 2019

sam-github commented Oct 17, 2019

sam-github commented Oct 17, 2019

nodejs-github-bot commented Oct 17, 2019

sam-github commented Oct 17, 2019

sam-github commented Oct 18, 2019

build: test Python 3.7 on Travis CI #29451

build: test Python 3.7 on Travis CI #29451

Conversation

cclauss commented Sep 5, 2019 • edited by sam-github

Checklist

sam-github commented Sep 5, 2019

Compile V8

Compile Node.js

Test JS Suites

Test C++ Suites

cclauss commented Sep 6, 2019

sam-github commented Sep 6, 2019

cclauss commented Sep 6, 2019

sam-github commented Sep 6, 2019

cclauss commented Sep 7, 2019 • edited

cclauss commented Sep 7, 2019

sam-github commented Sep 7, 2019

Trott commented Sep 9, 2019

Trott Sep 9, 2019

Choose a reason for hiding this comment

Trott commented Sep 9, 2019

sam-github commented Sep 9, 2019

cclauss commented Sep 9, 2019

cclauss commented Sep 9, 2019

sam-github commented Sep 9, 2019

bnoordhuis commented Sep 10, 2019

cclauss commented Sep 10, 2019

sam-github commented Oct 16, 2019

richardlau left a comment

Choose a reason for hiding this comment

sam-github commented Oct 16, 2019

sam-github commented Oct 16, 2019

richardlau commented Oct 16, 2019

nodejs-github-bot commented Oct 17, 2019

sam-github commented Oct 17, 2019

sam-github commented Oct 17, 2019

nodejs-github-bot commented Oct 17, 2019

sam-github commented Oct 17, 2019

sam-github commented Oct 18, 2019

cclauss commented Sep 5, 2019 •

edited by sam-github

cclauss commented Sep 7, 2019 •

edited