Add Python 3 to more build machines

[cclauss]

Continue the process started in #1644 of putting both Python 2 and Python 3 on various build machines so that we can perform tests and experimental builds on the Python 3.
nodejs/node#25759 (comment)

/R = @rvagg

[cclauss]

On those build machines where Python 3 is already installed, we need to do:
python3 -m pip install --upgrade pipenv # See test failures on nodejs/node#25878

[refack]

Need to add to ansible recipes (https://github.com/nodejs/build/search?p=3&q=python&type=&utf8=%E2%9C%93)
Might be a good chance to refactor common prerequisites.

[cclauss]

Can we please get this resolves and closed? 303 daze left...

[rvagg]

some help getting ansible working properly with python3 would be great, I can't run it on my mac anymore for machines that need ccache and have to resort to linux where it uses python2. The line causing problems is https://github.com/nodejs/build/blob/master/ansible/roles/baselayout/tasks/ccache.yml#L13 and there's a similar one in ansible/roles/jenkins-worker/tasks/partials/docker-host.yml.
Something about jinja2 processing of "{{ ccache_html_content.stdout | regex_findall('ccache-[0-9]+.[0-9]+(?:.[0-9]+)*.tar.gz') | map('regex_replace', 'ccache-') | map('regex_replace', '.tar.gz') | list | latest_version }}" the chaining of the maps seems to be the problem and I don't have the error at hand. If needed I could probably generate the error again.

[refack]

@rvagg does #1399 help?

(BTW this has the new "npm" like pipenv work flow, git fetch, pipenv install, pipenv shell)

[cclauss]

Should we close this one or is this still an issue?

[refack]

We still need Python3 on more machines.

[cclauss]

Can we track progress in a table similar to the one below?

Operating System	Architectures	Versions	Python 2	Python 3
GNU/Linux	x64	kernel >= 3.10, glibc >= 2.17	Tests pass	Tests pass
GNU/Linux	x64	kernel >= 3.10, musl >= 1.1.19	Tests pass	Tests fail
GNU/Linux	x86	kernel >= 3.10, glibc >= 2.17	Tests pass	Not installed
GNU/Linux	arm64	kernel >= 4.5, glibc >= 2.17	Tests pass	Not installed
GNU/Linux	armv7	kernel >= 4.14, glibc >= 2.24	Tests pass	Not installed
GNU/Linux	armv6	kernel >= 4.14, glibc >= 2.24	Tests pass	Not installed
GNU/Linux	ppc64le >=power8	kernel >= 3.13.0, glibc >= 2.19	Tests pass	Not installed
GNU/Linux	s390x	kernel >= 3.10.0, glibc >= 2.17	Tests pass	Not installed
Windows	x64, x86 (WoW64)	>= Windows 7/2008 R2/2012 R2	Tests pass	Not installed
Windows	x86 (native)	>= Windows 7/2008 R2/2012 R2	Tests pass	Not installed
Windows	arm64	>= Windows 10	Tests pass	Not installed
macOS	x64	>= 10.12	Tests pass	Not installed
SmartOS	x64	>= 18	Tests pass	Not installed
AIX	ppc64be >=power7	>= 7.1 TL05	Tests pass	Not installed
FreeBSD	x64	>= 11	Tests pass	Not installed

[sam-github]

Is #1674 (comment) needed on the AIX machines? If so, after updating them all to have python3 available, #1775 (comment), I could do it.

[sam-github]

• test-osuosl-ubuntu1404-ppc64_be-*: /usr/bin/python3 is 3.4.3
• test-osuosl-ubuntu1404-ppc64_le-*: /usr/bin/python3 is 3.4.3
• test-osuosl-aix61-ppc64_be-*: /opt/bin/python3 is 3.7.3
• test-osuosl-centos7-ppc64_le-*: /usr/bin/python3 is 3.6.8
• test-linuxonecc-rhel72-s390x-*: needs python3, WIP, investigating
• test-marist-zos13-s390x-*: needs python3, WIP, investigating

[sam-github]

I ran #1800 on test-osuosl-centos7-ppc64_le-1, if it all still looks good tomorrow, I'll update the others.

[centos@test-osuosl-centos7-ppc64--le-1 ~]$ ls -c1 /usr/bin/python*
/usr/bin/python3.6
/usr/bin/python3.6m
/usr/bin/python3
/usr/bin/python36
/usr/bin/python2.7
/usr/bin/python
/usr/bin/python2
[centos@test-osuosl-centos7-ppc64--le-1 ~]$ python3 --version
Python 3.6.8

[sam-github]

I've spent more than an hour and a half looking for s390x RPMs for a python3, no luck yet.

No s390x in:

• https://cbs.centos.org/repos/
• https://dl.fedoraproject.org/pub/epel/
• https://ius.io

I'm finding some suggestions that if I use subscription-manager, I might be able to get it. I'm not exactly sure what we would have legal entitlement to (or if it would even work), and we are not currently registered for any subscriptions.

• https://access.redhat.com/documentation/en-us/red_hat_software_collections/3/html/3.1_release_notes/chap-RHSCL#tabl-RHSCL-Components

I'm not very familiar with RH-derived distros, I might be missing something obvious.

@nodejs/python or @nodejs/platform-s390, suggestions?

[mhdawson]

@sam-github I suggest you talk to @miladfarca to see if he can get some ideas of who we might ask about python3 for the linuxOne machines.

[sam-github]

Updated second centos-ppc64_le:

% ssh test-osuosl-centos7-ppc64_le-2 python3 --version                                                                              
Python 3.6.8

[cclauss]

What is the Python 2 version on those machines?

We want python2 >= 2.7.9

[sam-github]

@cclauss see #1674 (comment), I'm updating it

[sam-github]

Sorry, misread your comment, you want to know about python2, not python3. @cclauss I'll go look.

[sam-github]

@cclauss

% ssh test-osuosl-ubuntu1404-ppc64_be-1 python2 --version
Python 2.7.6
% ssh test-osuosl-ubuntu1404-ppc64_le-1 python2 --version
Python 2.7.6
% ssh test-osuosl-aix61-ppc64_be-1 python2 --version
Python 2.7.11
% ssh test-osuosl-centos7-ppc64_le-1 python2 --version
Python 2.7.5

[sam-github]

@cclauss I don't understand why we need to update python2 to a supported version if we have python3. If it was easy, I'd just do it, but its not, so I am pushing back a bit.

Why should we update Python2 if a supported version of Python3 exists? Especially given python2 is EOL on Jan 1, 2020?

[cclauss]

Let's focus first and foremost of getting a Python >= 3.5 on these machines (and in the $PATH). If it was up to me, I would just take out legacy Python completely and step into the future. However, Node.js is not yet compatible with Python 3. This exercise is to set up the tests on Python 3 but it is not yet clear if those tests will pass and build a workable Node.js.

Therefor, Python 2 will be a part of Node.js beyond the 1/1/20 date so given that we will be depending on an unsupported codebase, we would like to uplevel the Python 2 to be >= 2.7.9 to encourage our users to avoid the most egregious attack vectors. If it is too much work to fix legacy Python in the near term, lets focus on upgrading Python as you suggest.

Thanks much for all your hard work.

[sam-github]

see: #1807

[sam-github]

Folks here, and @nodejs/python , we need more clarity on the expectation for python3. At least, I do.

Python3 has to be >= 3.5.0.

When system packages are available, I guess it will be in the system path, as python3.

Is this the intended setup, in other words, is it a requirement that python3 be in the default PATH?

And is this a requirement: #1674 (comment) ?

On test-osuosl-centos7-ppc64--le-1:

$ ./python3 -m pip install --upgrade pipenv
/opt/python-3.7/bin/python3: No module named pip

I don't know much about pip, and nothing about pipenv, so it would help if we had a relatively clear statement of what was required.

Possibly we want node itself to build with python3, if that is even possible. Is there some way I can test that manually? If I have a succinct recipe for what I should do manually on a machine once python3 is installed on it to verify the python3 environment fulfills expectations, that would be helpful.

EDIT: also see #1807 (comment)

[cclauss]

pip is the package installer for Python. It is the essential bootstrap for installing packages. Without pip you can not install pyenv or any other Python packages so we need pip. You can try python3 -m ensurepip to install it but the preference would be an install which already includes pip.

python3 --version
python3 -m pip --version

[thefourtheye]

Is this the intended setup, in other words, is it a requirement that python3 be in the default PATH?

If Python 3 is not there in the default PATH, we may have to update our scripts to detect Python 3 as well, I am afraid. Installing it at /usr/local should not create any problems, right?

Possibly we want node itself to build with python3, if that is even possible. Is there some way I can test that manually?

The ultimate goal of this activity is to build and test Node.js with both Python 3 and Python 2.

If I have a succinct recipe for what I should do manually on a machine once python3 is installed on it to verify the python3 environment fulfills expectations, that would be helpful.

As far as I am concerned, we just need Python 3 in the default PATH. pip is optional/good-to-have in my opinion, as we do not install many additional packages to test (I could be wrong here, I haven't checked all the scripts we have). @cclauss Do you think pip is necessary for us to proceed at this point?

[cclauss]

A Python install without pip is a painful thing so we should avoid it if we can.

https://github.com/nodejs/node/blob/master/Makefile#L1316 and below use pip to install flake8 but linting on all platforms is not required.

[sam-github]

For anyone finding their python3 is lacking pip, or ctypes, libffi and zlib (the -dev versions of these) might be missing.

[cclauss]

See the note above (#1674 (comment)) about ensurepip.

[sam-github]

@cclauss Perhaps I misunderstood that comment, don't you suggest not using ensurepip? Are you suggesting we should use it? Or suggesting that its an alternative to installing python3's dev dependencies before configuring and building python3?

[cclauss]

Are we done on this one? Can we start testing Python 3.6 on one of the Jenkins test machines? Python 3.7 is pending nodejs/node#29326 but Python 3.6 should be fair game to start testing.

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.