BUILD_FROM_SRC_DOC_AND_DOCKERIZED_STANDALONE_APM

[Jean-Baptiste-Lasselle]

Summary

I recently checked that the Build From Source Documentation of APM, is severely lacking infos. For instance :

• I found no where any explanation on how to tie APM up with NodeJS runtime,
• And no where explanations about the Python Environment that should be set to a 2.7 release version (latest update)

All in alll, you'll find all you need to manage this issue at : https://github.com/Jean-Baptiste-Lasselle/apm-dokerized-build-from-source

Motivation

Well i wanted to make hot configuration possible, meaning up to hot package reloading

I open this issue to quickly share the work I already achieved, an automated APM Build From Source.The recipe also shows how to standalone run APM, see : https://github.com/Jean-Baptiste-Lasselle/apm-dokerized-build-from-source

Describe alternatives you've considered

Eclipse Che

Additional context

...

[Jean-Baptiste-Lasselle]

I heard that Python 2.7 has reached EOL, so what about Atom IDE ?

[Arcanemagus]

I found no where any explanation on how to tie APM up with NodeJS runtime

apm is the Atom package manager, it's not intended to be something that is done as a standalone package. When bundled with Atom it has an appropriate Node.js runtime packaged with it.

And no where explanations about the Python Environment that should be set to a 2.7 release version (latest update)

You haven't provided any information on what errors you were seeing that led you to this, but I'm guessing that you were running into the ever present node-gyp issue with Python 3, which has nothing to do with Atom (or apm) specifically, and is a Node.js ecosystem wide issue.

I add that Python 2.7 has reached EOL, so what about Atom IDE ?

Atom itself has no dependency on Python, other than the previously mentioned node-gyp issue.

---

Well i wanted to make hot configuration possible, meaning up to hot package reloading

apm has almost nothing to do with this (other than installing the updated files), that's an issue entirely within Atom. Atom actually used to have live reloading for packages, but the edge cases surrounding it proved too unreliable to guarantee that things were reloaded correctly, so Atom now requires a restart in order to bring in the new package version.

Does that answer your questions?

[Jean-Baptiste-Lasselle]

Hi @Arcanemagus ,

Thank you so much for this quick and extensive answer !

Actually, I did understand that APM was not meant to be run on standalone, and I wrote my issue more to share an opportunity rather than pointing out an issue.

I also point the Python 2.7 EOL issue, because it is important, and so interesting, very much so indeed :

Atom itself has no dependency on Python, other than the previously mentioned node-gyp issue.

Well it's word (to be or not to be, that is not the question) :

• I agree, when you run atom or apm standalone (because it's now possible, cf. my repo), you don't bring (hence execute) any python2 runtime aboard.
• Yes, I, at some point, had an node-gyp issue, and resolved it. So thank you for pointing that out, and giving more well known details about it.
• Yet, whether node-gyp, or whoever might be responsible for it, I care first about your awesome work, Atom, which I fell in love with. Caring that much means I had to spawn up a repo and gives a proof that Atom 's Pipeline has Python 2 as a dependency.
• So yes, Atom doesn't run it, but soon, no pipeline will run Python2, and my biggest clients' Security Engineer will not care all I brought in thanks to Atom, they'll just cut it off their Information System. I don't wanna go back to Eclipse :) .
• Oh , I have to mention one important thing : every time I bring in Atom into my clients company, I bring it with 100% sovereignty . Meaning I import all 254 repos in https://github.com/atom/, inside the company, and add my devops work, to make it an autonomous Software factory. They can build from source on the inside, and security engineers can't block my work anymore, because it's integrated with an ISO 27 000 security management process (security reviews, OCI image vulnerability scanners, and so on (and other reasons, but not first, why i dockerized every single package)).
• Meaning also I have actually designed https://github.com/Jean-Baptiste-Lasselle/apm-dokerized-build-from-source months ago for a customer, and shared that just days ago, a few commits to modifying README.md and give first explanations, to share.
• As last point, I think that there might be some very unexpected interesting technical issue hidden under there, no one is responsible for, just something new.
• And there's another thing :
  • Atom is and was made by Github.
  • In a way (very important to discuss, If I explain a customer what sovereignty means, and they pay me a lot to do so), Atom belongs to Github.
  • Github was acquired by Microsoft. Hence Atom.
  • But Microsoft has invested huge amont of time and money into Visual Studio.
  • See where I'm going here? If not clear, I invite the whole community to do what needs to be done so that Microsoft knows that as soon as show one sign of "trying to get rid of atom", or restrict any of the fundamental rights of the Libre Manifesto we inherited from RMS (Stallman, GNU concepts etc), they can be sure we'll spawn it back up in weeks, with stronger support.

My customers care a lot about your work, Atom and they don't care one second about any Microsoft product.

You wouldn't imagine the crazy things I ended up doing, after success tearing off APM that way, it was a decisive moment in my project with my customer, who was literally mind blown.

To finish with that, what I'll do I'll share as soon as possible with the community a work I 'm currently finishing, I'll say when done. Will take weeks, I am now sick at home, and will have to work a lot for a customer when I'm back on foot, but I seriously plan to release before Christmas.

And my first goal will be to solve this nasty python 2.7 issue :

• (my typical customer) "Yeah, okay, but all in all, what the hell is it, being devops?"
• (me) "Being devops means that if it takesmeto change Linux so you can use the webapp YOU choose, then I'll do so. YOU decide what you want, not Linus Torvalds ( with all undued respect ;) ) (hum, I never had the opportunity to say that about Windows, it died before I had the opportunity) "

Thank you so much for your additional, helpful informations @Arcanemagus

And I love Atom, honestly, you guys changed my life with my keyboard !

Ttys

[Jean-Baptiste-Lasselle]

@Arcanemagus As sson as I have respawned My different tests, i'll give you details about :

• the error indeed mentioning node-gyp
• and I can tell you already that I run APM in a node Docker image. An official node docker image;
• Though in a Node Docker image, APM was still crying out for the node executable, and I of course learned about how APM and atom use their own Node / NPM.
• And instead of setting NODE_PATH, which I wanted to avoid caus I wanna leave Node Enviornment manageent all left to NVM, no tricking with Node, I did this :
  • Ok, APM is looking for node executable at path wedontcarewhattheexactpathis
  • And on the other hand, inside the node official docker image, the node executable is at path dontcaremuchmoreeither
  • So guess what I did ?

ln -s $EMBEDDED_NODE_HOME/node /pipeline/ops/bin/node
# where EMBEDDED_NODE_HOME is the folder, inside
# the NodeJS official Docker image, where the [node] executable is located on Filesystem

• And there I ended up in a new world of marvels...
• https://github.com/Jean-Baptiste-Lasselle/apm-dokerized-build-from-source/blob/392d38860f25a1461f2d2c491b976019cf9b9ac9/docker/autre3/Dockerfile#L118

@Arcanemagus Just to mention, Here is output of my automated build, with prooof I run APM standalone with 0 errors, just a "warning", that the Atom executable is not found :

Step 36/37 : RUN ./bin/apm --version
 ---> Running in 416e4f6dbd7f
apm  2.4.3
npm  6.2.0
node 8.16.2 x64
atom unknown
python 2.7.16
git 2.20.1
Removing intermediate container 416e4f6dbd7f
 ---> 427e06a64adb
Step 37/37 : CMD ["/bin/sh"]
 ---> Running in e8def61732ee
Removing intermediate container e8def61732ee
 ---> 8b2acbdb287d
[Warning] One or more build-args [AAAA] were not consumed
Successfully built 8b2acbdb287d
Successfully tagged pegasus-devops.io/apm_build_from_src:0.0.1
WARNING: Image for service hot_ide_workspace was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating apm_build_from_src ... done
WARNING: The BBBB variable is not set. Defaulting to a blank string.
Attaching to apm_build_from_src

• And whene I run inside container after startup :

jbl@poste-devops-typique:~$ docker exec -it apm_build_from_src sh
/pipeline/ops # ls -allh /pipeline/ops/
total 380
drwxr-xr-x    1 root     root        4.0K Oct 20 16:02 .
drwxr-xr-x    1 root     root        4.0K Oct 20 16:00 ..
-rw-r--r--    1 root     root          15 Oct 20 16:02 .embedded.node.home.pgs
drwxr-xr-x    8 root     root        4.0K Oct 20 16:00 .git
-rw-r--r--    1 root     root          20 Oct 20 16:00 .gitattributes
drwxr-xr-x    2 root     root        4.0K Oct 20 16:00 .github
-rw-r--r--    1 root     root          65 Oct 20 16:00 .gitignore
-rw-r--r--    1 root     root         109 Oct 20 16:00 .npmignore
-rw-r--r--    1 root     root         372 Oct 20 16:00 .pairs
-rw-r--r--    1 root     root         335 Oct 20 16:00 .travis.yml
-rw-r--r--    1 root     root           8 Oct 20 16:00 BUNDLED_NODE_VERSION
-rw-r--r--    1 root     root          92 Oct 20 16:00 CONTRIBUTING.md
-rw-r--r--    1 root     root        1.1K Oct 20 16:00 Gruntfile.coffee
-rw-r--r--    1 root     root        1.6K Oct 20 16:00 ISSUE_TEMPLATE.md
-rw-r--r--    1 root     root        1.0K Oct 20 16:00 LICENSE.md
-rw-r--r--    1 root     root        1.5K Oct 20 16:00 PULL_REQUEST_TEMPLATE.md
-rw-r--r--    1 root     root        3.5K Oct 20 16:00 README.md
-rw-r--r--    1 root     root         262 Oct 20 16:00 appveyor.yml
drwxr-xr-x    1 root     root        4.0K Oct 20 16:02 bin
-rw-r--r--    1 root     root         607 Oct 20 16:00 coffeelint.json
-rw-r--r--    1 root     root       37.9K Oct 20 16:00 deprecated-packages.json
drwxr-xr-x    1 root     root        4.0K Oct 20 16:02 lib
drwxr-xr-x    3 root     root        4.0K Oct 20 16:00 native-module
drwxr-xr-x    1 root     root        4.0K Oct 20 16:02 node_modules
-rw-r--r--    1 root     root      225.6K Oct 20 16:00 package-lock.json
-rw-r--r--    1 root     root        1.5K Oct 20 16:00 package.json
drwxr-xr-x    2 root     root        4.0K Oct 20 16:00 script
drwxr-xr-x    3 root     root        4.0K Oct 20 16:00 spec
drwxr-xr-x    1 root     root        4.0K Oct 20 16:00 src
drwxr-xr-x    7 root     root        4.0K Oct 20 16:00 templates
/pipeline/ops # ls -allh /pipeline/ops/bin/
total 28
drwxr-xr-x    1 root     root        4.0K Oct 20 16:02 .
drwxr-xr-x    1 root     root        4.0K Oct 20 16:02 ..
-rwxr-xr-x    1 root     root        1.2K Oct 20 16:00 apm
-rw-r--r--    1 root     root         702 Oct 20 16:00 apm.cmd
lrwxrwxrwx    1 root     root          19 Oct 20 16:02 node -> /usr/local/bin/node
-rwxr-xr-x    1 root     root         697 Oct 20 16:00 npm
-rw-r--r--    1 root     root         175 Oct 20 16:00 npm.cmd
-rwxr-xr-x    1 root     root         887 Oct 20 16:00 python-interceptor.sh
/pipeline/ops # apm --version
sh: apm: not found
/pipeline/ops # ls -allh /pipeline/ops/bin/apm
-rwxr-xr-x    1 root     root        1.2K Oct 20 16:00 /pipeline/ops/bin/apm
/pipeline/ops # /pipeline/ops/bin/apm --version
apm  2.4.3
npm  6.2.0
node 8.16.2 x64
atom unknown
python 2.7.16
git 2.20.1
/pipeline/ops # /pipeline/ops/bin/apm install hydrogen
/pipeline/ops/lib/command.js:127
          throw new Error('Could not determine Electron version');
          ^

Error: Could not determine Electron version
    at getResourcePath (/pipeline/ops/lib/command.js:127:17)
    at config.getResourcePath (/pipeline/ops/lib/command.js:141:18)
    at /pipeline/ops/lib/apm.js:81:20
    at _combinedTickCallback (internal/process/next_tick.js:132:7)
    at process._tickCallback (internal/process/next_tick.js:181:9)
/pipeline/ops # 

• So my next step : installing Electron, and then I expect apm install hydrogen to cry out for Atom . If not, I will run the cycle again, until there's nothing left at all missing, but Atom. Then I'll do a trick.

[Jean-Baptiste-Lasselle]

..

• Documentation for using apm programmatically with an Electron app #475
• Some installed packages could not be loaded because they contain native modules that were compiled for an earlier version of Atom. atom#19780

[Arcanemagus]

Since it sounds like that answered your question, and there isn't an issue with apm as it is used in Atom here, I'm going to close this issue.

Thanks for reaching out!

---

Because we treat our issues list as the Atom team's backlog, we close issues that are questions since they don't represent a task needing to be completed. For most questions about Atom there are a lot of options, check out:

• User Documentation — The Atom Flight Manual
• Developer Documentation — Atom API Documentation
• FAQ — The Atom FAQ on Discuss
• Message Board — Discuss, the official Atom and Electron message board
• Chat — Join the Atom Slack team

On Discuss and in the Atom Slack team, there are a bunch of helpful community members that should be willing to point you in the right direction.

[Jean-Baptiste-Lasselle]

https://github.com/Jean-Baptiste-Lasselle/apm-dokerized-build-from-source/raw/master/documentation/issues/images/ATOM_TEAM_SLACK_DOWN_Firefox_Screenshot_2019-10-22T10-05-14.135Z.png

Since it sounds like that answered your question, and there isn't an issue with apm as it is used in Atom here, I'm going to close this issue.

Thanks for reaching out!

Because we treat our issues list as the Atom team's backlog, we close issues that are questions since they don't represent a task needing to be completed. For most questions about Atom there are a lot of options, check out:

* User Documentation — [The Atom Flight Manual](http://flight-manual.atom.io)

* Developer Documentation — [Atom API Documentation](https://atom.io/docs/api/latest)

* FAQ — [The Atom FAQ on Discuss](https://discuss.atom.io/c/faq)

* Message Board — [Discuss, the official Atom and Electron message board](https://discuss.atom.io)

* Chat — [Join the Atom Slack team](http://atom-slack.herokuapp.com/)

On Discuss and in the Atom Slack team, there are a bunch of helpful community members that should be willing to point you in the right direction.

@Arcanemagus Thank you for your answer, which brings unexpected support to the point of view I here shared :

I propose my help about the Team 's infrastructure. I'd be very happy to.
I'll even pay for 1year of the chatops needed underlying servers, on personal money, if it can help.

Btw, what is this :

?

Sum up of my access try outs

• could no log into any of the 2 slacks
• could register & log into Atom Discussion

Eventually

I searched python 2.7 in discussions in Message Board , and... :

• I find this :
  
• And that :
  

[Jean-Baptiste-Lasselle]

• https://github.com/atom/apm/blob/master/src/generator/safemake.py
• Why no support Python3 nodejs/node-gyp#1687 (comment)
• https://github.com/cmake-js/cmake-js (so what java and node mean by "building native", is just that they call native language build tools...and process management libraries, like Process.Start(...) ... I c).

So now, my first serious idea

Let's change ElectronJs, so that it has anew feature :

• it would be possible to configura ElectronJS, so that it can use either node-gyp, or cmakejs, just by changing the value of one parameter in ElectronJS build configuration file.
• and extend it to as many node native code build library,
• So that ElectronJS has the abilty to switch from one native code build library to another :
  • while node-gyp works on python3 migration
  • ElectronJS recommends (and defaults to..) cmakejs, in the stead of node-gyp.

So, all in all, where I totally understand why you pointed out it's node-gyp's Team responisbility, I say they're currently working on it, and the problem is that ElectronJS has there fore to have the capability to switch between node native code build libraries