Add .cirrus.yml for FreeBSD (amd64, i386) CI/CD #2592

am11 · 2019-02-05T21:48:20Z

CirrusCI has added FreeBSD support in December 2018. Currently
only the amd64 images are provisioned.

cbsd is used to create an i386 jail environment. Kudos to Oleg
Ginzburg (@olevole), wouldn't have been possible without his
help (cbsd/cbsd#367).

For each pull request, it triggers a job consists of 'node.js version x
architecture' matrix.
When a tag is pushed, it creates a release idempotently, and publishes
artifacts to the release.

The versions of node.js and npm packages are selected from package
management. This is because node.js official distribution does not
(yet) offer FreeBSD binaries and nvm spends too much time compiling
node.js and npm from source.

pkg does not support installation of old version of packages with
pkg-install command, so the pkg-add approach has been elected to
install older node.js. For that matter, the exact version of node.js
and npm packages are resolved using pattern matching.

For each pull request, it triggers a job consists of 'node.js version x architecture' matrix. When a tag is pushed, it creates a release idempotently, and publishes artifacts to the release. The versions of node.js and npm packages are selected from package management. This is because node.js official distribution does not (yet) offer FreeBSD binaries and `nvm` spends too much time compiling node.js and npm from source. `pkg` does not support installation of old version of packages with `pkg-install` command, so the `pkg-add` approach has been elected to install older node.js. For that matter, the exact version of node.js and npm packages are resolved using pattern matching. `cbsd` is used to create an i386 jail environment. Co-authored-by: Oleg Ginzburg <olevole@olevole.ru>

In FreeBSD VM hosted on Google GCE, with `pkg search` command, it fails to retrieve the latest list even after `pkg update -f` with `latest` repository selected (checked with `pkg -vv search`). There seems to be some caching mechanism involved, which I couldn't find a way to avoid.

am11 · 2019-02-05T21:52:14Z

A successful job run for normal commit looks like this: https://cirrus-ci.com/build/5765193800876032, for tag https://cirrus-ci.com/build/5684006109577216 and release artifacts get uploaded to GH releases in node-sass' desired format: https://github.com/am11/node-sass/releases/tag/testbsd-x5.

am11 · 2019-02-05T21:56:20Z

.cirrus.yml

+        execPrefix: cbsd jexec jname=j11i386
+
+  env:
+    GH_API_TOKEN: ENCRYPTED[5e482f417304528148bb96eca8d030eacd6ab9972d482485fc4d42907283b995f658b351e0676e9493a37d815398f541]


@xzyfer, it seems like we would need to login to CirrucCI with organization admin's account and encrypt the personal access token using the following view:

This is only used for publishing artifacts. 8-)

nschonni · 2019-02-05T21:59:52Z

Nice!
Tagging @saper since he has been doing the FreeBSD builds so far

am11 · 2019-02-05T22:13:57Z

Thanks @nschonni! @saper, highly appreciate if you could review. :)

Was waiting for https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235412 review to go through, so we could use cbsd 12.0.4 from ports and clean up some code in the jail creation shell script. Currently, it is cloning the tag ref from cbsd repo for i386 jail creation. The longest job is less than eight minutes; create i386 jail, install tools, build node-sass and upload artifacts. amd64 jobs are about four minutes on average.

nschonni · 2019-11-02T01:22:54Z

@saper since you've been looking at the FreeBSD build again, maybe you can review and push any changes needed here

ltning · 2020-03-30T22:38:05Z

If this is required for FreeBSD builds to be automated .. poke ?

nschonni · 2020-04-21T23:03:17Z

@saper is this worth getting sorted out for the Node 14 builds?

saper

to bring that to april 2020 we need to update this a bit

saper · 2020-04-26T22:00:44Z

appveyor.yml

@@ -166,7 +166,7 @@
      - nodejs_version: 11
        GYP_MSVS_VERSION: 2015
        APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
-        
+
  install:


spurious change

saper · 2020-04-26T22:00:58Z

package.json

-    "mocha": "^3.1.2",
-    "mocha-lcov-reporter": "^1.2.0",
+    "mocha": "^3.5.3",
+    "mocha-lcov-reporter": "^1.3.0",
    "object-merge": "^2.5.1",


unrelated change?

saper · 2020-04-26T22:37:14Z

.cirrus.yml

+        npmTxzKey: npm-node8
+      - node_js: 6
+        nodeTxzKey: node6
+        npmTxzKey: npm-node6


no longer there

saper · 2020-04-26T22:37:25Z

.cirrus.yml

+
+  env:
+    matrix:
+      - node_js: 11


need to add node 13, 12

saper · 2020-04-26T22:37:44Z

.cirrus.yml

+  env:
+    matrix:
+      - node_js: 11
+        nodeTxzKey: node-11


ugly and unreadable name

saper · 2020-04-26T22:39:34Z

scripts/configure_freebsd_ci_jail.sh

+skelDirectory=$2
+cbsd_workdir=/usr/jails
+jail_arch="i386"
+jail_ver="11.2"


we are now at a 11.3

saper · 2020-04-26T22:40:21Z

.cirrus.yml

@@ -0,0 +1,69 @@
+freebsd_instance:
+  image: freebsd-11-2-release-amd64


shall we use

image_family: freebsd-11-3-snap

instead?

saper · 2020-04-26T23:47:31Z

.cirrus.yml

+          fullyQualifiedName="$(pwd)/$parent/${name}_binding.node";
+          mv "$file" "$parent/${name}_binding.node";
+          echo -e "New filename\072 $fullyQualifiedName";
+          ./scripts/upload-github-release-asset.sh github_api_token=$GH_API_TOKEN owner=sass repo=node-sass tag=$CIRRUS_TAG filename=$fullyQualifiedName;


How do we know it works? Should the pipeline fail if that fails? I used your key (could not be decrypted) and the publish script finished successfully.

saper · 2020-04-27T00:00:29Z

scripts/upload-github-release-asset.sh

+fi
+
+# Validate token.
+curl -o /dev/null -sH "$AUTH" $GH_REPO || { echo "Error: Invalid repo, token or network issue!";  exit 1; }


Assuming empty token:

curl -o /dev/null -sH "Authorization: token " https://api.github.com/repos/sass/node-sass && echo OK

This works, so no error reported

saper · 2020-04-27T00:01:59Z

.cirrus.yml

+      if test "$abi" = "freebsd:11:x86:32"; then
+        ./scripts/configure_freebsd_ci_jail.sh $jailName $CIRRUS_WORKING_DIR;
+        $execPrefix sed -i -- '' 's/quarterly/latest/g' /etc/pkg/FreeBSD.conf;
+        $execPrefix pkg update -f;


Why are we doing this only for 32 bit? Maybe this is the problem with having not update repository?

saper · 2020-04-27T00:18:04Z

.cirrus.yml

+      if test "$abi" = "freebsd:11:x86:32"; then
+        changeDir="cd /etc/skel &&"
+      fi
+      echo "$changeDir npm install --unsafe-perm" | $execPrefix /bin/sh


we should skip download here to ensure the binary is always built

nschonni · 2020-09-27T00:19:18Z

It might be nice to get this running for v5. Right now that's just looking at a simpler matrix of "Currently Supported Node", so this matrix could be smaller and might be easier to deal with for Cirrus. The GitHub App is setup for Cirrus now, but I'm not sure if it's going to work for it coming from a fork or not

am11 · 2020-09-27T01:41:06Z

In another project, I was working with GitHub Actions and cross compilation for FreeBSD and illumos. Once the cross compilation script is setup properly (basically ~10 lines of shell script), and a docker image is built and published with that cross compilation toolchain, that container can run on any CIaaS which supports kubernetes or more generally, CaaS.
Thanks to hub(1), uploading artifacts from is simple enough, e.g. this is currently working out quite nicely for my own stuff: https://github.com/am11/runtime/blob/4a95ab23d3a09262955f252e7869633989601d76/.github/workflows/main.yml#L29.

In short, we do not need FreeBSD VM to build FreeBSD binaries in the modern age of mankind. 😎

nschonni · 2020-09-27T02:37:52Z

@am11 interesting, I didn't think FreeBSD supported Docker since it used jails instead of LXC

am11 · 2020-09-27T02:59:37Z

Yup, native Docker port is still under development for FreeBSD. However, the cross compilation toolchain can be built for number of Unix-like OS, and it can execute on Linux. For instance, rust and dotnet runtime build their official packages with Linux.x64-based (glibc) containers that have cross compiling toolchains setup for alpine (muscle libc), arm64 etc. they also have FreeBSD CI legs using the same cross-compilation infra. I also used the same mechanics to configure a container that has cross compilation toolchain setup for illumous (derivative of SunOS), then used that container in GitHub Actions CI for dotnet runtime build (yaml link above). In order to run tests on the target platform, however, we would need a VM.

am11 and others added 3 commits February 5, 2019 07:36

Switch to Sass org account

eb5801f

am11 commented Feb 5, 2019

View reviewed changes

nschonni requested a review from saper February 5, 2019 21:59

fkorotkov mentioned this pull request Feb 5, 2019

FreeBSD all architectures via jail and example cirruslabs/cirrus-ci-docs#125

Open

am11 added 3 commits February 6, 2019 00:42

Use last version of mocha that supports node 0.x

fb33306

Use cbsd v12.0.4 from ports

4f6b9d8

Remove obsolete comment

0aa9518

network-shark mentioned this pull request Mar 7, 2019

Add Cirrus CI configuration example ansible/molecule#1807

Closed

nschonni mentioned this pull request Jan 31, 2020

FreeBSD assets missing for v4.13.1 #2837

Closed

saper requested changes Apr 26, 2020

View reviewed changes

saper reviewed Apr 26, 2020

View reviewed changes

saper reviewed Apr 27, 2020

View reviewed changes

saper mentioned this pull request Apr 30, 2020

Add .cirrus.yml for FreeBSD (amd64, i386) CI/CD #2911

Draft

saper added Discussion - Proposal npm build OS - FreeBSD labels Apr 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add .cirrus.yml for FreeBSD (amd64, i386) CI/CD #2592

Add .cirrus.yml for FreeBSD (amd64, i386) CI/CD #2592

am11 commented Feb 5, 2019

am11 commented Feb 5, 2019 •

edited

am11 Feb 5, 2019

nschonni commented Feb 5, 2019

am11 commented Feb 5, 2019

nschonni commented Nov 2, 2019

ltning commented Mar 30, 2020

nschonni commented Apr 21, 2020

saper left a comment

saper Apr 26, 2020

saper Apr 26, 2020

saper Apr 26, 2020

saper Apr 26, 2020

saper Apr 26, 2020

saper Apr 26, 2020

saper Apr 26, 2020

saper Apr 26, 2020

saper Apr 27, 2020 •

edited

saper Apr 27, 2020

saper Apr 27, 2020

nschonni commented Sep 27, 2020

am11 commented Sep 27, 2020

nschonni commented Sep 27, 2020

am11 commented Sep 27, 2020

		@@ -0,0 +1,69 @@
		freebsd_instance:
		image: freebsd-11-2-release-amd64

Add .cirrus.yml for FreeBSD (amd64, i386) CI/CD #2592

Are you sure you want to change the base?

Add .cirrus.yml for FreeBSD (amd64, i386) CI/CD #2592

Conversation

am11 commented Feb 5, 2019

am11 commented Feb 5, 2019 • edited

Choose a reason for hiding this comment

nschonni commented Feb 5, 2019

am11 commented Feb 5, 2019

nschonni commented Nov 2, 2019

ltning commented Mar 30, 2020

nschonni commented Apr 21, 2020

saper left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saper Apr 27, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nschonni commented Sep 27, 2020

am11 commented Sep 27, 2020

nschonni commented Sep 27, 2020

am11 commented Sep 27, 2020

am11 commented Feb 5, 2019 •

edited

saper Apr 27, 2020 •

edited