[Bug]: 4.0.2 armhf and arm64 tarballs error out on @parcel/watcher

[aptalca]

Is there an existing issue for this?

• I have searched the existing issues

OS/Web Information

• Web Browser: Chrome
• Local OS: Win10
• Remote OS: Ubuntu
• Remote Architecture: aarch64
• code-server --version: 4.0.2

Steps to Reproduce

1. Download and extract linux arm64 tarball for 4.0.2
2. Fix the argon2 issue (specific to 4.0.2) by installing the module
3. Start code-server

Expected

No errors

Actual

Error about @parcel/watcher

Logs

[04:31:53] Extension host agent started.
[04:31:55] [192.168.13.58][2ae76c94][ManagementConnection] New connection established.
[04:31:57] [192.168.13.58][ae08c7ad][ExtensionHostConnection] New connection established.
[04:31:57] [192.168.13.58][ae08c7ad][ExtensionHostConnection] <1551> Launched Extension Host Process.
[IPC Library: File Watcher (parcel, node.js)] Loading "@parcel/watcher" failed
[IPC Library: File Watcher (parcel, node.js)] Error: No native build was found for platform=linux arch=arm64 runtime=node abi=83 uv=1 armv=8 libc=glibc node=14.17.4
    loaded from: /app/code-server/vendor/modules/code-oss-dev/node_modules/@parcel/watcher

    at Function.load.path (/app/code-server/vendor/modules/code-oss-dev/node_modules/node-gyp-build/index.js:59:9)
    at load (/app/code-server/vendor/modules/code-oss-dev/node_modules/node-gyp-build/index.js:21:30)
    at Object.<anonymous> (/app/code-server/vendor/modules/code-oss-dev/node_modules/@parcel/watcher/index.js:1:42)
    at Module._compile (internal/modules/cjs/loader.js:1072:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
    at Module.load (internal/modules/cjs/loader.js:937:32)
    at Function.Module._load (internal/modules/cjs/loader.js:778:12)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at r (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:5:101)
    at e.load (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:4:1723)
    at e.load (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:3:10695)
    at d (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:5:10300)
    at Object.errorback (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:5:10421)
    at e.triggerErrorback (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:3:11059)
    at /app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:3:10765
    at e.load (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:4:1740)
    at e.load (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:3:10695)
    at d (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:5:10300)
    at i._loadModule (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:5:10430)
    at i._resolve (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:6:452)
    at i.defineModule (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:5:6142)
    at v (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:6:1702)
    at g (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:4:2557)
    at Object.<anonymous> (/app/code-server/vendor/modules/code-oss-dev/out/vs/platform/files/node/watcher/parcel/watcherApp.js:12:22221)
    at Object.<anonymous> (/app/code-server/vendor/modules/code-oss-dev/out/vs/platform/files/node/watcher/parcel/watcherApp.js:12:30915)
    at e._createAndEvalScript (/app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:4:2601)
    at /app/code-server/vendor/modules/code-oss-dev/out/vs/loader.js:4:2240
    at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:73:3)
[IPC Library: File Watcher (parcel, node.js)] Here are the modules that depend on it:
[IPC Library: File Watcher (parcel, node.js)]% [ 'vs/platform/files/node/watcher/parcel/parcelWatcherService' ]

Screenshot/Video

No response

Does this issue happen in VS Code?

• I cannot reproduce this in VS Code.

Are you accessing code-server over HTTPS?

• I am using HTTPS.

Notes

No response

[aptalca]

By the way, I saw the issue reports about @node-rs/argon2 and bugs with npm and cross-compiling (likely this issue is related as the watcher module works just fine on amd64, as well as when code-server is installed via yarn on arm). Did you consider using docker/qemu to build the armhf and aarch64 tarballs so you don't have to worry about such arch related bugs?

[jsjoeio]

This is interesting...So I opened a bug report semi-related to this back in January: parcel-bundler/watcher#93

But we never arrived at a solution.

I think the build from source fix we added back when reverting the argon2 change should fix this 🤔 #4829

Did you consider using docker/qemu to build the armhf and aarch64 tarballs so you don't have to worry about such arch related bugs?

When we were debugging, we did try using qemu locally on my macOS to emulate an aarch64 env but @code-asher and I ran into a ton of issues.

I think we are qemu in our docker workflow here:

code-server/.github/workflows/docker.yaml

Lines 19 to 20 in 8135d2e

	- name: Set up QEMU
	uses: docker/setup-qemu-action@v1

But I guess that would only affect the Docker builds.

I wonder if we should be using it in our main workflow here:

code-server/.github/workflows/ci.yaml

Lines 164 to 168 in 8135d2e

	# The release package does not contain any native modules
	# and is neutral to architecture/os/libc version.
	- name: Create release package
	run: yarn release
	if: success()

I don't know if that would break other stuff though. We'd have to look into it more.

[aptalca]

Right, qemu has some bugs with runtime, but builds are much more reliable.

Assuming the release package generated via yarn release (npm-package) indeed does not contain any binaries or os/arch specific things as the note says, then I'm thinking this part could be using qemu instead of cross-compile:

code-server/.github/workflows/ci.yaml

Lines 285 to 296 in 8135d2e

	package-linux-cross:
	name: Linux cross-compile builds
	needs: build
	runs-on: ubuntu-18.04
	timeout-minutes: 15
	strategy:
	matrix:
	include:
	- prefix: aarch64-linux-gnu
	arch: arm64
	- prefix: arm-linux-gnueabihf
	arch: armv7l

and that would also solve the argon2 issue. The whole standalone package creation step (including for amd64) could be a 3 option matrix, one for each arch.

And just to give you a heads up (not sure if this will be an issue for the next release or not), we, at linuxserver.io, publish docker images for code-server on amd64, armhf and aarch64 and we also do weekly os package updates. During the last update, we noticed that the yarn build no longer produces a successful build. It completes successfully, and seems to start up fine, but trying to access via browser results in a 404. This is with 4.0.2.
Interestingly, previous week's build with 4.0.2 had no issues. The only changes between the two builds are a few os packages updated (don't seem to be that relevant) and some npm packages used in the build.
That is the main reason I started looking into utilizing the standalone packages you guys publish rather than building it with yarn. And that's when I noticed these issues I mentioned above.

[jsjoeio]

Right, qemu has some bugs with runtime, but builds are much more reliable.

TIL!

I'm thinking this part could be using qemu instead of cross-compile:

Ah, I think I understand. cc @code-asher @jawnsy since you two have more experience in this area. What do you think about this idea?

and that would also solve the argon2 issue.

That would be amazing!

During the last update, we noticed that the yarn build no longer produces a successful build. It completes successfully, and seems to start up fine, but trying to access via browser results in a 404. This is with 4.0.2.
Interestingly, previous week's build with 4.0.2 had no issues. The only changes between the two builds are a few os packages updated (don't seem to be that relevant) and some npm packages used in the build.

That is odd indeed...Might have to investigate with @code-asher to see if we can figure out what the issue is. We have received an usually high number of 404 reports with 4.0.2. I've seen it a couple times but usually can't reproduce.

Thanks for bringing that up! We'll have to do some thorough testing before the next release is out. We have been publishing the latest main changes under the beta flag on npm. You can test this out with yarn global add code-server@beta. We're hoping to publish beta to Docker in the future too so folks like yourself can test releases earlier.

[code-asher]

qemu seems like it would be stabler, definitely down to switch

[zarekvane]

I successfully installed code-server in debian that is installed in termux proot environment with the following command, but after I entered the login page and input my password, it showed me the 404 page.

yarn global add code-server

[jsjoeio]

I'm going to start working on this this week. I'll update this comment with my notes as I go.

TODOs

• See if I can reproduce this with 4.1.0

[aptalca]

@jsjoeio
Thanks for the reminder. I tested 4.1.0 tarballs on both arm32v7 and arm64v8, and no more errors with @parcel/watcher when accessing the gui.

This issue can be closed (unless you wish to continue the qemu build discussion here).

[jsjoeio]

@aptalca you rock! Thanks for testing it and reporting back.

This issue can be closed (unless you wish to continue the qemu build discussion here).

I'll open a separate issue to track that and link your comments. Thanks so much for your help!