Build fails when cross compiling v12.x host x86-64 to arm

[facorreia]

I am trying to cross compile node v12.16.1 for arm. It appears that part of the build process is to build the bytecode_builtins_list_generator executable that will be used in the host machine to create header files dynamically. However the build process is using target compiler instead of host compiler where the final binary can not to be executed in the host machine.

bytecode_builtins_list_generator: node/out/Release/bytecode_builtins_list_generator: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=27e85c210648cdd73a0e84e1c367c3915259ff85, not stripped

compiler: arm-linux-gnueabi-g++ version 5.5.0

Host Platform: Linux 4.13.0-43-generic #48~16.04.1-Ubuntu SMP Thu May 17 12:56:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Target Platform: -march=armv7-a -mcpu=cortex-a9 -mtune=cortex-a9 -mfpu=neon

Node version: v12.16.1

Configure: ./configure --prefix=/work/deploy/node --shared --cross-compiling --shared-openssl --with-arm-fpu=neon --with-arm-float-abi=softfp --shared-openssl-includes=
/work/sdk/openssl/1.0.2s/include/ --shared-openssl-libname=crypto,ssl --shared-openssl-libpath=/work/sdk/openssl/1.0.2s/lib/
--dest-cpu=arm --dest-os=linux --without-snapshot --without-node-snapshot --without-report --without-dtrace --without-etw --without-npm --no-browser-globals --wi
thout-inspector --without-intl

config.gypi
########################################################
Do not edit. Generated by the configure script.
{ 'target_defaults': { 'cflags': [],
'default_configuration': 'Release',
'defines': [],
'include_dirs': [ '/work/sdk/openssl/1.0.2s/include/'],
'libraries': [ '-L/work/sdk/include/oss/openssl/1.0.2s/lib/',
'-lcrypto',
'-lssl']},
'variables': { 'arm_float_abi': 'softfp',
'arm_fpu': 'neon',
'arm_thumb': 0,
'arm_version': '7',
'asan': 0,
'build_v8_with_gn': 'false',
'coverage': 'false',
'debug_nghttp2': 'false',
'enable_lto': 'false',
'enable_pgo_generate': 'false',
'enable_pgo_use': 'false',
'force_dynamic_crt': 1,
'host_arch': 'x64',
'icu_small': 'false',
'is_debug': 0,
'llvm_version': '0.0',
'napi_build_version': '5',
'node_byteorder': 'little',
'node_debug_lib': 'false',
'node_enable_d8': 'false',
'node_install_npm': 'false',
'node_module_version': 72,
'node_no_browser_globals': 'true',
'node_prefix': '/work/deploy/node',
'node_release_urlbase': '',
'node_report': 'false',
'node_shared': 'true',
'node_shared_cares': 'false',
'node_shared_http_parser': 'false',
'node_shared_libuv': 'false',
'node_shared_nghttp2': 'false',
'node_shared_openssl': 'true',
'node_shared_zlib': 'false',
'node_tag': '',
'node_target_type': 'shared_library',
'node_use_bundled_v8': 'true',
'node_use_dtrace': 'false',
'node_use_etw': 'false',
'node_use_large_pages': 'false',
'node_use_large_pages_script_lld': 'false',
'node_use_node_code_cache': 'false',
'node_use_node_snapshot': 'false',
'node_use_openssl': 'true',
'node_use_v8_platform': 'true',
'node_with_ltcg': 'false',
'node_without_node_options': 'false',
'openssl_fips': '',
'openssl_is_fips': 'false',
'shlib_suffix': 'so.72',
'target_arch': 'arm',
'v8_enable_gdbjit': 0,
'v8_enable_i18n_support': 0,
'v8_enable_inspector': 0,
'v8_no_strict_aliasing': 1,
'v8_optimized_debug': 1,
'v8_promise_internal_field_count': 1,
'v8_random_seed': 0,
'v8_trace_maps': 0,
'v8_use_siphash': 1,
'v8_use_snapshot': 0,
'want_separate_host_toolset': 0}}

error message:

#####################################################
LD_LIBRARY_PATH=/node/out/Release/lib.host:/node/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /node/out/Release/obj/gen/generate-bytecode-output-root/builtins-generated; python ../../deps/v8/tools/run.py "/node/out/Release/bytecode_builtins_list_generator" "/node/out/Release/obj/gen/generate-bytecode-output-root/builtins-generated/bytecodes-builtins-list.h"
Traceback (most recent call last):
File "../../deps/v8/tools/run.py", line 12, in <module>
sys.exit(subprocess.call(sys.argv[1:]))
File "/usr/lib/python2.7/subprocess.py", line 523, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error
tools/v8_gypfiles/generate_bytecode_builtins_list.target.mk:13: recipe for target '/node/out/Release/obj/gen/generate-bytecode-output-root/builtins-generated/bytecodes-builtins-list.h' failed
make[2]: *** [/node/out/Release/obj/gen/generate-bytecode-output-root/builtins-generated/bytecodes-builtins-list.h] Error 1
make[2]: *** Waiting for unfinished jobs....
../deps/v8/src/torque/implementation-visitor.cc: In member function 'v8::internal::torque::VisitResult v8::internal::torque::ImplementationVisitor::InlineMacro(v8::internal::torque::Macro*, v8::base::Optional<v8::internal::torque::LocationReference>, const std::vector<v8::internal::torque::VisitResult>&, std::vector<v8::internal::torque::Block*>)':
../deps/v8/src/torque/implementation-visitor.cc:258:32: warning: 'macro_end' may be used uninitialized in this function [-Wmaybe-uninitialized]
assembler().Bind(macro_end);
^
../deps/v8/src/torque/torque-parser.cc: In function 'v8::base::Optional<v8::internal::torque::ParseResult> v8::internal::torque::{anonymous}::MakeTypeswitchStatement(v8::internal::torque::ParseResultIterator*)':
../deps/v8/src/torque/torque-parser.cc:1026:30: warning: 'accumulated_types' may be used uninitialized in this function [-Wmaybe-uninitialized]
: cases[i].type;
^
Makefile:101: recipe for target 'node' failed
make[1]: *** [node] Error 2

[bnoordhuis]

Does it work when you install qemu-user?

[bnoordhuis]

Having said that, our cross-compile job doesn't use qemu-user and works okay - bytecode_builtins_list_generator and mksnapshot are compiled for the host arch, not the target arch.

[facorreia]

Hi bnordhuis, I tested the qemu-user-static and it still fail.

I installed it with the command bellow
sudo apt-get install gcc-arm-linux-gnueabihf libc6-dev-armhf-cross qemu-user-static

and set those to crosscompile script
HOST_PREFIX_ARM="arm-linux-gnueabihf-"

export AR_host="${HOST_PREFIX_ARM}ar"
export CC_host="${HOST_PREFIX_ARM}gcc -L/usr/lib32/"
export CXX_host="${HOST_PREFIX_ARM}g++"
export LINK_host="${HOST_PREFIX_ARM}g++"
export CPP_host="${HOST_PREFIX_ARM}gcc -E"
export LD_host="${HOST_PREFIX_ARM}ld"
export AS_host="${HOST_PREFIX_ARM}as"
export CCLD_host="${HOST_PREFIX_ARM}gcc"
export NM_host="${HOST_PREFIX_ARM}nm"
export STRIP_host="${HOST_PREFIX_ARM}strip"
export OBJCOPY_host="${HOST_PREFIX_ARM}objcopy"
export RANLIB_host="${HOST_PREFIX_ARM}ranlib"
export F77_host="${HOST_PREFIX_ARM}g77

The bytecode_builtins_list_generator executable is still build by target toochain and now the build fail with the error message bellow:
/node/out/Release/bytecode_builtins_list_generator: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory

In additional to, I can cross compile node v8.17.0 successfully in the previous environment without any modification.

[lissyx]

I was needing to git bisect for running on RPi / Raspbian buster, from a Debian Sid. This is what I used:

#!/bin/bash

ARCH=arm

CC_VER="6.5.0"
        DEST_CPU="$ARCH"
        SUFFIX="$ARCH-rpi-linux-gnueabihf"
        TOOLCHAIN_NAME="$SUFFIX"

export TOOLCHAIN=$HOME/[...]/rpi-newer-crosstools-eb68350c5c8ec1663b7fe52c742ac4271e3217c5/x64-gcc-6.5.0/arm-rpi-linux-gnueabihf

export PATH=$TOOLCHAIN/bin:$PATH
export AR=$TOOLCHAIN/bin/$SUFFIX-ar
export CC="$TOOLCHAIN/bin/$SUFFIX-gcc -march=armv7-a"
export CXX="$TOOLCHAIN/bin/$SUFFIX-g++ -march=armv7-a"
export LINK=$TOOLCHAIN/bin/$SUFFIX-gcc
export LDFLAGS=" -Wl,-lstdc++ -Wl,-lm"

export AR_host=ar
export CC_host=gcc-8
export CXX_host=g++-8
export LINK_host="gcc-8 -Wl,-lstdc++ -Wl,-lm"

export CXXFLAGS_host="-isystem /usr/include/x86_64-linux-gnu/"
#export LDFLAGS_host="-lstdc++ -lm"

GYP_DEFINES="target_arch=$ARCH"
GYP_DEFINES+=" arm_version=7"
GYP_DEFINES+=" v8_target_arch=$ARCH"
GYP_DEFINES+=" host_os=linux"
export GYP_DEFINES

./configure \
        --dest-cpu=$DEST_CPU \
        --dest-os=linux \
        --with-arm-float-abi=hard \
        --with-arm-fpu=neon \
        --without-snapshot

[allenzt]

same issue, I tried the above suggestions and still encountered the problem. Anyone can give me more suggestions, thanks in advance.

[SwatiSonam]

I am facing the same issue while building nodejs version 13.1.0, can anyone help me here.

[greggwon]

It seems like there really should be a script that one can run to do this configure and build correctly. I know there are several options around the target processor. But a make target set or something really could be crafted around those details so that the right things could be done, and this could be tested on each release for the set of combinations known at each release point. Adding more combinations over time would then be a natural process.

[gengjiawen]

We are improving this on #35252. Please stay tuned.

[gengjiawen]

A sample build: https://github.com/gengjiawen/node-github-workflow/runs/1970162330?check_suite_focus=true I have tested on v15, not sure v12, let's take a look.

[Ubbo-Sathla]

Change the host system ，use Ubuntu to try ？

[JMLX42]

The problem is 'want_separate_host_toolset': 0 in config.gypi.

The value comes from configure.py (line 1059):

https://github.com/nodejs/node/blob/v12.22.12/configure.py#L1059

  want_snapshots = not options.without_snapshot
  o['variables']['want_separate_host_toolset'] = int(
      cross_compiling and want_snapshots)

Setting 'want_separate_host_toolset': 1 after calling ./configure does the trick.

Removing --without-snapshot from ./configure has the intended effect: 'want_separate_host_toolset': 1. And the build works. But I do not know what this change implies.

By reading this comment, I was under the impression that --with-snapshot was actually the problem. But I do not understand how their can be both a --with-snapshot and a --without-snapshot flag.

Update: some related comment that comes to the same conclusion imyller/meta-nodejs#31 (comment)

[JMLX42]

Removing --without-snapshot from ./configure has the intended effect: 'want_separate_host_toolset': 1. And the build works. But I do not know what this change implies.

I confirm it works with the Android NDK r25b and ANDROID_API=24:

apt-get update && apt-get install -y python gcc g++ gcc-multilib g++-multilib

export AR="ar"
export AR_host="ar"
export AR_target="llvm-ar"
export CC="gcc -m32"
export CC_host="gcc -m32"
export CC_target="armv7a-linux-androideabi24-clang"
export CXX="g++ -m32"
export CXX_host="g++ -m32"
export CXX_target="armv7a-linux-androideabi24-clang++"
export GYP_DEFINES="target_arch=arm v8_target_arch=arm android_target_arch=arm host_os=linux OS=android"

./configure --dest-cpu=arm --dest-os=android --openssl-no-asm --cross-compiling --shared --without-intl --without-npm
make -j$(nproc)