Mavsdk app fails to build on GLIBC 2.31

[AlexKlimaj]

When building https://github.com/ARK-Electronics/logloader on Jetson Linux 35.5.0, I get the following error.

Mavsdk installed from the most recent release.

jetson@jetson:~/code/logloader$ make
-- The CXX compiler identification is GNU 9.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenSSL: /usr/local/lib/libcrypto.so (Required is at least version "3.0.2")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/jetson/code/logloader/build
make[1]: Entering directory '/home/jetson/code/logloader/build'
make[2]: Entering directory '/home/jetson/code/logloader/build'
make[3]: Entering directory '/home/jetson/code/logloader/build'
Scanning dependencies of target logloader
make[3]: Leaving directory '/home/jetson/code/logloader/build'
make[3]: Entering directory '/home/jetson/code/logloader/build'
[ 66%] Building CXX object CMakeFiles/logloader.dir/src/LogLoader.cpp.o
[ 66%] Building CXX object CMakeFiles/logloader.dir/src/main.cpp.o
[100%] Linking CXX executable logloader
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `fstat64@GLIBC_2.33'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_wait_until_steady(unsigned int*, unsigned int, bool, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >)@GLIBCXX_3.4.29'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `std::__throw_bad_array_new_length()@GLIBCXX_3.4.29'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `stat64@GLIBC_2.33'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `pthread_cond_clockwait@GLIBC_2.34'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `pthread_create@GLIBC_2.34'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `pthread_mutex_trylock@GLIBC_2.34'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `std::__exception_ptr::exception_ptr::_M_release()@CXXABI_1.3.13'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `__libc_single_threaded@GLIBC_2.32'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `pthread_join@GLIBC_2.34'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `pthread_detach@GLIBC_2.34'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `std::__exception_ptr::exception_ptr::_M_addref()@CXXABI_1.3.13'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.8.0: undefined reference to `pthread_once@GLIBC_2.34'
collect2: error: ld returned 1 exit status
make[3]: *** [CMakeFiles/logloader.dir/build.make:102: logloader] Error 1
make[3]: Leaving directory '/home/jetson/code/logloader/build'
make[2]: *** [CMakeFiles/Makefile2:76: CMakeFiles/logloader.dir/all] Error 2
make[2]: Leaving directory '/home/jetson/code/logloader/build'
make[1]: *** [Makefile:130: all] Error 2
make[1]: Leaving directory '/home/jetson/code/logloader/build'
make: *** [Makefile:3: all] Error 2
jetson@jetson:~/code/logloader$

jetson@jetson:~/code/logloader$ ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.14) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
jetson@jetson:~/code/logloader$ ldd

[julianoes]

Yes, we switched to newer dockcross images with GCC 11 because of nasty std::filesystem linking problems.
a5d2d98

What's Jetson Linux 35.5.0 based on? Any chance you could move to something newer?

[AlexKlimaj]

@julianoes It is based on Ubuntu 20.04. Jetpack 5 is going to be around for a long time still. Jetpack 6 based on Ubuntu 22.04 is currently in developer preview.

jetson@jetson:~/code/logloader$ uname -a
Linux jetson 5.10.192-tegra #1 SMP PREEMPT Mon Feb 19 20:19:53 PST 2024 aarch64 aarch64 aarch64 GNU/Linux
jetson@jetson:~/code/logloader$

[AlexKlimaj]

Just confirmed that 2.4.0 works.

jetson@jetson:~/Downloads$ sudo dpkg -i libmavsdk-dev_2.4.0_debian12_arm64.deb
[sudo] password for jetson:
dpkg: warning: downgrading libmavsdk-dev from 2.8.0 to 2.4.0
(Reading database ... 165682 files and directories currently installed.)
Preparing to unpack libmavsdk-dev_2.4.0_debian12_arm64.deb ...
Unpacking libmavsdk-dev (2.4.0) over (2.8.0) ...
Setting up libmavsdk-dev (2.4.0) ...
Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
jetson@jetson:~/Downloads$
jetson@jetson:~/Downloads$
jetson@jetson:~/Downloads$ cd
jetson@jetson:~$ cd code/logloader/
jetson@jetson:~/code/logloader$ make clean
All build artifacts removed
jetson@jetson:~/code/logloader$ make
-- The CXX compiler identification is GNU 9.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenSSL: /usr/local/lib/libcrypto.so (Required is at least version "3.0.2")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/jetson/code/logloader/build
make[1]: Entering directory '/home/jetson/code/logloader/build'
make[2]: Entering directory '/home/jetson/code/logloader/build'
make[3]: Entering directory '/home/jetson/code/logloader/build'
Scanning dependencies of target logloader
make[3]: Leaving directory '/home/jetson/code/logloader/build'
make[3]: Entering directory '/home/jetson/code/logloader/build'
[ 66%] Building CXX object CMakeFiles/logloader.dir/src/LogLoader.cpp.o
[ 66%] Building CXX object CMakeFiles/logloader.dir/src/main.cpp.o
[100%] Linking CXX executable logloader
make[3]: Leaving directory '/home/jetson/code/logloader/build'
[100%] Built target logloader
make[2]: Leaving directory '/home/jetson/code/logloader/build'
make[1]: Leaving directory '/home/jetson/code/logloader/build'
   text    data     bss     dec     hex filename
1154111   12416   10744 1177271  11f6b7 build/logloader
jetson@jetson:~/code/logloader$

[julianoes]

Well, if you can help me figure out the std::filesystem path issue, I'm happy to go back to dockcross-lts. I couldn't.

[dakejahl]

Well, if you can help me figure out the std::filesystem path issue, I'm happy to go back to dockcross-lts. I couldn't.

@julianoes is there an issue somewhere that captures the details? I can take a look.

[dakejahl]

Hmm I just built main using linux-arm64-lts without any issues.

[julianoes]

Building is fine. What happens is that you build on top of libmavsdk on arm64 and use std::filesystem::path and it crashes when you run it.

[dakejahl]

Hmm it's not crashing on me, this is on the Jetson Orin Nano Ubuntu 20.04 and Jetpack 5. Logloader uses fs::path without crashing. I'm guessing the crashing is happening on systems with newer glibc? Perhaps backwards compatibility got broken. On which hardware do you test arm64?

jetson@jetson:~/code/logloader$ ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.9) 2.31
jetson@jetson:~/code/logloader$ uname -ar
Linux jetson 5.10.104-tegra #1 SMP PREEMPT Sun Mar 19 07:55:28 PDT 2023 aarch64 aarch64 aarch64 GNU/Linux

[julianoes]

Did you try against an older mavsdk library?

[dakejahl]

I'm building/installing MAVSDK main from source and linking into logloader (which uses fs::path) and I don't have any crashing issues. I'm suggesting that if mavsdk is built with the lts dockcross image that it uses the older (<2.34) glibc and when linking into an app built with the newer glibc, the crashes happen. So I would say we probably need separate ubuntu 20.04/22.04 debs similar to amd64 to handle the glibc/gcc discrepancies

[dakejahl]

If there's any way you know how to reproduce let me know. We just built logloader on the pi4 and linked against libmavsdk 2.4.0 and there is no crashing issue.

pi@ARK-Pi6X:~ $ uname -a
Linux ARK-Pi6X 6.6.20+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.20-1+rpt1 (2024-03-07) aarch64 GNU/Linux
pi@ARK-Pi6X:~ $ ldd --version
ldd (Debian GLIBC 2.36-9+rpt2+deb12u4) 2.36

[julianoes]

I'm suggesting that if mavsdk is built with the lts dockcross image that it uses the older (<2.34) glibc and when linking into an app built with the newer glibc, the crashes happen.

Yes, that sounds right.

[julianoes]

I had it crashing building this: https://github.com/julianoes/siyi-a8-mini-camera-manager/

[julianoes]

@dakejahl were you able to reproduce the crash?

[dakejahl]

No we haven't seen any crashing, we run into the linking error if we install from deb so we've just been building mavsdk from source. A simple solution could be to host two arm64 debs, one for arm64-lts and one for arm64. That would cover 20.04 vs 22.04 on the Jetsons.

[julianoes]

Ok, I will try to create a small example application that demonstrates the crash. Then I'll leave it up to you to fix it, or maintain two versions, although I think that would be confusing and painful.

[julianoes]

Ah, maybe it's now dawning on me. So you're saying it does not crash if both build against an older version of glibc but once we mix it, that's when it explodes.

[dakejahl]

Well that was the theory, but Alex built logloader on the Pi using newer glibc

pi@ARK-Pi6X:~ $ uname -a
Linux ARK-Pi6X 6.6.20+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.20-1+rpt1 (2024-03-07) aarch64 GNU/Linux
pi@ARK-Pi6X:~ $ ldd --version
ldd (Debian GLIBC 2.36-9+rpt2+deb12u4) 2.36

and installed libmavsdk 2.4.0 which uses older glibc and he didn't have any crashing.

I will try to create a small example application that demonstrates the crash

That would be great!

[julianoes]

And did he use std::filesystem::path?

[dakejahl]

Yeah logloader uses it in a couple of places

[julianoes]

Ah I see. You build it on Ubuntu 20.04. Which Debian version did you try? Bullseye or Bookworm?

[dakejahl]

Bookworm, Debian 12

[AlexKlimaj]

Where did we land on this?

[100%] Linking CXX executable logloader
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `fstat64@GLIBC_2.33'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_wait_until_steady(unsigned int*, unsigned int, bool, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >)@GLIBCXX_3.4.29'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `std::__throw_bad_array_new_length()@GLIBCXX_3.4.29'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `stat64@GLIBC_2.33'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `pthread_cond_clockwait@GLIBC_2.34'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `pthread_create@GLIBC_2.34'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `pthread_mutex_trylock@GLIBC_2.34'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `std::__exception_ptr::exception_ptr::_M_release()@CXXABI_1.3.13'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `__libc_single_threaded@GLIBC_2.32'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `pthread_join@GLIBC_2.34'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `pthread_detach@GLIBC_2.34'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `std::__exception_ptr::exception_ptr::_M_addref()@CXXABI_1.3.13'
/usr/bin/ld: /usr/lib/libmavsdk.so.2.10.2: undefined reference to `pthread_once@GLIBC_2.34'
collect2: error: ld returned 1 exit status
make[3]: *** [CMakeFiles/logloader.dir/build.make:102: logloader] Error 1
make[3]: Leaving directory '/home/jetson/code/logloader/build'
make[2]: *** [CMakeFiles/Makefile2:76: CMakeFiles/logloader.dir/all] Error 2
make[2]: Leaving directory '/home/jetson/code/logloader/build'
make[1]: *** [Makefile:130: all] Error 2
make[1]: Leaving directory '/home/jetson/code/logloader/build'
make: *** [Makefile:8: install] Error 2

[julianoes]

@AlexKlimaj and @dakejahl: I'm sorry this is one of those cases where I would like to support each and everyone but I also don't want to waste all my day maintaining CI with old compilers, etc. If you can make a PR to add the lts dockcross images back in and commit to helping with those going forward, then I'm happy to accept the PR.

[AlexKlimaj]

#2304