[v0.9.3] Amalgamation for Android broken

[d4wud]

Amalgamation for Android still breaking in the recent release:

from mxnet_predict0.cc:3:
[...]/mxnet/mxnet/amalgamation/../dmlc-core/include/dmlc/logging.h:18:22: fatal error: execinfo.h: No such file or directory
 #include <execinfo.h>
                      ^
compilation terminated.
make: *** [mxnet_predict0.d] Error 1

Commenting out that #include <execinfo.h> creates the further error:

In file included from mxnet_predict0.cc:4:0:
[...]/mxnet/amalgamation/../src/ndarray/ndarray.cc:16:30: fatal error: opencv2/opencv.hpp: No such file or directory
 #include <opencv2/opencv.hpp>
                              ^
compilation terminated.
make: *** [mxnet_predict0.d] Error 1

It looks like the USE_OPENCV = 0 is being ignored?

[piiswrong]

you need to update submodules.

Try a fresh clone

[d4wud]

I cloned fresh and retried, same error as before. @piiswrong any ideas? Is this working for you?

[piiswrong]

USE_OPENCV was accidentally turned on in amalgamation/Makefile. Should be fixed by 01ae4df
@howard0su

[d4wud]

@piiswrong just tested, still breaking:

In file included from jni/predictor.cc:4:0:
jni/../mxnet_predict-all.cc:49:28: fatal error: packet/sse-inl.h: No such file or directory
 #include <packet/sse-inl.h>
                            ^
compilation terminated.

[piiswrong]

try make -DANDROID=1?

@howard0su USE_SSE is not turned off properly for JS and android. Moving it to make file doesn't work

[piiswrong]

should be fixed by #4797

[PhanTask]

@d4wud Hi, I met this trouble too. Do you succeed now?

[d4wud]

@piiswrong thanks for working on this.

I'm still having the problem posted above with #include <packet/sse-inl.h> on fix #4797

Any ideas?

[piiswrong]

Try adding -DMSHADOW_USE_SSE=0 to https://github.com/dmlc/mxnet/blob/master/amalgamation/Makefile#L20?

[d4wud]

I tried adding that to the CFLAGS in Makefile with no affect.

[d4wud]

@piiswrong Also no affect adding it to DEFS at line 20

[piiswrong]

Are you sure you are using a fresh clone of latest code? could you post the content of your mxnet_predict-all.cc?
Here is mine, it doesn't include sse-inl.h:
https://gist.github.com/piiswrong/deeb345ccc5c83426d2249ba86105cfc

[d4wud]

I did a fresh clone. On the most recent commit there are all kinds of new problems.

When I set head to your fix #4797 I am still seeing the #include <packet/sse-inl.h>. I've attached my mxnet_predict-all.cc.

mxnet_predict-all.cc.zip

[piiswrong]

make ANDROID=1 is working for me with most recent commit on Ubuntu14.

I'm out of ideas...

If anyone can reproduce this please try to debug or provide more information

[d4wud]

Wow that is really strange. All I can say is that my environment (Mac) amalgamates perfectly for v0.8.0 so I don't see how the problem could be outside mxnet.

@PhanTask Are you able to reproduce this error on c07fe6f?

[piiswrong]

You can try debugging it by removing the parts of mxnet_predict-all.cc that's causing problem.

[PhanTask]

@d4wud I cloned the latest version, but I can't even generate mxnet_predict-all.cc now.
I built OpenBLAS and wrote this in makefile:

export OPENBLAS_ROOT=/home/rokim/OpenBLAS
export CC=/home/rokim/android-toolchain/bin/arm-linux-androideabi-gcc
export CXX=/home/rokim/android-toolchain/bin/arm-linux-androideabi-g++

When I run "make ANDROID=1", I met two errors.
First error:

/home/rokim/android-toolchain/bin/arm-linux-androideabi-g++   -std=c++11 -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -fPIC -MD -MF nnvm.d \
		-I `pwd`/../ -I `pwd`/../include \
		-D__MIN__=0 -c nnvm.cc
arm-linux-androideabi-g++: error: unrecognized command line option '-msse2'

The second:

	-I /home/rokim/mxnet/dmlc-core/include \
	-D__MIN__=0 dmlc-minimum0.cc > dmlc.d
In file included from /home/rokim/mxnet/dmlc-core/include/dmlc/io.h:14:0,
                 from ../dmlc-core/src/io/line_split.cc:2,
                 from dmlc-minimum0.cc:6:
/home/rokim/mxnet/dmlc-core/include/dmlc/./logging.h:18:22: fatal error: execinfo.h: No such file or directory
 #include <execinfo.h>
                      ^
compilation terminated.

I commented execinfo.h in /home/rokim/mxnet/dmlc-core/include/dmlc/./logging.h but the first error still exists.

But I backed up the mxnet I used yesterday, on which I can still reproduce the error as yours, with the same mxnet_predict-all.cc:

#if defined(__MACH__)
#include <mach/clock.h>
#include <mach/mach.h>
#endif
#if !defined(__WIN32__)
#include <sys/stat.h>
#include <sys/types.h>
#if !defined(__ANDROID__) && (!defined(MSHADOW_USE_SSE) || MSHADOW_USE_SSE == 1)
#include <emmintrin.h>
#endif
#endif

#include <algorithm>
#include <array>
#include <assert.h>
#include <atomic>
#include <cblas.h>
#include <cctype>
#include <cfloat>
#include <chrono>
#include <climits>
#include <cmath>
#include <condition_variable>
#include <cstddef>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <ctime>
#include <deque>
#include <dirent.h>
#include <errno.h>
**#include <execinfo.h>**
#include <fstream>
#include <functional>
#include <inttypes.h>
#include <iostream>
#include <istream>
#include <limits>
#include <list>
#include <map>
#include <memory>
#include <mutex>
#include <new>
#include <ostream>
**#include <packet/sse-inl.h>** 
#include <queue>
#include <random>
#include <regex>
#include <sched.h>
#include <set>
#include <src/c_api/c_api_error.cc>
#include <src/c_api/c_api_graph.cc>
#include <src/c_api/c_api_symbolic.cc>
#include <src/core/graph.cc>
#include <src/core/node.cc>
#include <src/core/op.cc>
#include <src/core/pass.cc>
#include <src/core/symbolic.cc>
#include <src/pass/gradient.cc>
#include <src/pass/infer_shape_type.cc>
#include <src/pass/order_mutation.cc>
#include <src/pass/place_device.cc>
#include <src/pass/plan_memory.cc>
#include <src/pass/saveload_json.cc>
#include <sstream>
#include <stdbool.h>
#include <stddef.h>
#include <stdexcept>
#include <stdint.h>
#include <stdlib.h>
#include <streambuf>
#include <string>
#include <thread>
#include <time.h>
#include <tuple>
#include <type_traits>
#include <typeindex>
#include <typeinfo>
#include <unordered_map>
#include <unordered_set>
#include <utility>
#include <vector>

[d4wud]

@piiswrong this is what I'm seeing as well. Any idea how to proceed?

[piiswrong]

Seems to be a mac specific problem. I'll look into it

[piiswrong]

Made a fix 7c197d6

I tested it works on Mac 10.11.6

[dongwu92]

I did a fresh clone and make in amalgamation with Mac 10.12, and get the compile error in iOS project:
mxnet_predict-all.cc:617:19: Use of undeclared identifier 'backtrace'
mxnet_predict-all.cc:10826:3: Unknown type name 'cudaStream_t'
...

I set USE_CUDA = 0 in config.mk while making mxnet.
Any idea to solve?

[dongwu92]

Could anyone attach the generated mxnet_predict-all.cc file with newest version here, thanks!

[d4wud]

Hello Thread,

Some things have indeed been fixed, but I'm still not able to amalgamate. New crash on 0f6d583:

In file included from jni/predictor.cc:4:0:
jni/../mxnet_predict-all.cc:54:36: fatal error: src/c_api/c_api_error.cc: No such file or directory
 #include <src/c_api/c_api_error.cc>
                                    ^
compilation terminated.
make: *** [jni_libmxnet_predict.o] Error 1

I've attached my mxnet_predict-all.cc.zip:

mxnet_predict-all.cc.zip

[tcw165]

I found the generated mxnet_predict0.d is not referenced by the target, mxnet_predict-all.o. So many paths are broken. e.g. <src/c_api/c_api_error.cc> is living in the /nnvm/src/c_api/a_api_error.cc.

I manually set CFLAGS += -I${MXNET_ROOT}/ -I${MXNET_ROOT}/mshadow/ -I${MXNET_ROOT}/dmlc-core/include -I${MXNET_ROOT}/nnvm -I${MXNET_ROOT}/nnvm/include -I${MXNET_ROOT}/include -I${OPENBLAS_ROOT} -I${OPENBLAS_ROOT}/include -D__MIN__=$(MIN) and it could compile more but, it fails in the end, the message is:

jni/../mxnet_predict-all.cc:22749:25: error: 'log2f' was not declared in this scope

[PhanTask]

@d4wud Try to add DEFS in mxnet/nnvm/amalgamation/makefile:

DEFS+=-DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_SSE=0 -DDMLC_LOG_STACK_TRACE=0 -DMSHADOW_FORCE_STREAM -DMXNET_USE_OPENCV=0 -DMXNET_PREDICT_ONLY=1 -DDISABLE_OPENMP=1
export CFLAGS = -std=c++11 -Wall -O3 -Wno-unknown-pragmas -funroll-loops -Iinclude -fPIC $(DEFS)

[d4wud]

I am attempting amalgamation on the most recent commit 39a2c0c and progressing beyond where I was getting before. Unfortunately I'm not getting an error message, just simply a failure after a warning:

jni/../mxnet_predict-all.cc:17359:42: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       if (static_cast<int>(ignore_label) == k) {
                                          ^
make: *** [jni_libmxnet_predict.o] Error 1

This is on a Mac. On Linux it is breaking as well but with a different error:

In file included from jni/predictor.cc:4:0:
jni/../mxnet_predict-all.cc:22:26: fatal error: machine/ansi.h: No such file or directory
 #include <machine/ansi.h>
                          ^
compilation terminated.
Makefile:62: recipe for target 'jni_libmxnet_predict.o' failed
make: *** [jni_libmxnet_predict.o] Error 1

@piiswrong any ideas?

@PhanTask I tried your recommended changes but still no success.

[piiswrong]

Are you using FreeBSD? Seems like a freebsd specific problem.

For mac, there should have been an error before the warning

[d4wud]

@piiswrong no we're on Ubuntu 16.04.1 LTS.

On Mac I only see warnings. I've attached the entire output of make.
out.txt

[piiswrong]

How are you compiling it? You need -std=c++11. What's your compiler version?

[piiswrong]

As you can see here: ARM-based arm-linux-androideabi-
Toolchains are associated with gcc. What's your gcc version for your toolchain?

[leo-scorpio]

@d4wud Hi, I met the same issue. You can try adding " #define _GLIBCXX_USE_C99 1 " into the generated "mxnet_predict-all.cc"(or the proper position in the "amalgamation.py") and make again.

[leo-scorpio]

But I am not sure if this is the best solution, as there is no need to add such definition in the v0.8.0 setup.

[d4wud]

OK @leo-scorpio I'll give it a shot. Thanks!

@piiswrong here is my GCC version:

$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin16.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[d4wud]

@leo-scorpio #define _GLIBCXX_USE_C99 1 had no affect unfortunately.

[PhanTask]

@d4wud I met the same problem but I solved it by recreating NDK toolchain. You can try to recreate Standalone Toolchain with libc++ headers and libraries since the GNU STL library(libstdc++, by default) seems to not fully support c++11 features(such as function stod stoi stof to_string).

[PhanTask]

@d4wud Adding param '--stl=libc++' when running python make_standalone_toolchain.py works to me.
For more details:
android/ndk#82 (comment)
https://developer.android.com/ndk/guides/cpp-support.html

[d4wud]

@PhanTask I gave it a shot, but now something's breaking in the standalone NDK.

/mxnet/NDK_SATC/arm64_api24/include/c++/4.9.x/cstddef:43:25: fatal error: stddef.h: No such file or directory
 #include_next <stddef.h>
                         ^
compilation terminated.
make: *** [dmlc.d] Error 1

Any ideas what I'm doing wrong that would break something that fundamental as the link to stddef.h?

[PhanTask]

@d4wud I didn't met this error. It seems that there are many possible causes, you can google it for more solutions. Or maybe you can try those(I don't know if they works):

1. use arm-linux-androideabi-clang and clang++ (which I used)
2. add '-stdlib=libc++' to CFLAGS in Makefile.
3. Check your PATH(export PATH=$PATH:<ndk standalone chain>/bin)
4. use android-ndk-r13b and use api 21 (which I used):
   python make_standalone_toolchain.py --arch arm --api 21 --stl=libc++ --install-dir <ndk standalone toolchain>
   BTW I'm on Ubuntu 14.04.

[nihilityworld]

@PhanTask
I have tried to build as what you say, but I got
/opt/ndk/android-toolchain/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lc++ clang38++: error: linker command failed with exit code 1 (use -v to see invocation) Makefile:74: recipe for target 'jni_libmxnet_predict.so' failed make: *** [jni_libmxnet_predict.so] Error 1 .
Also I have tried to find libc++.so and find it only in the file /lib64.
Then I add '-L/opt/ndk/android-toolchain/lib64' to LDFLAGS in Makefile, and got an wanning
'warning: skipping incompatible /opt/ndk/android-toolchain/lib64/libc++.so while searching for c++'.
Using 'readelf -h libc++.so' to check it vision shows that it was a 64-bit library.
So the 'jni_libmxnet_predict.so' need a 32-bit library libc++.so and I cannot find one in android-toolchain.
How can I solve this problem?

[nihilityworld]

@PhanTask
I have find a 32-bit library libc++.so in /opt/ndk/android-ndk-r13b/sources/cxx-stl/llvm-libc++/libs/armeabi and succeed to get the libmxnet_predict.so.
But when I run an Android app with it, the app break and the Error is:
java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "__xstat" referenced by "libmxnet_predict.so"...

[PhanTask]

@nihilityworld Try to add libc++.so(or libc++_shared.so, which I used actually) to your android project(same location as libmxnet_predict.so) to see if it works.

[PhanTask]

@nihilityworld Also, I didn't add libc++.so or libc++_shared.so path in LDFLAGS manually, and didn't met your error and warning when running make ANDROID=1. So it is strange that everyone seems to met different errors and have different solutions...Good luck...

[nihilityworld]

@PhanTask Can you send me your 'libmxnet_predict.so' file? My e-mail address is 980078316@qq.com. Thanks!

[PhanTask]

@nihilityworld OK, I'm outside now, I will send it to you later.

[PhanTask]

@nihilityworld Sent it to you already, with libc++_shared.so. Please check.

[nihilityworld]

@PhanTask I have tried the library you send to me, but it break and I get the error:
Caused by: org.dmlc.mxnet.MxnetException: Failed loading Op pool1_norm1 of type LRN: [09:52:27] jni/../mxnet_predict-all.cc:22014: Check failed: op != nullptr Operator LRN is not registered
How to modify mxnet_predict-all.cc to register LRN?

I have tried mxnet/tests/python/mxnet_predict_example.py with the params and symbol file android used and can get right result.

[PKUEcho]

Hello guys, have you resolved the issues and built the android library successfully? I am still encountering the issues and post it here but seems there's no response. Really looking forwards to help.

[JianTse]

@PhanTask
I succeed to get the libmxnet_predict.so.
But when I run an Android app with it, the app break and the Error is:
java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "__cxa_thread_atexit" referenced by "libmxnet_predict.so"...
Can you send me your 'libmxnet_predict.so' file? My e-mail address is 280144557@qq.com.
Thanks!

[ZaccurLi]

@PhanTask @PetroWu @piiswrong @d4wud
Can anyone share his/her successful method of compiling amalgamation for Android pls?
I fellowed the "How to compile Amalgamation for android with NNPACK? #4969" and "Amalgamation on Android failed #4888", but there are same errors as above.
Pls show more details about the compile method, thanks you very much.

[edmBernard]

I achieve to compile mxnet for android with this ugly method: #6889
I don't try with NNPACK

[ZaccurLi]

I have achieved to compile mxnet amalgamation for android successfully, and put the steps at #7146
How to compile Amalgamation for android?
thanks a lot!

[novioleo]

@PhanTask what your mxnet_predict.so version?
have you met this problem when i used mxnet_predict.so which i compiled in android:

java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "_ZTVN4dmlc2io23IndexedRecordIOSplitterE"

i tried so many times~
can you give me some advice?
or send me your mxnet_predict.so. nsnovio@gmail.com

[zhenglaizhang]

@novioleo hi, i met with same error with you, compied for arm64, do you have any clue for this? thanks in advance!
with c++fit i checked that it should have something to do with dmlc::io::IndexedRecordIOSplitter but didn't know how to solve that yet.

[novioleo]

@zhenglaizhang i need to add a line to "dmlc-minimum0.cc":
#include "../dmlc-core/src/io/recordio_split.cc"

if you encounter the similar problem in the next compile progress,you should try to add relevant .cc file to dmlc-minimum0.cc or mxnet_predict0.cc.

[zhenglaizhang]

@novioleo yeah, thanks for the info, i succeeded in building the jni so.