[ROCm] ROCm support?

[lshqqytiger]

I already had a look on #8095 and #8267.
But when I tried to build Oneflow on dev_rocm branch, I got:

(oneflow-dev-gcc7) ~/oneflow/build$ source ../../env.sh
(oneflow-dev-gcc7) ~/oneflow/build$ cmake .. -DBUILD_CUDA=OFF -DBUILD_ROCM=ON -DONEFLOW=ON -DUSE_CLANG_FORMAT=OFF -DCMAKE_BUILD_TYPE=Release -DTHIRD_PARTY=ON -DTREAT_WARNINGS_AS_ERRORS=OFF -DTHIRD_PARTY_MIRROR=aliyun -DBUILD_HWLOC=OFF -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++
(...)
[100%] Building CXX object lib/TableGen/CMakeFiles/LLVMTableGen.dir/TGParser.cpp.o
[100%] Linking CXX static library ../libLLVMTableGen.a
[100%] Built target LLVMTableGen
Scanning dependencies of target oneflow_tblgen
[100%] Linking CXX executable ../../bin/oneflow_tblgen
ld.lld: error: undefined symbol: std::__throw_bad_array_new_length()
(...)
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[3]: *** [tools/OneFlowTableGen/CMakeFiles/oneflow_tblgen.dir/build.make:96: bin/oneflow_tblgen] Error 1
gmake[2]: *** [CMakeFiles/Makefile2:68360: tools/OneFlowTableGen/CMakeFiles/oneflow_tblgen.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:68421: tools/OneFlowTableGen/CMakeFiles/install-oneflow-tblgen.dir/rule] Error 2
gmake: *** [Makefile:20186: install-oneflow-tblgen] Error 2
CMake Error at cmake/oneflow.cmake:281 (find_library):
  Could not find LLVMSupportLib using the following names: LLVMSupport
Call Stack (most recent call first):
  CMakeLists.txt:329 (include)


-- Configuring incomplete, errors occurred!

env.sh:

#!/bin/bash

export ROCM_PATH=/opt/rocm
echo $ROCM_PATH
export HIP_PATH=${ROCM_PATH}/hip
export AMDGPU_TARGETS="gfx900;gfx906;gfx1010;gfx1030"
export PATH=${ROCM_PATH}/bin:${ROCM_PATH}/llvm/bin:${ROCM_PATH}/hcc/bin:${ROCM_PATH}/hip/bin:$PATH

export LD_LIBRARY_PATH=${ROCM_PATH}/lib:${ROCM_PATH}/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${ROCM_PATH}/hip/lib:${ROCM_PATH}/llvm/lib:${ROCM_PATH}/opencl/lib/x86_64:$LD_LIBRARY_PATH
export C_INCLUDE_PATH=${ROCM_PATH}/include:${ROCM_PATH}/llvm/include:/opencl/include:${C_INCLUDE_PATH}
export CPLUS_INCLUDE_PATH=${ROCM_PATH}/include:${ROCM_PATH}/llvm/include:/opencl/include:${CPLUS_INCLUDE_PATH}
export PATH=${ROCM_PATH}/miopen/bin:${ROCM_PATH}/rocblas/bin:${ROCM_PATH}/hipsparse/bin:$PATH
export LD_LIBRARY_PATH=${ROCM_PATH}/miopen/lib:${ROCM_PATH}/rocblas/lib:$LD_LIBRARY_PATH
export MIOPEN_SYSTEM_DB_PATH=${ROCM_PATH}/miopen/share/miopen/db/
export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH
export LIBRARY_PATH=/usr/lib64:$LIBRARY_PATH

export RCCL_PATH=$ROCM_PATH/rccl
export NCCL_PATH=$ROCM_PATH/rccl
export LD_LIBRARY_PATH=$RCCL_PATH/lib:$LD_LIBRARY_PATH

export MIOPEN_FIND_MODE=3

rocm-smi

I searched about undefined symbol: std::__throw_bad_array_new_length() error.
It seems that there's some conflicts between C/C++ compiler and linker.
In env.sh of #8095, there's /opt/rocm/hip/samples/1_Utils/hipInfo/hipInfo.
And I found that /opt/rocm/hip/samples directory existed on specific ROCm version, but it removed on ROCm 5.3.0 or earlier version.
These informations above make it difficult to determine exactly which version of ROCm I should use and what caused the error.
Are Oneflow planning to support ROCm? Or had tries to support ROCm been frustrated/stopped?

Environment:
Ubuntu 22.04.1 LTS
Conda 22.9.0 (Because I used commit 5b985c863ef73fc3f3b597289fdcf740d52cf71c of conda-env, CMake found and used python 3.6.13 of conda env oneflow-dev-gcc7)
ROCm 5.3.0.50300-63~22.04

[Ldpe2G]

May be you can try with this repo: https://github.com/yuguo-Jack/oneflow/tree/0.9.1-rocm, which can be built successfully on the HYGON DCU plantform. The HYGON DCU plantform also support ROCm software stack.

Compile OneFlow on HYGON DCU plantform

env.sh

#!/bin/bash

#get DTKROOT by env.sh location
DTKROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
export DTKROOT=$(dirname $DTKROOT)
export AMDGPU_TARGETS="gfx906"
export ROCM_PATH=/opt/rocm
export HIP_PATH=${ROCM_PATH}/hip
export MIOPEN_FIND_MODE=3
export PATH=$ROCM_PATH/bin:$ROCM_PATH/llvm/bin:$ROCM_PATH/hip/bin:${ROCM_PATH}/miopen/bin:$PATH
export LD_LIBRARY_PATH=${ROCM_PATH}/lib:${ROCM_PATH}/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${ROCM_PATH}/hip/lib:${ROCM_PATH}/llvm/lib:$LD_LIBRARY_PATH
export C_INCLUDE_PATH=${ROCM_PATH}/include:${ROCM_PATH}/llvm/include${C_INCLUDE_PATH:+:${C_INCLUDE_PATH}}
export CPLUS_INCLUDE_PATH=${ROCM_PATH}/include:${ROCM_PATH}/llvm/include${CPLUS_INCLUDE_PATH:+:${CPLUS_INCLUDE_PATH}}

export RCCL_PATH=$ROCM_PATH/rccl
export NCCL_PATH=$ROCM_PATH/rccl
export LD_LIBRARY_PATH=$RCCL_PATH/lib:$LD_LIBRARY_PATH

export HSA_FORCE_FINE_GRAIN_PCIE=1
# export HIP_VISIBLE_DEVICES=0,1,2,3
export MIOPEN_COMPILE_PARALLEL_LEVEL=1

rocm-smi

install cmake

 wget https://github.com/Kitware/CMake/releases/download/v3.24.2/cmake-3.24.2.tar.gz
 tar zxf cmake-3.24.2.tar.gz
 cd cmake-3.24.2
 ./bootstrap
 make -j64
 sudo make install

build oneflow

pip install -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com -r dev-requirements.txt

mkdir build

cmake .. -DBUILD_CUDA=OFF -DBUILD_ROCM=ON -DONEFLOW=ON -DUSE_CLANG_FORMAT=OFF -DCMAKE_BUILD_TYPE=Release -DTHIRD_PARTY=ON -DTREAT_WARNINGS_AS_ERRORS=OFF -DTHIRD_PARTY_MIRROR=aliyun -DBUILD_HWLOC=OFF -DCMAKE_C_COMPILER=${ROCM_PATH}/llvm/bin/clang -DCMAKE_CXX_COMPILER=${ROCM_PATH}/llvm/bin/clang++ -DBUILD_TESTING=ON -DBUILD_RDMA=OFF

make -j64

Run tests after build

${ONEFLOW_PROJECT_DIR}/build/bin/oneflow_testexe

cd ${ONEFLOW_PROJECT_DIR}/python/oneflow/test/modules，
python3 -m unittest discover -v -s ./ -p "*test_*.py"

[lshqqytiger]

I tried to build not using conda env, but same error occurred.
And I'm using Ubuntu 22.04.1, not CentOS. So I couldn't install devtoolset-7 and I don't know how I can install corresponding libraries for Ubuntu.
Should I downgrade the GCC and G++ versions from 11.3.0 to 7?

[Ldpe2G]

Try this solution https://root-forum.cern.ch/t/compiling-cling-error-numeric-limits-is-not-a-member-of-std/48812 ?

[Ldpe2G]

or build oneflow inside a centos container

[lshqqytiger]

I tried to build on a docker container (RHEL) with Devtoolset-7, but it failed with another error.
Thank you for help! But I'm tired now and will try it again next time.

[Ldpe2G]

May be next time, you can try to build oneflow with the ROCM official docker image. 😄

[Boom-Hacker]

7900xtx?what is dcu