libgrpc.a libssl.so.3 linking error for a Ubuntu 22.04 gRPC/protobuf application

[gitterhitter]

What version of gRPC and what language are you using?

gRPC v1.62.0
C++

What operating system (Linux, Windows,...) and version?

Ubuntu 22.04.1 LTS

What runtime / compiler are you using (e.g. python version or version of gcc)

cmake version 3.22.1
gcc and g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

What did you do?

Followed the instructions at 'https://grpc.io/docs/languages/cpp/quickstart/' with updated cmake options to build gRPC dependencies from source with static linking. Specifically the following commands were executed as root:

cd /root
export MY_INSTALL_DIR=/opt/grpc/grpc-1.62.0
mkdir -p $MY_INSTALL_DIR
export PATH="$MY_INSTALL_DIR/bin:$PATH"
sudo apt update
sudo apt install -y build-essential autoconf libtool pkg-config
git clone --recurse-submodules -b v1.62.0 --depth 1 https://github.com/grpc/grpc
cd grpc
mkdir -p cmake/build
pushd cmake/build
cmake -DgRPC_INSTALL=ON -DCMAKE_BUILD_TYPE=Release -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=$MY_INSTALL_DIR \
-DgRPC_ABSL_PROVIDER=module -DgRPC_CARES_PROVIDER=module -DgRPC_PROTOBUF_PROVIDER=module \
-DgRPC_RE2_PROVIDER=module -DgRPC_SSL_PROVIDER=module -DgRPC_ZLIB_PROVIDER=module \
-Dprotobuf_BUILD_SHARED_LIBS=OFF ../..
make --jobs=$(nproc)
make install
popd
echo "/opt/grpc/grpc-1.62.0/lib" | tee /etc/ld.so.conf.d/grpc1.conf
ldconfig
which protoc
/opt/grpc/grpc-1.62.0/bin/protoc --version # output is 'libprotoc 25.1'.

The gRPC/protobuf build and install above are successful. All good so far.
Next, I compile my application and attempt to link its libraries, but I get this linking error:

/usr/bin/ld: /opt/grpc/grpc-1.62.0/lib/libgrpc.a(ssl_transport_security.cc.o): undefined reference to symbol 'SSL_CTX_set_keylog_callback@@OPENSSL_3.0.0'
/usr/bin/ld: /lib/x86_64-linux-gnu/libssl.so.3: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

What did you expect to see?

100% successful gRPC/protobuf build and install (yes, I saw this).
100% successful build for custom Ubuntu 22.04 gRPC/protobuf application (no, I did not see this).

What did you see instead?

Unexpected linker errors in my custom Ubuntu 22 gRPC/protobuf application, specifically:

/usr/bin/ld: /opt/grpc/grpc-1.62.0/lib/libgrpc.a(ssl_transport_security.cc.o): undefined reference to symbol 'SSL_CTX_set_keylog_callback@@OPENSSL_3.0.0'
/usr/bin/ld: /lib/x86_64-linux-gnu/libssl.so.3: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

Given that I built gRPC with shared-libs off, why is libssl.so.3 needed?

Anything else we should know about your project / environment?

My custom application is being ported from Ubuntu 16 to Ubuntu 22. No issues exist on Ubuntu 16 using gPRC v1.54.0. After experiencing this linking issue with v1.54.0 on Ubuntu 22, I upgraded gRPC to v1.62.0 but the issue remains. Has anyone else encountered a similar linking issue? How did you resolve it?

I see a similar issue here (googleapis/google-cloud-cpp#9063) but the original poster in that case was compiling gRPC using packages rather than source (modules).

If I search libssl.so.3 for its symbols, I see the undefined symbol:

# readelf -Ws /lib/x86_64-linux-gnu/libssl.so.3 | grep SSL_CTX_set_keylog_callback
583: 0000000000035cc0    12 FUNC    GLOBAL DEFAULT   15 SSL_CTX_set_keylog_callback@@OPENSSL_3.0.0

Any suggestions for next steps in debugging or added visibility into the issue?

[gitterhitter]

I believe the issue described above may have been related to older/out-of-date libraries that were on my development platform at alternative/additional library search paths. Performing some clean-up on my development platform enabled me to resolve/get-past this issue.

[yashykt]

Such build issues are always tricky/annoying to debug. Glad you got it fixed.