produce .deb with cpack #165

abe-winter · 2023-10-21T01:38:58Z

What changed

add a new workflow, debian.yml, which produces .deb packages for bookworm aarch64 + x86_64
pkg-config changes:
- hardcode some paths to /usr. ⚠️ warning: this may break some direct-install flows
- remove Requires line -- it adds like a minute of CPU-bound processing inside the consumer's cmake config
this is being consumed here add .deb example module-example-cpp#1

Remaining work

Add a release flow (any way to upload these somewhere public). Current flow is to manually copy the artifact to our GCS bucket

Cleanup

These aren't merge blockers but should be fixed eventually:

Untangle example dependencies instead of deleting with sed

acmorrow

Overall, I'd like to know more about the intended use case for debian packages for the C++ SDK is right now. Are we planning to offer them on our own apt repository? I'm somewhat doubtful that we are in a position to correctly manage versions, especially given that we have yet to produce any tagged releases of this project.

src/viam/api/config/viam-cpp-sdk-libviamapi.pc.in

acmorrow · 2023-10-23T14:21:51Z

src/viam/api/config/viam-cpp-sdk-libviamapi.pc.in

@@ -7,7 +7,6 @@ Name: @PROJECT_NAME@-libapi
 Description: @PROJECT_DESCRIPTION@
 URL: @PROJECT_HOMEPAGE_URL@
 Version: @PROJECT_VERSION@
-Requires: gRPC++ >= @VIAMCPPSDK_GRPCXX_VERSION@ protobuf >= @VIAMCPPSDK_PROTOBUF_VERSION@


Similarly, these need to remain. Otherwise, projects using pkg-config to consume the SDK will fail to link.

I just read your comment that said "remove Requires line -- it adds like a minute of CPU-bound processing inside the consumer's cmake config". I don't see how that could be true. These files aren't consumed by cmake, only by pkg-config.

example run with Requires in pkg-config (last line has time):

# cmake .. -G Ninja -- The CXX compiler identification is GNU 12.2.0 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found PkgConfig: /usr/bin/pkg-config (found version "1.8.1") -- Checking for one of the modules 'viam-cpp-sdk-libviamsdk' -- Checking for one of the modules 'viam-cpp-sdk-libviamapi' -- Configuring done (31.1s)

Ah, I see. But why are you using pkg-config to obtain the SDK rather than using find_pacakge, which will use the cmake infra that the SDK provides?

src/viam/sdk/config/viam-cpp-sdk-libviamsdk.pc.in

acmorrow · 2023-10-23T14:24:14Z

src/viam/sdk/config/viam-cpp-sdk-libviamsdk.pc.in

 libdir=${prefix}/@CMAKE_INSTALL_LIBDIR@
-includedir=${prefix}/@CMAKE_INSTALL_INCLUDEDIR@
+includedir=${prefix}/@CMAKE_INSTALL_INCLUDEDIR@/viam/sdk


I'm not sure why this was changed. The viam/api tacked onto the include path for the libviamapi.pc.in is peculiar to the viamapi library, and is a side effect of some unfortunate naming issues in our api repository. But the sdk library doesn't have those problems, and the intended include root for the sdk library is definitely the INSTALL_INCLUDEDIR.

acmorrow · 2023-10-23T14:28:40Z

src/viam/sdk/config/viam-cpp-sdk-libviamsdk.pc.in


 Name: @PROJECT_NAME@-libviamsdk
 Description: @PROJECT_DESCRIPTION@
 URL: @PROJECT_HOMEPAGE_URL@
 Version: @PROJECT_VERSION@
-Requires: gRPC++ >= @VIAMCPPSDK_GRPCXX_VERSION@ protobuf >= @VIAMCPPSDK_PROTOBUF_VERSION@ @PROJECT_NAME@-libviamapi >= @PROJECT_VERSION@


As above, I don't think these Requires can or should be removed.

I think this constraint on the libviamapi version should remain as well, but looking more closely it looks slightly wrong, in that it doesn't exactly match the name at src/viam/api/config/viam-cpp-sdk-libviamapi.pc.in. I think maybe that .pc.in should have Name: @PROJECT_NAME@-lib**viam**api instead.

acmorrow · 2023-10-23T14:31:22Z

src/viam/sdk/CMakeLists.txt

+set(CPACK_ARCHIVE_COMPONENT_INSTALL ON)
+set(CPACK_COMPONENTS_ALL viam-cpp-sdk_dev)
+set(CPACK_PACKAGE_CONTACT "packaging@viam.com")
+set(CPACK_DEBIAN_PACKAGE_DEPENDS "libprotobuf-dev, libgrpc++-dev")


This list looks pretty incomplete. We definitely have dependencies on other development libraries. Also, I'd be hoping to see a split between dependencies required for build, for runtime, and for development. Does cpack let us express those? I'd also be interested to know if cpack has any ability to auto-detect these package dependencies rather than requiring us to explicitly enumerate them.

the viam-cpp-sdk .deb is a development library; we'll presumably give it a -dev suffix if we release. we're not creating a runtime library yet I think. given that, are -dev deps okay for now?

This change would generate deps automatically ... in theory:

-set(CPACK_DEBIAN_PACKAGE_DEPENDS "libprotobuf-dev, libgrpc++-dev") +set(CPACK_DEBIAN_PACKAGE_GENERATE_SHLIBS ON) +set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON)

But it detects runtime deps instead of -dev libraries:

Depends: libabsl20220623 (>= 0~20220623.0-1), libboost-log1.74.0 (>= 1.74.0+ds1), libc6 (>= 2.34), libgcc-s1 (>= 4.2), libgrpc++1.51 (>= 1.51.1), libgrpc29 (>= 1.51.1), libprotobuf32 (>= 3.21.12), libstdc++6 (>= 11)

My downstream module fails with:

/usr/include/viam/sdk/components/generic/generic.hpp:6:10: fatal error: google/protobuf/descriptor.h: No such file or directory 6 | #include <google/protobuf/descriptor.h> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ compilation terminated. /usr/include/viam/sdk/common/proto_type.hpp:5:10: fatal error: boost/variant/get.hpp: No such file or directory 5 | #include <boost/variant/get.hpp>

added libboost-log-dev in 82b6883

acmorrow · 2023-10-23T14:33:35Z

src/viam/sdk/CMakeLists.txt

+
+set(CPACK_GENERATOR "DEB")
+set(CPACK_ARCHIVE_COMPONENT_INSTALL ON)
+set(CPACK_COMPONENTS_ALL viam-cpp-sdk_dev)


Maybe this should be using CPACK_COMPONENTS_GROUPING to get separate debs for different roles?

separate debs for different roles

roles = development vs runtime?

Yes, typically on a deb based platform, a library ships at least three packages:

A libfoo package which contains the runtime role components (typically the explicitly versioned library file libfoo.so.x.y.z and its soname symlink libfoo.so.x).

A libfoo-dev development role package which contains the headers and the dev time lib symlink libfoo.so so that -lfoo finds the library.

A libfoo-dbg debug role package which contains debug info.

acmorrow · 2023-10-23T14:34:38Z

.github/workflows/debian.yml

+    - uses: actions/upload-artifact@v3
+      with:
+        name: debian-${{ matrix.label }}
+        path: build/viam-cpp-sdk-*.deb


What does the complete list of debs that gets made for a given matrix entry look like right now?

single .deb for viam-cpp-sdk
https://github.com/viamrobotics/viam-cpp-sdk/actions/runs/6593088962/job/17914991253#step:10:15

Uploaded /__w/viam-cpp-sdk/viam-cpp-sdk/build/viam-cpp-sdk-0.0.0-Linux.deb

.github/workflows/debian.yml

acmorrow · 2023-10-23T14:56:01Z

.github/workflows/debian.yml

+    - name: create build
+      run: mkdir build
+    - name: cmake
+      working-directory: ./build
+      run: cmake .. -G Ninja
+    - name: build
+      working-directory: ./build
+      run: ninja libviamsdk.so && cpack


Maybe better to use cmake as the driver, like:

cmake -S . -B ./build -G Ninja cmake --build ./build --target <targets> -- <ninja-options>

Also, do you need to install before you cpack? I'd sort of expect so, but I could be wrong. If so, thats a cmake --install

cmake command changes done in c6a3cf7 and 70bacb1

it seems to work without --install, but it's possible some of my workarounds are caused by not doing an install

…leasing .deb packages yet

abe-winter added 7 commits October 17, 2023 14:28

successful .deb if you remove examples

da72d8b

debian packaging workflow

1d6bba7

hardcode prefix, lowercase grpc

40be3cb

dev deps for dpkg, pkg-config fixes

2bbd119

remove Requires from pc.in -- it takes 56 seconds

e36a88d

use smaller runner to hopefully not get stuck in build queue

82c5fb2

hardcode /usr prefix

e7aa793

acmorrow requested changes Oct 23, 2023

View reviewed changes

acmorrow reviewed Oct 23, 2023

View reviewed changes

.github/workflows/debian.yml Show resolved Hide resolved

acmorrow reviewed Oct 23, 2023

View reviewed changes

abe-winter added 7 commits October 23, 2023 13:54

don't hardcode /usr prefix

8f501a5

on dispatch instead of on push; this is manual for now, we are not re…

60550da

…leasing .deb packages yet

todo to remove sed command

b6b1da2

use cmake commands instead of ninja

c6a3cf7

temporarily restore on: push

6c73a95

cpack in build dir

70bacb1

add boost-log dep

82b6883

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

produce .deb with cpack #165

produce .deb with cpack #165

abe-winter commented Oct 21, 2023

acmorrow left a comment

acmorrow Oct 23, 2023

acmorrow Oct 23, 2023

abe-winter Oct 23, 2023

acmorrow Oct 24, 2023

acmorrow Oct 23, 2023

acmorrow Oct 23, 2023

acmorrow Oct 23, 2023

abe-winter Oct 23, 2023 •

edited

acmorrow Oct 23, 2023

abe-winter Oct 23, 2023

acmorrow Oct 24, 2023

acmorrow Oct 23, 2023

abe-winter Oct 23, 2023

acmorrow Oct 23, 2023

abe-winter Oct 23, 2023

produce .deb with cpack #165

Are you sure you want to change the base?

produce .deb with cpack #165

Conversation

abe-winter commented Oct 21, 2023

What changed

Remaining work

Cleanup

acmorrow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abe-winter Oct 23, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abe-winter Oct 23, 2023 •

edited