Build fails when vendoring

[vladklokun]

Describe the bug

When vendoring dependencies, building Kubescape fails because pkg-config cannot find built git2go artifacts. pkg-config cannot find the artifacts because it is looking for them in the wrong directory: the vendor directory instead of the git2go directory.

Environment

OS: Ubuntu 22.04 LTS
Version: current Git master. Kubescape version is irrelevant, since it is unable to build.

Steps To Reproduce

Steps to reproduce the behavior:

1. Clone the repository
2. Run make libgit2
3. Run go mod vendor
4. Run make build
5. Observe the results

Expected behavior

The build system is able to build the Kubescape executable and put it in the working directory.

Actual Behavior

The build system fails with an error:

$ make build
# [SNIPPED FOR BREVITY]
k8s.io/client-go/kubernetes/typed/storage/v1
github.com/kubescape/opa-utils/reporthandling/results/v1/prioritization
github.com/kubescape/opa-utils/reporthandling/helpers/v1
github.com/libgit2/git2go/v33
# pkg-config --cflags --static -- /home/kvd/sandbox/kubescape/vendor/github.com/libgit2/git2go/v33/static-build/instal
l/lib/pkgconfig/libgit2.pc
Failed to open '/home/kvd/sandbox/kubescape/vendor/github.com/libgit2/git2go/v33/static-build/install/lib/pkgconfig/li
bgit2.pc': No such file or directory
No package '/home/kvd/sandbox/kubescape/vendor/github.com/libgit2/git2go/v33/static-build/install/lib/pkgconfig/libgit
2.pc' found
pkg-config: exit status 1
golang.org/x/mod/semver
# [SNIPPED FOR BREVITY]

Additional context

I’ve run into this while I was trying to change the code of an upstream module—opa-utils. I wanted to change upstream code and adjust Kubescape accordingly.

While this use case is better served by using Go workspaces, as I have discovered later, I think that vendoring modules should not break the build process.

A workaround that helped me with vendoring before I moved on to Go workspaces was:

1. Bulid the git2go: make libgit2
2. Vendor the packages: go mod vendor
3. Copy the bulit libgit2 directory to the vendor directory: cp -r ./git2go ./vendor/github.com/kubescape/libgit2/git2go/v33
4. Run the build: make build

[github-actions]

Hey, Welcome to this repo, Congratulations on opening your issue. Keep Contributing to Kubescape

We'll try to review and add you work as soon as possible and a maintainer will get back to you soon!

[Meyazhagan]

@vladklokun I too got similar error message while make build. After Copying the bulit libgit2 directory to the vendor, build succeeds. Shall I debug it !

[Meyazhagan]

@vladklokun In go.mod file, I saw replace github.com/libgit2/git2go/v33 => ./git2go was this causing the issue

[Meyazhagan]

@dwertent shall I work on this !

[Aman123lug]

@Meyazhagan can i work with you man?

[0xt3j4s]

Hey @vladklokun and @Meyazhagan,
Since there have been no updates on this issue for a long time now, I tried to tinker a bit and followed the steps to reproduce the bug. The build (make build) fails with an error as given below:

make: *** [Makefile:25: build] Error 2

To get the context and more details, I tried the following:

$ make         
git submodule update --init --recursive
cd git2go; make install-static
make[1]: Entering directory '/home/tejas/Projects/Kubescape/kubescape/git2go'
go install --tags "static" ./...
make[1]: Leaving directory '/home/tejas/Projects/Kubescape/kubescape/git2go'
go build -v -tags="gitenabled,static" .
github.com/libgit2/git2go/v33
# pkg-config --cflags --static -- /home/tejas/Projects/Kubescape/kubescape/vendor/github.com/libgit2/git2go/v33/static-build/install/lib/pkgconfig/libgit2.pc
Package vendor/github.com/libgit2/git2go/v33/static-build/install/lib/pkgconfig/libgit2.pc was not found in the pkg-config search path.
Perhaps you should add the directory containing `/home/tejas/Projects/Kubescape/kubescape/vendor/github.com/libgit2/git2go/v33/static-build/install/lib/pkgconfig/libgit2.pc.pc'
to the PKG_CONFIG_PATH environment variable
Package '/home/tejas/Projects/Kubescape/kubescape/vendor/github.com/libgit2/git2go/v33/static-build/install/lib/pkgconfig/libgit2.pc', required by 'virtual:world', not found
pkg-config: exit status 1
make: *** [Makefile:25: build] Error 2

I think there is some issue with git2go in the dependencies, i.e., in the go.mod. For this, I also referred to @amirmalka's comment regarding the removal of git2go and updating of the scanning logic of a repository.

I am interested in working on this issue. How should I proceed further?

[matthyx]

@vladklokun

[vladklokun]

@0xt3j4s Hi! Please feel free to pick up any issues, especially if you see no activity inside them for a prolonged period of time.

Regarding your question, I would suggest you start by understanding why pkg-config is looking in the vendor directory and is there a way to make it look in the git2go directory. The make logs you have posted might help you. Take a look at the directories the tools in the build process look in and check if they make sense. For example, the following line looks suspicious:

Perhaps you should add the directory containing `/home/tejas/Projects/Kubescape/kubescape/vendor/github.com/libgit2/git2go/v33/static-build/install/lib/pkgconfig/libgit2.pc.pc'

Should pkg-config look there? Why the double extension? And so on.

As an aside, git2go docs might help you, too. I remember they have multiple ways to install and suggest different options when using it with your project (static build, dynamic build, git submodule etc), so take a look there. This might give you an understanding on which one we use and how this might affect vendoring. If we’re lucky, you might be able to score a quick win on how to adjust the build process to not break when vendoring.

Next, I would take a look at our go.mod replace directives. Maybe it’s one of them that breaks the build when vendoring. Which might be related to adjustments we make to support the httphandler part of Kubescape. Or not.

All in all, this is might be a rabbit hole that would require you to understand how we fetch and link against git2go and how that interacts with vendoring: the directory structure, linker calls and so on.

Good luck!

[0xt3j4s]

Sure, I'll try to figure out why pkg-config is looking in the vendor directory and also read the documentation of git2go.
Thank you for the insight

[dwertent]

Resolved in release v3.0.4