Consider keeping go-git as "pure Go" library

[radeksimko]

Hi folks,
Firstly - thank you for all the effort that goes into maintaining this library! ❤️

Over in https://github.com/hashicorp/hc-install we noticed that https://github.com/go-git/go-git/releases/tag/v5.5.0 introduced a CGO dependency https://github.com/pjbgf/sha1cd as part of #618 which made our Windows build fail.

hashicorp/hc-install#82

I appreciate that the library does have fallbacks (as introduced in 5.5.1), making it also work in non-CGO build environment. The (perhaps unfortunate?) reality is however, that Go has CGO_ENABLED=1 defaults. This, in combination with us distributing a library (rather than an app) means that our own downstream is very likely to run into compilation issues on some platforms, as cross-compilation is generally difficult with CGO.

We can of course just disable CGO and suggest downstream to do the same, but before taking that potentially painful step, I wanted to ask if you'd consider any resolution that would keep go-git as a pure Go library without any CGO dependency? It would help library consumers who wouldn't have to tweak the default options and wouldn't have to ask all of downstream to do the same.

One of the main reasons we use go-git is to make minimal assumptions about the environment, but this recent change introduces an assumption, making the proposition far less interesting, to the point that I am considering abandoning go-git it for os/exec and calling out to git. It will introduce another type of headaches for end users who may not have git installed, but we trade it for the headache of downstream libraries having to set CGO_ENABLED=0. 🤷🏻

I will understand if the answer is no and respect it, but I wanted to share the context with the hope that you'll consider it.

[pjbgf]

@radeksimko Thank you for raising this. TL;DR: I introduced the CGO dependency into go-git as a temporary step before sha1cd implements SIMD optimisations. But that should soon go away.

For context, go-git previously did not support collision detection (which both Git CLI and libgit2 uses, as mentioned in the PR). The collision detection in itself is a lot less efficient than normal SHA1, given the additional state and operations it requires. The main reason why I ended up defaulting to the cgo version of sha1cd is that it is currently 10x faster than the "pure Go" version, so the drive was combining security and performance by default.

For short and long terms, I am working on:

1. Add SIMD optimisations to sha1cd, so we can default to the Go version instead - removing the need for CGO.
2. Add sha256 support into go-git, so it is ready for when the major Git servers are able to deal with it. Then when SHA256 SHA-NI optimisations merges into upstream Go, we will get the best of both worlds: go-git with faster and more secure hashes.

From my point of view, we should arrive there quite soon.

[radeksimko]

I think it would still be worth pursuing a pure Go solution, but FYI the recent Go 1.20 release contains a helpful change: https://tip.golang.org/doc/go1.20#cgo

The go command now disables cgo by default on systems without a C toolchain. More specifically, when the CGO_ENABLED environment variable is unset, the CC environment variable is unset, and the default C compiler (typically clang or gcc) is not found in the path, CGO_ENABLED defaults to 0. As always, you can override the default by setting CGO_ENABLED explicitly.

It still doesn't quite help in the context of a library, as I initially described above, since we typically support last two stable releases, i.e. 1.19 + 1.20, so we'd at least have to wait for 1.21 to land, but it looks like a progress in the right direction. It may help any consumers with apps however, who can afford to just start building against 1.20.

[pjbgf]

@radeksimko thanks for following up and yes, that would indeed be very helpful.

On the PureGo front, I am almost done on getting sha1cd beating the CGO implementation's performance on AMD64. Once that is done we should be able to set it as the default.

[siennathesane]

I stumbled across this issue, I can confirm upgrading to v5.5.1 resolves the issue.

[radeksimko]

@mxplusb It resolves the issue for those who explicitly set CGO_ENABLED=0, or folks using Go 1.20 (which defaults to CGO_ENABLED=0 when gcc (C toolchain) isn't installed.

For anyone else (on Go < 1.20 or not wanting to explicitly set CGO_ENABLED=0 e.g. for reasons explained above), this is still an ongoing issue.

[mfreeman451]

this is broken, please fix.