Reproducible build

[nasdf]

These changes are required to make the build reproducible.

• Adding -s -w flags to ldflags strips out non-deterministic debug info.
• Setting buildid to an empty value removes the random go build id.
• The trimpath flag should be set via GOFLAGS=-trimpath.

This is the Dockerfile I've been using to test for deterministic binaries.

FROM golang:buster
WORKDIR /home/ipfs/go/src/github.com/ipfs/
RUN git clone -b reproducible https://github.com/nasdf/go-ipfs
WORKDIR /home/ipfs/go/src/github.com/ipfs/go-ipfs
RUN GOFLAGS=-trimpath make build

This script will run the build and print a sha256 of the final binary.

#!/bin/sh

docker build . -t repro/ipfs
docker run -it --rm repro/ipfs sha256sum cmd/ipfs/ipfs
docker rmi repro/ipfs
docker builder prune -f

Still need to test cross compiling for all platforms. Any feedback would be appreciated.

[welcome]

Thank you for submitting this PR!
A maintainer will be here shortly to review it.
We are super grateful, but we are also overloaded! Help us by making sure that:

• The context for this PR is clear, with relevant discussion, decisions
  and stakeholders linked/mentioned.

• Your contribution itself is clear (code comments, self-review for the
  rest) and in its best form. Follow the code contribution
  guidelines
  if they apply.

Getting other community members to do a review would be great help too on complex PRs (you can ask in the chats/forums). If you are unsure about something, just leave us a comment.
Next steps:

• A maintainer will triage and assign priority to this PR, commenting on
  any missing things and potentially assigning a reviewer for high
  priority items.

• The PR gets reviews, discussed and approvals as needed.

• The PR is merged by maintainers when it has been approved and comments addressed.

We currently aim to provide initial feedback/triaging within two business days. Please keep an eye on any labelling actions, as these will indicate priorities and status of your contribution.
We are very grateful for your contribution!

[Stebalien]

There be balrogs here.

I need to be able to:

1. Checkout the go-ipfs source locally.
2. Run make install.
3. Checkout the https://github.com/ipfs/go-ds-s3/ reop.
4. Run make install IPFS_VERSION=/path/to/go-ipfs/source (s3 datastore plugin).
5. Start the IPFS daemon, correctly loading the s3 plugin.

Unfortunately, we've found that:

1. The bare -trimpath flag works with plugins when go-ipfs is built with go build github.com/....
2. The bare -trimpath flag does not work with plugins when go-ipfs is built with make build.

Somewhere in the deep dark reaches of Khazad-dûm (the go compiler), the plugin loader gets confused.

If you can figure out how to reconcile this... I'll be very happy.

[nasdf]

Looking into this now.

Some interesting things I found when building without bare -trimpath.

The only difference between binaries is this path not being trimmed.

/tmp/go-build971257389/b082/_cgo_gotypes.go

I'm guessing this is another issue with the compiler. Maybe an easier one to fix than the plugin loader.

[Stebalien]

Note: I haven't checked this for at least one go release cycle. The situation may have improved.

[nasdf]

It seems the situation has not improved golang/go#31354.

Building the plugins within the same path as the application seems to be a viable option.

This project is able to build plugins with -trimpath via this method.

[nasdf]

I've managed to get the go-ds-s3 plugin working by building inside of the IPFS project root.

Steps I've taken to produce the binary and plugin:

Clone go-ds-s3 repo to extensions folder.

Copy go.mod to extensions.mod

Copy go.sum to extensions.sum

Edit extensions.mod with the following:

require (
	...
	github.com/ipfs/go-ds-s3 v0.0.0
)

replace github.com/ipfs/go-ds-s3 => ./extensions/go-ds-s3

Edit extensions/go-ds-s3/go.mod with the following:

replace github.com/ipfs/go-ipfs => ../../

Copy line below from extensions/go-ds-s3/go.sum to extensions.sum

github.com/ipfs/go-ipfs v0.8.0/go.mod h1:iU9jeKJrinNYdWSNZyTjpfcmGHH4oflEySOUboEQsoQ=

Build go-ipfs with -trimpath

GOFLAGS=-trimpath make build

Build go-ds-s3 plugin with -trimpath

go build -trimpath -modfile extensions.mod --buildmode=plugin -o s3plugin.so extensions/go-ds-s3/plugin/main/main.go

[Stebalien]

That's a pretty big step backwards in terms of usability. Plugins are already confusing.

[Stebalien]

What happens if you use -trimpaths and the asm/gcflags?

[Stebalien]

Another acceptable solution would be to make the Makefile not override user-provided flags, but instead set defaults.

[nasdf]

That's a pretty big step backwards in terms of usability. Plugins are already confusing.

I totally agree. My goal was to first find a solution and then work on developer ergonomics. Possibly a Dockerfile with everything setup or a script that automates the steps above. I'm hoping to move everything to https://github.com/ipfs/distributions once I find an acceptable solution.

What happens if you use -trimpaths and the asm/gcflags?

Using the asm/gcflags somehow overrides -trimpath and causes the build to be non reproducible. The Go team suggests to avoid using those flags and use -trimpath instead.

Update: I tried a build using both asm/gcflags and -trimpath. It was not compatible with plugins.

Another acceptable solution would be to make the Makefile not override user-provided flags, but instead set defaults.

I think this is a good solution. I wasn't able to successfully set -ldflags via GOFLAGS but that is probably my lack of experience with Makefiles.

[nasdf]

I've managed to reduce the steps as well as allow building from an outside directory.

Clone go-ipfs and go-ds-s3 to the same directory.

git clone -b v0.8.0 https://github.com/ipfs/go-ipfs.git
git clone -b v0.7.0 https://github.com/ipfs/go-ds-s3.git

Edit the ./go-ds-s3/go.mod.

go mod edit -replace "github.com/ipfs/go-ipfs=../go-ipfs"

Edit the ./go-ipfs/go.mod.

go mod edit -require="github.com/ipfs/go-ds-s3@v0.7.0"
go mod edit -replace "github.com/ipfs/go-ipfs=../go-ds-s3"
go mod tidy
# I'm unsure why this is required but possibly due to nested directory structure
go get github.com/ipfs/go-ds-s3/plugin

Comment out this line in ./go-ipfs/mk/golang.mk.

#override GOFLAGS += "-asmflags=all='-trimpath=$(GOPATH)'" "-gcflags=all='-trimpath=$(GOPATH)'"

Build ipfs from ./go-ipfs:

GOFLAGS=-trimpath make build

Build go-ds-s3 from ./go-ipfs:

go build -trimpath --buildmode=plugin -o go-ds-s3.so ../go-ds-s3/plugin/main/main.go

[aschmahmann]

Just grabbing this as a spot for a GitHub thread.

It depends what type of reproducible build you're going for. For example, the builds produced for dist.ipfs.io (at least the Linux ones at the moment) have support for CGO which IIUC means that your steps for reproducibility also need to include a shared understanding of the C compiler used.

It's reasonably possible that we'll be releasing binaries built with CGO support in more platforms in the future (e.g. Windows + OS X) if we want to support databases like SQLite, or other libraries which which require CGO.

The area that seems more likely to be able to deal with reproducible builds in this context is within https://github.com/ipfs/distributions. We recently moved over to using Docker containers to do the builds which should help out on the reproducibility front. However, we can make plenty of changes there without running into issues with local builds, etc. since people doing local builds are unlikely to use the heavier deployment + cross compilation machinery in ipfs/distributions.

[nasdf]

We are aiming to have reproducible builds for multi-factor releases (multiple parties sign off on the same binary.). CGO support would be preferable as we want to release the same set of binaries as dist.ipfs.io.

I agree having it in the distributions repository makes more sense. My limited understanding of Makefiles is why I ended up copying those build flags in directly. I will make a PR on the distributions repo once I solve the plugin issue.

[kpcyrd]

I was sent to this issue while investigating the rebuild failure on reproducible.archlinux.org and the patch I've appended below makes the package reproducible for us.

As @nasdf says, the asm/gcflags interfere with -trimpath, according to golang/go#47256 (comment) setting the low-level commands like this isn't really supported. :)

diff --git a/mk/golang.mk b/mk/golang.mk
index 0b2a2c55a..2c7f24d75 100644
--- a/mk/golang.mk
+++ b/mk/golang.mk
@@ -12,7 +12,7 @@ GOTFLAGS ?=
 unexport GOFLAGS
 # Override so we can combine with the user's go flags.
 # Try to make building as reproducible as possible by stripping the go path.
-override GOFLAGS += "-asmflags=all='-trimpath=$(GOPATH)'" "-gcflags=all='-trimpath=$(GOPATH)'"
+override GOFLAGS += "-trimpath"
 
 ifeq ($(tarball-is),1)
        GOFLAGS += -mod=vendor

[...]
  -> Generating .MTREE file...
  -> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: go-ipfs 0.9.0-1 (Sat 17 Jul 2021 11:12:48 AM CEST)
  -> built succeeded! built packages can be found in /var/lib/archbuild/reproducible/testenv/pkgdest
==> comparing artifacts...
  -> Package 'go-ipfs-0.9.0-1-x86_64.pkg.tar.zst' successfully reproduced!
% b2sum go-ipfs-0.9.0-1-x86_64.pkg.tar.zst /var/lib/archbuild/reproducible/testenv/pkgdest/go-ipfs-0.9.0-1-x86_64.pkg.tar.zst 
c38ba9a0bfc03115a4a4c594b9999f72eacf4b12782f08f1a4599e86985ed05be1c7df5f968d01278e4f488f1762d046b5db18dc3e552967493a90614cd82f70  go-ipfs-0.9.0-1-x86_64.pkg.tar.zst
c38ba9a0bfc03115a4a4c594b9999f72eacf4b12782f08f1a4599e86985ed05be1c7df5f968d01278e4f488f1762d046b5db18dc3e552967493a90614cd82f70  /var/lib/archbuild/reproducible/testenv/pkgdest/go-ipfs-0.9.0-1-x86_64.pkg.tar.zst
% 

[aschmahmann]

My understanding is that this PR is still undergoing work. I'm going to switch it to a draft PR and whenever this is ready for review (i.e. builds pass, we have tests confirming plugins still work, there aren't unrelated dependencies added to go-ipfs, etc.) please switch back to a non-draft PR and reach out.

Thanks for taking a look at this 😄.

[RubenKelevra]

Sooo... Arch does reproduceable builds right now. Can we upstream these changes? :)