Multi-architecture Fabric builds for arm64 (Apple Silicon) and Docker

[jkneubuh]

This discussion is an opportunity to review the impacts of PR #3877 as a general approach for providing multi-architecture builds of Fabric, starting with the 2.5 LTS release.

Providing support for multi-arch runtimes for Fabric has opened a long, complicated, and interleaved discussion with tendrils reaching far back into the original vision for PKCS11 and HSM. During the release-2.5 cycle, we started "peeling" back the layers, starting with what was originally a feature request to add support for Fabric binaries and Docker images for the new M1 / Apple Silicon macs. In this initial work, the updates encountered minimal turbulence, focusing on two points of turbulence:

• Use of buildx to prepare multi-arch Docker images with QEMU emulating the target architecture at build time.
• Introduction of gcc cross-compilers to statically link CGO dependencies on the C sqlite wrapper.

After the initial port was complete, we started turning alpha builds of the release-2.5 branch, and encountered a number of SIGSEGV violations running fabric binaries in Docker. Unwinding the dependencies between in-docker builds, release builds, native builds, and builds using the system golang with Fabric Makefiles has been a real challenge. The dependencies between all of these systems, usage contexts, and tools has turned into a bin-packing exercise, for which no single solution exists.

The assumption up to this point (and fatally flawed), was that a single make/build/docker system would be able to serve the needs of all contexts in which Fabric would be employed. In common usage patterns this is less of a concern, but the cross-overs into PKCS11 and HSM support domain have not been successful. Consider the following (legacy) Jira tickets, closed as WONTFIX/NO-SOLUTION/GO-AWAY lingering without resolution:

• FAB-2883
• FAB-3196
• FAB-18176
• FAB-16618

For PKCS-enabled environments, a dynamically built Fabric runtime, derived FROM ubuntu is required.

For all other environments, a statically built Fabric runtime, derived FROM alpine is strongly desired.

Herein lies the crux of the issue, that Alpine's mechanism for embedding libmusl runtimes is fundamentally incompatible with golang and CGO. Up to this point we have been "lucky," in that the experimental support for the golang-alpine image has worked well. With the introduction of the arm64, nothing really works correctly in this environment, unless the Fabric binaries have been linked statically, which breaks PKCS11 and HSM support.

Unwinding these dependencies has been challenging, considering the cross-platform "support" matrix of build/for/arch/os/runtime/binaries/pkcs/etc. is well over 100 entries and counting. PR #3877 resolves these dependencies, presenting a simplified view of the overall release practices by "binning" the environments into four different categories:

1. Local builds (make) run directly on the host system, using the system go. Nothing fancy here: just go.

2. Release builds (make release) use the system go cross-compiler to prepare statically linked executables . In cases requiring CGO (e.g. sqlite), the CC=... argument specifies the compiler and external CGO link options as "static."

3. Docker builds (make docker) package the statically linked binaries onto alpine, without support for PKCS11. No build of any kind is executed within the container - it's literally a simple COPY of the arch-specific, statically linked binaries into a Docker container.

4. PKCS11 and HSM Docker containers must be prepared by building FROM golang, using the system golang compiler and CGO to construct a dynamically linked executable. At runtime, the binaries will load vendor-specific HSM .so modules. PKCS11 docker images are NOT currently build outputs generated by Fabric, and support is only provided through documentation and a reference fabric-test suite for study by HSM system integrators.

This compromise brings a great simplification, stability, and predictability to Fabric's supported runtimes. Docker images are as small as possible, avoiding issues with the libc runtime by statically compiling under a cross-compiler. HSM implementations are free to extend the dynamically linked Fabric runtimes, avoiding the constraints that doomed the legacy JIRA issues above.

Most encouraging, is that it ... "just works!"

Please help identify cases where this is not so, and call them out here in this GH discussion, so we can make this just work for everyone.

[jkneubuh]

Example / reference build outputs using the new pipeline, based on PR #3877, and published to a GH fork:

• v2.5.0-alpha3 release binaries
• multi-arch docker images:

ghcr.io/jkneubuh/fabric-peer:2.5.0-alpha3
ghcr.io/jkneubuh/fabric-orderer:2.5.0-alpha3
ghcr.io/jkneubuh/fabric-tools:2.5.0-alpha3

[pfi79]

Do I understand correctly that the main build will be in a "make release"?

[pfi79]

Therefore, there is no reproducibility and repeatability of the build.

[pfi79]

I will build at my place, you will build at your place and the builds can be different

[jkneubuh]

The builds will differ between our environments based on:

• the version of go.
• the version of the GCC libc used by go/CGO when building the static binaries.

The version of go is specified in the Makefile and passed as a parameter in the release pipeline. This can be matched to the release rev, or vary if you want to build custom images with a new version of go.

The version of CC and libc may differ between our environments. You can reproduce the reference / release binaries by installing reference versions of the GCC cross-compilers, setting CC=... in the input environment and running the release target. This is effectively the "full tool chain," allowing you to reproduce the release builds run at GitHub.

Practically speaking, we are now relying on GO idioms to generate cross-platform, static binaries. Are the reproducibility concerns theoretical, or are there areas in the libc / static link / CGO toolchain where we would encounter differences in the runtimes between our builds?

[pfi79]

I read from you that the problem is in golang:alpine.

Maybe you need to fix their image https://github.com/docker-library/golang?

And if they don't accept the fix, put it with us.

[jkneubuhl]

Fabric has been extending the highly experimental golang-alpine base. It may be possible to resolve the issues in the base image, but it is not clear how and/or where to contribute a fix, even if we were able to discover the root cause. The golang #19938 issue has been opened since 2017.

Running the static link with musl-gcc from within the alpine image seems to help (see PR #3872), but has the side-effect of breaking pretty much everything about the way we run with pkcs11, which requires the dynamic load of .so for HSM modules.

Go project DockerHub:

golang:<version>-alpine
This variant is highly experimental, and not officially supported by the Go project (see golang/go#19938 for details).

The main caveat to note is that it does use musl libc instead of glibc and friends, which can lead to unexpected behavior. See this Hacker News comment thread for more discussion of the issues that might arise and some pro/con comparisons of using Alpine-based images.

I understand that the use of the system golang to build binaries does introduce a path where a locally built Docker images may differ from the official release images, in that they will include libc runtimes from the host GCC compiler. Is this a theoretical concern? Or are there areas where you are predicting troubles based on the usage of custom compilers, pkcs11 builds, etc. from the host system?

Another area we may want to investigate would be the migration from Alpine out to the ubuntu images. These are stable, but are significantly larger in size. The PR in question tries to find a good compromise between these separate concerns.

[jkneubuh]

Hi @C0rWin and @pfi79 and @denyeart and @davidkel

I tried switching the base docker image from alpine over to ubuntu:20.04. (This is what we are using in the reference binary builders at GH.)

The full image size, including gcc, golang, fabric binaries, etc. etc. is well over a GB in size, which is "too large."

With a multi-stage build, trimming out gcc, golang, etc. and leaving only Fabric binaries dynamically linked against the system libc adds about 60 MB to the image size over alpine. This is quite manageable, and so far seems 100% stable.

jkneubuh@gala fabric % docker images 
hyperledger/fabric-orderer   ubuntu    50d036398935   8 minutes ago    96.7MB
hyperledger/fabric-orderer   alpine    6bce754b0547   35 minutes ago   40MB
ubuntu                       20.04     43ed104e759f   12 days ago      65.7MB
alpine                       latest    d3156fec8bcb   4 weeks ago      7.46MB

So we either:

• switch from alpine to ubuntu:20.04 or
• run static builds on a reference system with cross-compilers to link in libc at compile time or
• abandon multi-platform support for Fabric or
• do something else?

The latter two alternatives aren't viable, IMO. Switching over to ubuntu as the base image has a high chance of success for building Fabric images linked in with pkcs11 and HSM shared libraries.

I think this is a winner. The cost is 60MB of (cached) image layer size to our base images.

Thoughts?

[pfi79]

I like this solution better

[C0rWin]

@jkneubuh have you tried debian-slim image?

[jkneubuh]

No. Can we please focus on finding a stable solution for all environments, first and foremost.

• ubuntu:20.04 is THE reference environment used to build the supported release binaries.
• Images built FROM ubuntu:20.04 are looking good. "Rock solid."
• Images with pkcs11 support will invariably be FROM ubuntu:20.04 +/- installation of HSM .so modules. This is the "same image," +/- installation of golang, cc, etc.

After the multi-arch images are in place, we can consider a shift to an alternate, e.g. debian-slim. There are many different contexts being juggled in this PR.

[C0rWin]

Ok, so switching to ubuntu sounds like a good choice.

[jkneubuh]

Candidate PRs for multi-arch support:

• Switch base docker image from golang-alpine to ubuntu:20.04 #3882 :

FROM ubuntu:20.04
make GO_TAGS=${GO_TAGS}

• COPY static release build artifacts into alpine Docker - DO NOT MERGE #3877 :
  on host OS: reproducible static binaries with golang's cross-arch build and libc runtimes:

${CC}=gcc-musl GOOS=${TARGETOS} GOARCH=${TARGETARCH} make release ...

FROM ALPINE 
COPY release/${TARGETOS}-${TARGETARCH}/bin /usr/local/bin

[davidkel]

Thumbs up to move to Ubuntu images and drop alpine. Ubuntu also supports arm64 well such as Raspberry pi so I would expect a quality linux-arm64 image there as well

[jkneubuh]

All set!

Fabric 2.5.0-alpha3 and fabric-ca 1.5.6-beta3 are looking good on the M1. It took some real work to unwind all the tendrils. Thank you all for the input, testing, feedback, and encouragement. This was a big project. :|

High-level summary of the updates for M1 / multi-arch:

• Dev builds (make) are dynamic link binaries, using the system golang toolchain.
• Release builds (make release) prepare static link binaries, using golang's cross-compiler tool chain.
• Docker builds (make docker) prepare dynamic link binaries, FROM ubuntu.
• Docker builds (GitHub release action) generate multi-arch {arm64, amd64}, dynamic link binaries FROM ubuntu. Builds are generated in GHA via buildx running under QEMU.
• PKCS11 builds (both local and in Docker) are dynamic link binaries, enabling a runtime load of vendor HSM .so libs.
• Fabric CA release binaries prepare static link binaries, using an additional GCC/musl cross-complier to link a libc and sqlite runtime.

And finally ... my desk is quiet again - no more fan noise!!!