feat: add moby/buildkit #21500

ponkio-o · 2024-04-02T08:21:45Z

moby/buildkit: concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit

$ aqua g -i moby/buildkit

How to confirm if this package works well

Reviewers aren't necessarily familiar with this package, so please describe how to confirm if this package works well.
Please confirm if this package works well yourself as much as possible.

Command and output

root@f5204bd5a9d4:/workspace# buildctl
NAME:
   buildctl - build utility

USAGE:
   buildctl [global options] command [command options] [arguments...]

VERSION:
   v0.13.1

COMMANDS:
   du               disk usage
   prune            clean up build cache
   prune-histories  clean up build histories
   build, b         build
   debug            debug utilities
   help, h          Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --debug                enable debug output in logs
   --addr value           buildkitd address (default: "unix:///run/buildkit/buildkitd.sock")
   --log-format value     log formatter: json or text (default: "text")
   --tlsservername value  buildkitd server name for certificate validation
   --tlscacert value      CA certificate for validation
   --tlscert value        client certificate
   --tlskey value         client key
   --tlsdir value         directory containing CA certificate, client certificate, and client key
   --timeout value        timeout backend connection after value seconds (default: 5)
   --wait                 block RPCs until the connection becomes available
   --help, -h             show help
   --version, -v          print the version

If files such as configuration file are needed, please share them.

Reference

README
- https://github.com/moby/buildkit#exploring-dockerfiles

[moby/buildkit](https://github.com/moby/buildkit): concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit

ponkio-o · 2024-04-02T08:32:04Z

Why doesn't show [buildctl] on right the command name when execute the aqua g?

In my understand, it display if files[].name exists on registry.yaml.

suzuki-shunsuke · 2024-04-02T08:44:48Z

Why doesn't show [buildctl] on right the command name when execute the aqua g?

This is because aqua g checks files only on the top level setting.
This means aqua g doesn't check files in version_overrides.
So you need to set files on the top level setting too for aqua g.
I don't think this is ideal, but this is the current behaviour.

ponkio-o · 2024-04-02T08:48:50Z

This is because aqua g checks files only on the top level setting.

ah, I see. Thank you for teaching me!

suzuki-shunsuke · 2024-04-02T23:36:57Z

darwin has only buildctl, but linux has other tools.
I guess buildkitd is needed.

BuildKit is composed of the buildkitd daemon and the buildctl client. While the buildctl client is available for Linux, macOS, and Windows, the buildkitd daemon is only available for Linux and *Windows currently.

root@b1bfb9dbf142:/workspace# tree /root/aquaproj-aqua/pkgs/github_release/github.com/moby/buildkit/v0.13.1/buildkit-v0.13.1.linux-arm64.tar.gz
/root/aquaproj-aqua/pkgs/github_release/github.com/moby/buildkit/v0.13.1/buildkit-v0.13.1.linux-arm64.tar.gz
`-- bin
    |-- buildctl
    |-- buildkit-cni-bridge
    |-- buildkit-cni-firewall
    |-- buildkit-cni-host-local
    |-- buildkit-cni-loopback
    |-- buildkit-qemu-arm
    |-- buildkit-qemu-i386
    |-- buildkit-qemu-mips64
    |-- buildkit-qemu-mips64el
    |-- buildkit-qemu-ppc64le
    |-- buildkit-qemu-riscv64
    |-- buildkit-qemu-s390x
    |-- buildkit-qemu-x86_64
    |-- buildkit-runc
    `-- buildkitd

Windows

root@539bd41cb7dd:/workspace# tree /root/aquaproj-aqua/pkgs/github_release/github.com/moby/buildkit/v0.13.1/buildkit-v0.13.1.windows-arm64.tar.gz
/root/aquaproj-aqua/pkgs/github_release/github.com/moby/buildkit/v0.13.1/buildkit-v0.13.1.windows-arm64.tar.gz
`-- bin
    |-- buildctl.exe
    `-- buildkitd.exe

suzuki-shunsuke · 2024-04-02T23:41:05Z

*.provenance.json have been released.
Maybe we can verify these provenance with slsa-verifier.

ponkio-o · 2024-04-02T23:46:47Z

darwin has only buildctl, but linux has other tools.

Yes. buildctl is a client CLI, like kubectl in Kubernetes.

I guess buildkitd is needed.

The buildkitd is probably not necessary. Because buildkitd is generally run as a container.
https://docs.docker.com/build/drivers/docker-container/
For example, when you run docker buildx build, the buildkitd container is launched in your laptop.

Of course it may be possible to run it, but I don't think aqua supports many daemon-type tools for use on Linux. (Like WebServer, Database...)

suzuki-shunsuke · 2024-04-02T23:46:59Z

*.provenance.json have been released. Maybe we can verify these provenance with slsa-verifier.

It didn't work well.

        slsa_provenance:
          type: github_release
          asset: buildkit-{{.Version}}.{{.OS}}-{{.Arch}}.provenance.json

No certificate provided, trying Redis search index to find entries by subject digest
Verifying artifact /tmp/585610650: FAILED: error searching rekor entries: no matching entries found

FAILED: SLSA verification failed: error searching rekor entries: no matching entries found
INFO[0010] Verification by slsa-verifier failed temporarily, retring  aqua_version=2.25.2 env=linux/amd64 package_name=moby/buildkit package_version=v0.13.1 program=aqua registry=standard retry_count=1 wait_time=902ms
No certificate provided, trying Redis search index to find entries by subject digest
Verifying artifact /tmp/585610650: FAILED: error searching rekor entries: no matching entries found

FAILED: SLSA verification failed: error searching rekor entries: no matching entries found
INFO[0011] Verification by slsa-verifier failed temporarily, retring  aqua_version=2.25.2 env=linux/amd64 package_name=moby/buildkit package_version=v0.13.1 program=aqua registry=standard retry_count=2 wait_time=289ms
No certificate provided, trying Redis search index to find entries by subject digest
Verifying artifact /tmp/585610650: FAILED: error searching rekor entries: no matching entries found

FAILED: SLSA verification failed: error searching rekor entries: no matching entries found
INFO[0012] Verification by slsa-verifier failed temporarily, retring  aqua_version=2.25.2 env=linux/amd64 package_name=moby/buildkit package_version=v0.13.1 program=aqua registry=standard retry_count=3 wait_time=40ms
No certificate provided, trying Redis search index to find entries by subject digest
Verifying artifact /tmp/585610650: FAILED: error searching rekor entries: no matching entries found

FAILED: SLSA verification failed: error searching rekor entries: no matching entries found
INFO[0012] Verification by slsa-verifier failed temporarily, retring  aqua_version=2.25.2 env=linux/amd64 package_name=moby/buildkit package_version=v0.13.1 program=aqua registry=standard retry_count=4 wait_time=147ms
No certificate provided, trying Redis search index to find entries by subject digest
Verifying artifact /tmp/585610650: FAILED: error searching rekor entries: no matching entries found

FAILED: SLSA verification failed: error searching rekor entries: no matching entries found
ERRO[0013] install the package                           aqua_version=2.25.2 env=linux/amd64 error="verify a package with slsa-verifier: verify with slsa-verifier" package_name=moby/buildkit package_version=v0.13.1 program=aqua registry=standard
FATA[0013] aqua failed                                   aqua_version=2.25.2 env=linux/amd64 error="it failed to install some packages" program=aqua
[ERROR] Build failed linux/amd64

Support for provenance attestations moby/buildkit#3240

suzuki-shunsuke · 2024-04-02T23:51:54Z

The buildkitd is probably not necessary. Because buildkitd is generally run as a container.
https://docs.docker.com/build/drivers/docker-container/
For example, when you run docker buildx build, the buildkitd container is launched in your laptop.

Of course it may be possible to run it, but I don't think aqua supports many daemon-type tools for use on Linux. (Like WebServer, Database...)

I see. It makes sense. Thank you.

- aquaproj#21500 (comment)

suzuki-shunsuke · 2024-04-03T00:03:32Z

$ slsa-verifier version
  ____    _       ____       _             __     __  _____   ____    ___   _____   ___   _____   ____
 / ___|  | |     / ___|     / \            \ \   / / | ____| |  _ \  |_ _| |  ___| |_ _| | ____| |  _ \
 \___ \  | |     \___ \    / _ \    _____   \ \ / /  |  _|   | |_) |  | |  | |_     | |  |  _|   | |_) |
  ___) | | |___   ___) |  / ___ \  |_____|   \ V /   | |___  |  _ <   | |  |  _|    | |  | |___  |  _ <
 |____/  |_____| |____/  /_/   \_\            \_/    |_____| |_| \_\ |___| |_|     |___| |_____| |_| \_\
slsa-verifier: Verify SLSA provenance for Github Actions

GitVersion:    2.5.1
GitCommit:     eb7007070baa04976cb9e25a0d8034f8db030a86
GitTreeState:  clean
BuildDate:     2024-03-25T14:54:53
GoVersion:     go1.21.8
Compiler:      gc
Platform:      darwin/arm64

$ slsa-verifier verify-artifact \
  --provenance-path buildkit-v0.13.1.darwin-arm64.provenance.json \
  --source-uri github.com/moby/buildkit \
  buildkit-v0.13.1.darwin-arm64.tar.gz
No certificate provided, trying Redis search index to find entries by subject digest
Verifying artifact buildkit-v0.13.1.darwin-arm64.tar.gz: FAILED: error searching rekor entries: no matching entries found

FAILED: SLSA verification failed: error searching rekor entries: no matching entries found

$ slsa-verifier verify-artifact --help
Verifies SLSA provenance on artifact blobs given as arguments (assuming same provenance)

Usage:
  slsa-verifier verify-artifact [flags] artifact [artifact..]

Flags:
      --build-workflow-input map[]     [optional] a workflow input provided by a user at trigger time in the format 'key=value'. (Only for 'workflow_dispatch' events on GitHub Actions). (default map[])
      --builder-id string              [optional] the unique builder ID who created the provenance
  -h, --help                           help for verify-artifact
      --print-provenance               [optional] print the verified provenance to stdout
      --provenance-path string         path to a provenance file
      --provenance-repository string   image repository for provenance with format: <registry>/<repository>
      --source-branch string           [optional] expected branch the binary was compiled from
      --source-tag string              [optional] expected tag the binary was compiled from
      --source-uri string              expected source repository that should have produced the binary, e.g. github.com/some/repo
      --source-versioned-tag string    [optional] expected version the binary was compiled from. Uses semantic version to match the tag

suzuki-shunsuke · 2024-04-03T00:05:20Z

Hmm. We give up verifying provenance for now.

suzuki-shunsuke · 2024-04-03T00:30:45Z

v4.157.0 is out 🎉
https://github.com/aquaproj/aqua-registry/releases/tag/v4.157.0

suzuki-shunsuke · 2024-04-03T00:42:33Z

About provenance, I opened a discussion.

How do we verify release assets with provenance? moby/buildkit#4821

ponkio-o added 2 commits April 2, 2024 17:18

feat: add moby/buildkit

c234df2

[moby/buildkit](https://github.com/moby/buildkit): concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit

fix: command name

63b9be5

Merge branch 'main' into feat/moby/buildkit

39a8900

suzuki-shunsuke added the enhancement New feature or request label Apr 2, 2024

suzuki-shunsuke added this to the v4.156.1 milestone Apr 2, 2024

suzuki-shunsuke mentioned this pull request Apr 2, 2024

feat: add moby/buildkit/buildctl and moby/buildkit/buildkitd #1876

Closed

suzuki-shunsuke linked an issue Apr 2, 2024 that may be closed by this pull request

Add moby/buildkit #1870

Closed

fix(moby/buildkit): fix files

e422904

fix(moby/buildkit): remove buildkitd

847ec43

- aquaproj#21500 (comment)

suzuki-shunsuke merged commit 4e03543 into aquaproj:main Apr 3, 2024
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add moby/buildkit #21500

feat: add moby/buildkit #21500

ponkio-o commented Apr 2, 2024

ponkio-o commented Apr 2, 2024 •

edited

suzuki-shunsuke commented Apr 2, 2024 •

edited

ponkio-o commented Apr 2, 2024

suzuki-shunsuke commented Apr 2, 2024

suzuki-shunsuke commented Apr 2, 2024

ponkio-o commented Apr 2, 2024 •

edited

suzuki-shunsuke commented Apr 2, 2024 •

edited

suzuki-shunsuke commented Apr 2, 2024

suzuki-shunsuke commented Apr 3, 2024 •

edited

suzuki-shunsuke commented Apr 3, 2024

suzuki-shunsuke commented Apr 3, 2024

suzuki-shunsuke commented Apr 3, 2024

feat: add moby/buildkit #21500

feat: add moby/buildkit #21500

Conversation

ponkio-o commented Apr 2, 2024

How to confirm if this package works well

ponkio-o commented Apr 2, 2024 • edited

suzuki-shunsuke commented Apr 2, 2024 • edited

ponkio-o commented Apr 2, 2024

suzuki-shunsuke commented Apr 2, 2024

suzuki-shunsuke commented Apr 2, 2024

ponkio-o commented Apr 2, 2024 • edited

suzuki-shunsuke commented Apr 2, 2024 • edited

suzuki-shunsuke commented Apr 2, 2024

suzuki-shunsuke commented Apr 3, 2024 • edited

suzuki-shunsuke commented Apr 3, 2024

suzuki-shunsuke commented Apr 3, 2024

suzuki-shunsuke commented Apr 3, 2024

ponkio-o commented Apr 2, 2024 •

edited

suzuki-shunsuke commented Apr 2, 2024 •

edited

ponkio-o commented Apr 2, 2024 •

edited

suzuki-shunsuke commented Apr 2, 2024 •

edited

suzuki-shunsuke commented Apr 3, 2024 •

edited