docker backend support buildx cross platform builds without containerd-snapshotter

[chris-smith-zocdoc]

Fixes #19994

Currently the docker cross-platform build support relies on using containerd-snapshotter https://www.pantsbuild.org/2.19/docs/docker#buildx-support which may not be desirable. The feature itself is still experimental and testing within our org encountered both performance problems and bugs that prevent adoption.

We're currently building/publishing cross-platform images using buildx and the docker-container driver

This is a very rough draft to allow the docker backend to support buildx cross platform builds without containerd-snapshotter.

I outlined the challenges in slack but I'll repeat them here for posterity.

• When performing cross-platform builds using the docker-continer driver, you cannot export the resulting manifest to the docker driver (the default one). You can export a single image, but you need to either build with a single valued --platform flag or not specify
• Push and Build and not separate cli commands when using buildx in this configuration, so the pants rules for package and publish need to behave differently when using buildx cross platform without containerd-snapshotter

Example commands that this would invoke

docker_image(
    ...
    build_platform = [
        "linux/amd64",
        "linux/arm64",
    ],
)

package

docker buildx build --output=type=docker --pull=False --tag --file src/Dockerfile .

publish

docker buildx build --platform=linux/amd64,linux/arm64 --pull=False --push --tag --file src/Dockerfile .

Misc Discussion Points

Should the package goal should prefer to build all platforms or to export the image to the docker driver? Should be configurable? It is somewhat awkward to have package not actually build all the artifacts, but at least for our use case, we prefer to build and test the image in the CI pipeline, which necessitates outputting to docker driver.

Should pants detect if containerd-snapshotter is being used?
docker info -f '{{ .DriverStatus }}' can be used for this If not, should there be a setting on the subsystem?

Testing locally

If you want to test locally, first you must create the buildx builder like

docker buildx create --name builder --driver docker-container --use --bootstrap

Right now I'm forcing the use of a particular buildx builder via env var on the subsystem, but I don't love this as it applies to all targets. I think adding a field for this on docker_image would make sense (--builder cli flag)

[docker]
use_buildx = true
env_vars = [
  "BUILDX_BUILDER=builder",
]

[chris-smith-zocdoc]

The publish goal is awkward for these targets because while they have a package already, its not usable. We need to re-invoke docker buildx build with the --push flag and --platform

[chris-smith-zocdoc]

I'll need to move all tag/registry logic into the DockerBuildSetup rule also

[chris-smith-zocdoc]

build_docker_image was split up so I can re-use the majority of it from publish

[chris-smith-zocdoc]

We need to apply slightly different options to package and publish. Theres probably a nicer way I could clean this up

[chris-smith-zocdoc]

--push is a "shortcut" flag, we can accomplish the same thing if we set --output=type=image,push=true

[kaos]

I like the general direction here.

We likely want this to be optional at every step--possibly with some detection logic to do the sensible thing by default. (like using the docker info .. call to detect build driver.)

Also, in case publish does a build+push combo, avoid requesting the normal BuiltPackage for that target, as that would be a wasted build.

I'm about to head off on easter vacation, so will be more or less offline until April 8.

[chris-smith-zocdoc]

Also, in case publish does a build+push combo, avoid requesting the normal BuiltPackage for that target, as that would be a wasted build.

This does happen right now, how can I accomplish this?

I'm about to head off on easter vacation, so will be more or less offline until April 8.

Enjoy your vacation!