Install to-disk fails with latest containers-common

[ckyrouac]

This took awhile to track down. I'm going to continue investigating but I wanted to document what I've found so far.

The failure happens when attempting a bootc install to-disk using an image built from a base image with at least one extra layer, e.g.

FROM quay.io/centos-bootc/centos-bootc-dev:stream9
RUN dnf install -y tmux

If the image is built locally bootc install to-disk works correctly. The failure happens when pushing the image to a repo (only tested with quay.io), clearing out the image from local storage via podman system prune --all, then running bootc install to-disk. Here's example output of the failure:

[test@fedora-39 ~]$ sudo podman run --pid=host --network=host --privileged --security-opt label=type:unconfined_t -v /var/lib/containers:/var/lib/containers -v .:/output -v /dev:/dev -e RUST_LOG=debug quay.io/ckyrouac/bootc-lldb bootc install to-disk --via-loopback --generic-image --skip-fetch-check /output/test.raw
Trying to pull quay.io/ckyrouac/bootc-lldb:latest...
Getting image source signatures
...
ERROR Installing to disk: Creating ostree deployment: Performing deployment: Importing: Parsing layer blob sha256:5d35bfe747b2c76a01310e69a14daa90baf352a11f990f73d4ce3e1917668719: Failed to invoke skopeo proxy method FinishPipe: remote error: corrupted blob, expecting sha256:dede69b8181937a545b87707fbe4ace2ee9936459ffd43b7ba957324861992a0

So, the OpenImage call to the skopeo proxy is failing.

The latest version of containers-common found in Fedora39/40 repos sets pull_options.enable_partial_images=true in /usr/share/containers/storage.conf. This is the change that started causing this error. Toggling enable_partial_images to false resolves the error. I'm not familiar enough with this stack to know the root cause of this yet. I'll continue digging but I'm sure someone else would be able to track this down a lot quicker if you think it's urgent.

[cgwalters]

The latest version of containers-common found in Fedora39/40 repos sets pull_options.enable_partial_images=true in /usr/share/containers/storage.conf. This is the change that started causing this error. Toggling enable_partial_images to false resolves the error.

Ugh. Fun...thanks for finding and debugging this.

[cgwalters]

It's actually really embarrassing that this wasn't caught by our CI, needs fixing

[cgwalters]

Actually wait this is the -dev image which is intentionally tracking git main, I don't think this has hit f40 or stream9 proper yet. I see a lot of activity in https://src.fedoraproject.org/rpms/containers-common/commits/rawhide and what I bet is happening here is those spec files are being pulled into the copr.

cc @rhatdan @lsm5

[cgwalters]

And yes, we need to add bootc test gating to containers-common and skopeo pretty soon.

[ckyrouac]

hmm interesting, earlier in the week this was happening regardless of which base image I used. Just went to verify that and now this bug only happens with the -dev base image.

[lsm5]

Actually wait this is the -dev image which is intentionally tracking git main, I don't think this has hit f40 or stream9 proper yet. I see a lot of activity in https://src.fedoraproject.org/rpms/containers-common/commits/rawhide and what I bet is happening here is those spec files are being pulled into the copr.

cc @rhatdan @lsm5

The last build of containers-common on the podman-next copr was an automatic rebuild of the rawhide sources from sometime back. I disabled this automatic rebuild after we got rawhide to a sane-enough state.

Let me know if you need an update to the fedora or copr rpm. I can do a one-off build.

We're currently working on a packit workflow from upstream c/common to downstream containers-common rpm, like we have for podman and the rest, with automatic builds going to podman-next right after every upstream commit to main. I'm hoping that change will land early next week.

[ckyrouac]

so this works now using any base image. I'm not sure what changed. I guess something in the base images or in quay.io?

[cgwalters]

@vrothberg