Publish windows lifecycle images

[ekcasey]

Lifecycle 0.9.0 will support windows. We need to publish windows lifecycle images to enable platforms like pack to use untrusted windows builder images.

I think we should make the buildpacksio/lifecycle:0.9.0 tag point to a manifest list that includes manifests for both linux and windows images. Then when the tag is pulled the correct image for the docker daemon platform will be resolved. In addition, we may want to publish simple os specific tags pointing to the individual manifests (the golang image tags provide a good example of this pattern https://hub.docker.com/_/golang).

It's also worth noting that the windows lifecycle image will need to include foreign layers (media type: application/vnd.docker.image.rootfs.foreign.diff.tar.gzip). We probably don't need to do anything special to make this happen if we use either a Dockerfile or ggcr to produce these images.

[natalieparellano]

Some thoughts after experimenting a bit with this today:

• For Linux, we should change to using the build-and-push action from Docker (https://github.com/docker/build-push-action) instead of the current random one (https://github.com/elgohr/Publish-Docker-Github-Action)
• For Windows, it doesn't seem that Docker's action currently supports Windows (although there is an open issue). I found this one: https://github.com/mr-smithers-excellent/docker-build-push
  • Do we want to use this action? Alternatively I think we can use something like azure/docker-login@v1 + writing out the commands.
  • Regardless of what we use, we'll need to adapt the bash/interpolation in the current get_metadata step to work on Windows
• For the manifest list, we need to enable experimental features in the docker CLI. I believe this is possible but haven't got it working yet.

[ekcasey]

I was wondering if, instead of using a Dockerfile, we should write a simple program in go. Given the ecosystem of helpers we have for generating images it shouldn't be too difficult. I think it would make it easy to make and test changes if we do things like add new labels.

[natalieparellano]

Ah, that's an interesting idea. We could look into it. I wonder if we'd still need to enable experimental features in the daemon to create the manifest list. It seems that updating the daemon.json in a GitHub-hosted GHA runner isn't possible, so we'd need our own self-hosted runner if updating the config is required.

[natalieparellano]

I may have been confused... it seems that to enable the manifest list experimental feature we just need to update the config.json, which should be possible.

[natalieparellano]

I started looking into this today - the most complicated piece (I believe) will be extracting the downloaded lifecycle tgz into a layer tar format. pack currently does this here: https://github.com/buildpacks/pack/blob/5c7a6934b057103b25a2b3a3537e24c05707cc5d/internal/builder/builder.go#L682 in an inconveniently unexported function. I'd like to be able to take advantage of all the cross platform goodies that we are currently adding - perhaps this is something that could be factored out into the library... thoughts? cc @micahyoung @jromero

[micahyoung]

Apologies for the late chime-in.

Having this in a standalone utility should work well, though extracting that embedLifecycleTar logic looks like it would need to be reworked to be appropriately generic for moving to imgutil if that was the intended destination. So long as your LayerWriter is the imgutil/layer.WindowsWriter for Windows and tar.Writer for Linux, the generated images should be fine.

One potential concern right now though is that we haven't solved baselayer-less images yet for Windows (e.g. we haven't made pack package-buildpack work for Windows yet) and these lifecycle images would unusable in some scenarios by the same issue that makes build packages not "just work" already on Windows.

The gist of that problem is that the Windows daemon will only pull/store images that have 6 specific files in the bottom-most layer (details). I only discovered this experimentally and it is not documented in moby, the OCI spec nor is it an issue for registries, just for Windows Docker daemons (not sure about containerd/k8s).

My ideal hope is that we'll be able to solve that without adding these 6 special files to any images (just have pack/lifecycle dynamically add those files before they land on the daemon). If that approach doesn't work, we'll need to add these special files to every single-layer Windows image that we publish (likely as a second layer that pack/lifecycle discards). This latter approach would probably have to be spec'd out and all that. The work to drive this out is coming up after our main pack build-for-Windows issue is complete.

But with all that said, I don't feel like the work for lifecycle images has to be blocked since the registries can store the these images and --publish scenarios would be fine with them.

[micahyoung]

And regarding using manifest lists, we'll likely run into issues using them in remote scenarios where we use imgutil to fetch them with remote.NewImage("lifecycle-manifest-list", ...) and we don't use a something like a remote.FromBaseImage("windows-image") option. Without an option, imgutil (technically GGCR) defaults to linux/amd64.

pack build --publish has the same issue with manifest-list builders right now (on our current Windows WIP). It partially works since pack always does local.NewImage for the build container image so it would get the "windows" image (note the true here) but then lifecycle will then get the "linux" image because it uses remote.NewImage.

The fix could be to add a new option to remote.NewImage like remote.WithPlatform({OS:"windows"}) - a hacked version of which we had a while back - where lifecycle could pass in its runtime.GOOS.

But all this still doesn't need to block this issue's work. The images and manifest lists will hopefully be the same, even if we can't yet use them for all scenarios.

After discussion with @ameyer-pivotal , it feels like scenarios having manifests lists with mixed OSes won't be very common (if even possible), so we can disregard for now.

[micahyoung]

After some experimentation and discussion with @ameyer-pivotal, the easiest first iteration of lifecycle images that we could think of would be to use mcr.microsoft.com/windows/nanoserver:1809 as the base layer, put the lifecycle layer on top of that and publish that set as the lifecycle image.

Using that would guarantee the lifecycle image could be pulled to a daemon (1809 can be pulled to any Windows daemon, even newer ones), and since it's a foreign layer, the base layer's data would never actually "live" on the registry.

What would you all think about that approach?

[ekcasey]

@natalieparellano @micahyoung

After some experimentation and discussion with @ameyer-pivotal, the easiest first iteration of lifecycle images that we could think of would be to use mcr.microsoft.com/windows/nanoserver:1809 as the base layer, put the lifecycle layer on top of that and publish that set as the lifecycle image.

That makes sense to me. Unlike buildpackages the lifecycle images should be runnable so I think we do in fact need valid base image.

Because of the more complex labeling logic required by multi-api support I went ahead and implemented an image generating helper in #347 - https://github.com/buildpacks/lifecycle/pull/347/files#diff-42bf4cd03a39193c16208248a0c2e7d30 .I also changed the packaging logic (also b/c of the API changes).

The new tools/image/main.go helper doesn't do windows images yet but I am hoping it will be easy to add using the new archive helpers. We may need to use ggcr primitive to create a manifest list OR add that functionality to imgutil.

[natalieparellano]

Fork with working GHA for creating the Windows image: https://github.com/natalieparellano/lifecycle/runs/949910201?check_suite_focus=true

Now on to the manifest list part...

[natalieparellano]

Current approach using the docker cli to create the manifest list: https://github.com/natalieparellano/lifecycle/actions/runs/196468263

It created the images here: https://hub.docker.com/repository/docker/natalieparellano/lifecycle/tags?page=1

Tested them out on both platforms, they work :)

[ekcasey]

Latest images available at buildpacksio/lifecycle:0.9.0