[TODO]: CI - Assess if provenance: false is still necessary

[polarathene]

Description

SLSA (about page):

• Docker docs on Provenance Attestations.
• Attestation formats spec for SLSA Provenance.

A feature that provides additional insights / transparency into the image build process to establish more trust with users.

---

Earlier this year we had some CI issues with the docker/build-push-action upgrade (v3.3), that was partially affected by Github Actions not having upgraded their Docker package to a 2023 release (v23+) in their CI runner VMs.

Here is the relevant provenance section of that action today:

  if (await toolkit.buildx.versionSatisfies('>=0.10.0')) {
    if (inputs.provenance) {
      args.push('--provenance', inputs.provenance);
    } else if ((await toolkit.buildkit.versionSatisfies(inputs.builder, '>=0.11.0')) && !BuildxInputs.hasDockerExporter(inputs.outputs, inputs.load)) {
      // if provenance not specified and BuildKit version compatible for
      // attestation, set default provenance. Also needs to make sure user
      // doesn't want to explicitly load the image to docker.
      if (GitHub.context.payload.repository?.private ?? false) {
        // if this is a private repository, we set the default provenance
        // attributes being set in buildx: https://github.com/docker/buildx/blob/fb27e3f919dcbf614d7126b10c2bc2d0b1927eb6/build/build.go#L603
        args.push('--provenance', BuildxInputs.resolveProvenanceAttrs(`mode=min,inline-only=true`));
      } else {
        // for a public repository, we set max provenance mode.
        args.push('--provenance', BuildxInputs.resolveProvenanceAttrs(`mode=max`));
      }
    }

In particular the BuildxInputs.hasDockerExporter(inputs.outputs, inputs.load)) line is calling this method from docker/actions-toolkit/buildx/inputs.ts:

  public static hasDockerExporter(exporters: string[], load?: boolean): boolean {
    return load || Inputs.hasExporterType('docker', exporters);
  }

• This line was added later in Feb 2023 (part of v4), but the logic itself was roughly the same with an internal method from Jan 2023 (part of v3).
  • That v3 change had a follow-up PR to properly handle provenance input being false or unset instead of enabling the feature by default (this avoids a bug when the feature is not compatible with the BuildKit version AFAIK, but would otherwise enable implicitly if no explicit provenance: false was set).
  • Given that implicit behaviour, this was still carrying a breaking change in the v3 series (introduced in v3.3.0).
  • v3.3.1 release addressed this by reverting the breaking change via defaulting to provenance: false (even when the BuildKit version would support --provenance).

However v4 of docker/build-push-action restores the breaking change to use provenance: true as the default (introduced via this PR).

• As the current provenance logic snippet from the context.ts file displayed above shows, this remains the default in the current v5.
• Opt-out of --provenance must remain explicit after v3 of docker/build-push-action.
• Thus the question becomes: "Do we need to opt-out? What is the impact and pro/cons of either outcome going forward?".

---

It is noted that despite the problem in v3.3, the underlying buildx version (and technically buildkit too), is where this is introduced. Thus you'd still get the problem on deprecated v2 releases of docker/build-push-action which lack the provenance input to opt-out.

The buildx 0.10.0 release notes do have a disclaimer regarding SLSA provenance being dependent upon OCI-compliant images and a known compatibility issue with registries.

---

WIP

[polarathene]

References of known/historical concerns related to this feature:

Compute platforms that support containers:

• Some service providers AWS Lamda and GCR (Google Cloud Run) lack multi-arch support
  • Additionally notes the change between Docker manifest v2 to OCI v1. While this comment points out that the industry is moving towards OCI image spec, and it is fully backwards compatible with the earlier Docker v2 manifest (I'm probably getting some of the jargon here mixed up 😅 ).
  • Google Cloud Run had trouble processing the manifest list initially but was resolved in Feb 2023. The Google Cloud Registry however was noted as not resolved, with advice to use Google Artifact Registry instead, which appears to be the successor to their Cloud Registry offering.
  • Tracking issue: Default image output from buildx v0.10 cannot run on Google Cloud Run or AWS Lambda docker/buildx#1533

Registries:

• GHCR has also been observed to not handle provenance: true well, resulting in multiple separate images published instead with unexpected UX bug.
  • The UI bug displays an unknown/unknown architecture and fails to parse any metadata such as a description to display. A comparison of provenance: false vs provenance: true can be seen here (failure is assumed to be parsing an OCI index, or manifest attestations): Manifest not displayed in GHCR UI when using attestations docker/build-push-action#900
  • As of July 2023, this is still known to be an issue: Manifest not displayed in GHCR UI when using attestations docker/build-push-action#900 (comment)
• It's also been observed on DockerHub with single platform image push (provenance enabled implicitly, which is known to create a multi-platform index even for single platform build/push): docker manifest does not work with output from buildx v0.10 docker/buildx#1509 (comment)
• Similar for GCR (Google Container Registry, not Google Cloud Run) with null virtual size and creation timestamp set as the unix epoch (1st Jan 1970) for a manifest list (multi-arch index of manifests AFAIK? Related details).
  • User raised a separate issue for their report with more details. Link has a maintainer explain that the GCR web UI misinterprets the OCI image, which incorrectly displays multiple images instead of a single one.
  • Another GCR report notes 3 artifacts per push, with two having of the artifacts creation timestamp as unix epoch.
• When registries are incorrectly supporting OCI image / index, it can result in problems pulling the published image as the incorrect artifact is tagged. If you reference the correct one (which lacks the tag intended for it), it would pull successfully. For some registries the failure is a bit cryptic, stating the image could not be found despite being present in the registry.
• unknown/unknown knowingly represents the provenance attestation. Registries probably shouldn't be displaying it 🤷‍♂️

Docker CLI:

• docker manifest inspect is a known feature that would fail with the transition to OCI images.
  • The incompatibility was resolved for the Docker CLI with v23 of Docker (Feb 2023).
  • There is plans to merge the functionality with docker buildx imagetools.
  • This CLI feature/subcommand has been noted as not seeing active development - with advice encouraging to adopt docker buildx imagetools instead of docker manifest commands (Advice is from July 2023, thus the earlier resolved docker manifest inspect incompatibility may not be sufficient for some use-cases). The linked comment references multi-platform indexes being incompatible, which may influence the OCI docker manifest inspect compatibility improvement (as even single images built with provenance enabled creates a multi-platform manifest index).
  • Related is a merging support feature being implemented for docker manifest (eg: docker manifest create). This notes a concern with OCI indexes / manifest lists being iterated through with BuildKit attestations (used by provenance feature, but not limited to it AFAIK) as being the platform unknown/unknown which has been reported elsewhere as a UX concern.

Other projects:

• The popular Watchtower image (which our docs do have an example for) was also affected by this.
  • Support was cited as dependent upon docker manifest inspect being compatible](Error Could not do a head request for  containrrr/watchtower#1528 (comment)).
  • But later resolved by the actual logic of just updating their URL request Accept header to include the new OCI index mimetype: feat: add oci image index support containrrr/watchtower#1533 (released as part of v1.5.2)

Docker / Buildx / BuildKit releases:

• Buildx release 0.10.0 enables provenance by default (not just the docker/build-push-action) to build OCI-compliant images when BuildKit 0.11.0 or newer is present. From Docker v23, BuildKit was version bumped and the legacy docker image builder was swapped for the BuildKit builder backend as the new default (previously required DOCKER_BUILDKIT=1).
  • While Buildx is 0.10.x with Docker v23, BuildKit is also 0.10.x, thus no implicit provenance.
  • Docker v24 in May bumped BuildKit to 0.11.x where the condition for implicitly enabling provenance for builds with buildx becomes enabled.
  • In July, a patch version of Docker v24 bumps Buildx to the 0.11.x release series.

---

GHCR manifest example with unknown/unknown

docker manifest inspect ghcr.io/drwetter/testssl.sh:3.2

Outputs (one unknown/unknown entry listed per platform):

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 862,
         "digest": "sha256:619554481e9b41518277e294932b60b3390b8713ab6a638f5ef19e8f193379cc",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 862,
         "digest": "sha256:f89063c9b784240b4ed10ad459c3c89e43250ded3de9b06c7266dcabfe40e3ae",
         "platform": {
            "architecture": "386",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 862,
         "digest": "sha256:5bf2938aba604da399c0ff43acf3eeb29c9206d89a6e599dbb6f8e4505664968",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 862,
         "digest": "sha256:da694c491a7151f2f56a7f6bd021e1f2f5790b6cee4ff042cafe0a65bad56767",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 862,
         "digest": "sha256:76f44fab69aa7e80a97e310eecd0188818386a5824fe7ea23f14e60d4c79ec21",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v6"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 862,
         "digest": "sha256:0b9c637422b5f60d39f1240056ffb2452e2e9ba2c6037efec101b8a6598e5818",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:17c821e18a46d1aaf867e579d25a8891bf5d1a1532a530716dfdef7b36304667",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:f09d51401eb4d778b856e8d6cbb0fe9f25e9e13f4906f95ef126e4bbd05931e9",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:2be5fbbc48115d1bf3256f2b36e3fbdcad5de02f5a582024082acf0e450ae915",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:ef1808567f649b9d6c8021b36431ab97ae459a616749c5dee43c1597b0b3d5c3",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:b7ade340375c74877f80cd4c8a462e7965f892e4654fc11f6150bfbf98ac0d40",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:cdfdde55b992d4f2caa8c34309ea4ccae2bc9eb790d73d7d9058c8ea4d161c10",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}