Can't use ratify with private ECR repository

[roeishuster]

What happened in your environment?

I tried to run ratify in a new EKS cluster, using the guide in this URL: https://ratify.dev/docs/quickstarts/ratify-with-aws-signer
I got to the final step of actually deploying a pod with an image, but I get an error from Ratify:
time=2024-05-15T13:35:40.271102033Z level=info msg=mutating image {myaccountid}.dkr.ecr.us-east-1.amazonaws.com/test/ratify/server:main-latest component-type=server go.version=go1.21.9 trace-id=73fe066c-4975-4b4e-a56d-f490890dd671 │
│ time=2024-05-15T13:35:40.271377639Z level=warning msg=auth provider failed with err, could not get ECR auth token for {myaccountid}.dkr.ecr.us-east-1.amazonaws.com/test/ratify/server:main-latest: could not retrieve ECR auth token collection: not found, ResolveEndpointV2 component-type=referrerStore go.version=go1.21.9 trace-id=73fe066c-4975-4b4e-a56d-f490890dd671

My helm install command:
helm install ratify
ratify/ratify --atomic
--namespace gatekeeper-system
--set-file notationCerts={./aws-signer-notation-root.cert}
--set featureFlags.RATIFY_EXPERIMENTAL_DYNAMIC_PLUGINS=true
--set serviceAccount.create=false
--set oras.authProviders.awsEcrBasicEnabled=true
--set featureFlags.RATIFY_CERT_ROTATION=true

What did you expect to happen?

Ratify should have checked if the image I provided is singed with notation.

What version of Kubernetes are you running?

1.29

What version of Ratify are you running?

1.1.1

Anything else you would like to add?

I used a private ECR repository in my AWS organization, I gave access both on Role side and ECR Repo side.

Are you willing to submit PRs to contribute to this bug fix?

• Yes, I am willing to implement it.

[FeynmanZhou]

@jlbutler Would you mind helping troubleshoot this issue with ECR? Thanks

[roeishuster]

@FeynmanZhou
Just so you know, this worked on chat version 1.11.0, and I tried to use notationCerts instead of notationCert and faced a different issue.
So the working configuration right now is chart version 1.11.0 with notationCert value.

[priteshbandi]

chart version-1.11.0
Ratify Version: v1.0.0

chart version-1.12.0 (latest)
Ratify Version: v1.1.0

Ratify v1.1.0-v1.0.0 diff: v1.0.0...v1.1.0
There were some changes in auth provierbut doesn't seems to be realted to ecr

The error seems to be thrown from

ratify/pkg/common/oras/authprovider/aws/awsecrbasic.go

Line 170 in aafd330

return provider.AuthConfig{}, errors.Wrapf(err, "could not get ECR auth token for %s", artifact)

[emalprokt]

I can back this, this is the exact error on our side too when trying to use a private ECR repo with the latest chart version.
I get the following series of logs/errors sequentially from ratify:

1. ecrAuthToken for xxxx.dkr.ecr.us-west-2.amazonaws.com does not exist
2. AWS ECR basic artifact=xxxx.dkr.ecr.us-west-2.amazonaws.com/image-name:latest, registry=xxxx.dkr.ecr.us-west-2.amazonaws.com, region=us-west-2
3. auth provider failed with err, could not get ECR auth token for xxxx.dkr.ecr.us-west-2.amazonaws.com/image-name:latest: could not retrieve ECR auth token collection: not found, ResolveEndpointV2
4. attempting to use anonymous credentials

My implementation is through the helm chart.
helm install ratify ratify/ratify --atomic -f ratify_values.yaml --namespace gatekeeper-system.

IAM access is given from both sides and proper service account is created and attached from the values.yaml file. I've double checked and triple checked to make sure nothing is missing while deploying this. I've followed the steps from the Ratify docs here and double checked the config matches mine too.

Potential Solution?

I don't know if this is related, however looking at the Ratify debug logs about not found, ResolveEndpointV2, a very similar issue has been raised in the aws-sdk-go-v2 repo here and this seems to be the solution.
`

[akashsinghal]

Thanks for the investigation @emalprokt. @priteshbandi would it be possible to send a PR to update the github.com/aws/aws-sdk-go-v2/service/ecr package to the latest version and the validate it on your end? Unfortunately, I don't have access to an AWS account to validate the fix.

Should just involve updating the package: go get -u github.com/aws/aws-sdk-go-v2/...

[susanshi]

PR for package upgrade was merged last week, the dev build is available here

@roeishuster, would you be able to help with validation? Many thanks!

Sample install command for dev build..

helm install ratify
oci://ghcr.io/deislabs/ratify-chart-dev/ratify --atomic
--version 0-dev
--namespace gatekeeper-system
--set featureFlags.RATIFY_CERT_ROTATION=true
--set otherfeaturesflags...

[emalprokt]

The ECR change seems to be working if deployed through its own helm chart as referred to by @susanshi

helm install ratify \
        oci://ghcr.io/deislabs/ratify-chart-dev/ratify --atomic \
        --version 0-dev \
        --namespace gatekeeper-system \
        --set featureFlags.RATIFY_CERT_ROTATION=true \
        --set serviceAccount.create=false \
        --set serviceAccount.name=ratify-admin \
        --set oras.authProviders.awsEcrBasicEnabled=true \
        --set sbom.enabled=true \
        --set sbom.disallowedLicenses={"MPL"} \
        --set 'sbom.disallowedPackages[0].name'="busybox" \
        --set 'sbom.disallowedPackages[0].version'="1.36.1-r0"

I created the service-account ratify-admin manually with proper ECR read IAM role attached.

If I try to deploy an image without an SBOM artifact attached, I get

│ time=2024-05-27T13:39:55.023395931Z level=info msg=verify result for subject xxx.dkr.ecr.us-west │
│   "verifierReports": [                                                                                    │
│     {                                                                                                     │
│       "subject": "xxx.dkr.ecr.us-west-2.amazonaws.com/docker.io/library/alpine@sha256:82d1e9d7ed │
│       "isSuccess": false,                                                                                 │
│       "message": "verification failed: Error: no verifier report, Code: NO_VERIFIER_REPORT, Component Typ │
│     }                                                                                                     │
│   ]                                                                                                       │
│ } component-type=server go.version=go1.21.10 namespace= trace-id=47dc3735-f801-43fa-9761-980cfeb1d892 

This means that authentication to ECR is successful.

Potential Pitfalls

Using the dev image with v1.1.1 chart does not work as there seem to be breaking changes between v1.1.1 and the latest changes. Best to update the whole helm chart to the dev's one.

Update: This dev image also works with the v1.2.0-rc.1 release. You'll just need to replace the image and tag values.