Policy-controller failed to validate pods

[prudnitskiy]

Description

Any resource deployment in the GKE cluster with google KMS enabled failed to be validated. Validation ends with a stack trace (see below)

Short notes on setup: we use GKE with the workload identity enabled. We use google KMS to validate images. policy-webhook service account has permission for GKMS to read and use keys to verify image signatures.

Default behavor set to warn. Policy has been loaded successfully, no issues. Any validation request ends with the log below:

{"level":"info","ts":"2023-03-17T15:00:36.429Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000f3ddd0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"3928\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000de1700), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:3928, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.security.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.0.0.13:40854\", RequestURI:\"/mutations?timeout=10s\", TLS:(*tls.ConnectionState)(0xc000bdf3f0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc000de1740)}","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-17T15:00:37.121Z","logger":"policy-controller","caller":"defaulting/defaulting.go:158","msg":"Kind: \"/v1, Kind=Pod\" PatchBytes: [{\"op\":\"replace\",\"path\":\"/spec/containers/0/image\",\"value\":\"index.docker.io/library/alpine@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc001dc8517\"}]","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"user@domain.com"}
{"level":"info","ts":"2023-03-17T15:00:37.121Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"user@domain.com","admissionreview/uid":"d9dc9392-9575-4f56-8938-4072d231b7b5","admissionreview/allowed":true,"admissionreview/result":"nil"}
{"level":"info","ts":"2023-03-17T15:00:37.126Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc0000e6630), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"4121\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc0006a3ec0), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:4121, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.security.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.0.0.12:58300\", RequestURI:\"/validations?timeout=10s\", TLS:(*tls.ConnectionState)(0xc000bdf970), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc0006a3f00)}","commit":"89ef904-dirty"}
{"level":"error","ts":"2023-03-17T15:00:37.133Z","logger":"policy-controller","caller":"validation/validation_admit.go:180","msg":"Failed the resource specific validation","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"user@domain.com","stacktrace":"knative.dev/pkg/webhook/resourcesemantics/validation.validate\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/resourcesemantics/validation/validation_admit.go:180\nknative.dev/pkg/webhook/resourcesemantics/validation.(*reconciler).Admit\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/resourcesemantics/validation/validation_admit.go:79\nknative.dev/pkg/webhook.admissionHandler.func1\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/admission.go:123\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2109\nnet/http.(*ServeMux).ServeHTTP\n\tnet/http/server.go:2487\nknative.dev/pkg/webhook.(*Webhook).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/webhook.go:262\nknative.dev/pkg/network/handlers.(*Drainer).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/network/handlers/drain.go:113\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991"}
{"level":"info","ts":"2023-03-17T15:00:37.133Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"user@domain.com","admissionreview/uid":"fb77a23b-571c-4306-801f-fe014dd791b1","admissionreview/allowed":true,"admissionreview/result":"nil"}
{"level":"info","ts":"2023-03-17T15:00:37.226Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc0003cf5f0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"6930\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc0003e6300), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:6930, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.security.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.0.0.12:52796\", RequestURI:\"/mutations?timeout=10s\", TLS:(*tls.ConnectionState)(0xc0000b6c60), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc0003e6340)}","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-17T15:00:37.233Z","logger":"policy-controller","caller":"defaulting/defaulting.go:158","msg":"Kind: \"/v1, Kind=Pod\" PatchBytes: null","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"status","knative.dev/userinfo":"system:node:gke-cspservice-wasp-default-medium-b16a-4d91f919-ns74"}
{"level":"info","ts":"2023-03-17T15:00:37.233Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"status","knative.dev/userinfo":"system:node:gke-cspservice-wasp-default-medium-b16a-4d91f919-ns74","admissionreview/uid":"1f5abef9-63ef-4a97-b918-b6d2682a56df","admissionreview/allowed":true,"admissionreview/result":"nil"}
{"level":"info","ts":"2023-03-17T15:00:39.133Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc0004ecc60), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"8456\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc0007e0740), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:8456, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.security.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.0.0.12:52796\", RequestURI:\"/mutations?timeout=10s\", TLS:(*tls.ConnectionState)(0xc0000b6c60), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc0007e0780)}","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-17T15:00:39.141Z","logger":"policy-controller","caller":"defaulting/defaulting.go:158","msg":"Kind: \"/v1, Kind=Pod\" PatchBytes: null","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"status","knative.dev/userinfo":"system:node:gke-cspservice-wasp-default-medium-b16a-4d91f919-ns74"}
{"level":"info","ts":"2023-03-17T15:00:39.141Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"status","knative.dev/userinfo":"system:node:gke-cspservice-wasp-default-medium-b16a-4d91f919-ns74","admissionreview/uid":"979ac80c-8569-4c72-bc9a-5e01b6b7a573","admissionreview/allowed":true,"admissionreview/result":"nil"}

Version

policy-controller: 0.7.0
chart version: 0.5.2 -> 0.5.4
kubernetes version: 1.24.9-gke.3200

[hectorj2f]

@prudnitskiy Thanks for opening the issue. I am looking at your logs, but I cannot find any specific error related to the KMS or policy validation. Could you share the error you get when trying to create a pod that matches your policy (mode: enforce)? I think I'll need the rest of the logs to get enough information so I can understand what is the problem.

[prudnitskiy]

This log is for policy reject. Policy successfully rejected the request:

{"level":"info","ts":"2023-03-17T19:06:12.645Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000e9f440), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"3932\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000de5500), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:3932, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.security.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.0.3.13:54728\", RequestURI:\"/mutations?timeout=10s\", TLS:(*tls.ConnectionState)(0xc000d5cbb0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc000de5540)}","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-17T19:06:13.371Z","logger":"policy-controller","caller":"defaulting/defaulting.go:158","msg":"Kind: \"/v1, Kind=Pod\" PatchBytes: [{\"op\":\"replace\",\"path\":\"/spec/containers/0/image\",\"value\":\"index.docker.io/library/alpine@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc001dc8517\"}]","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"user@domain.com"}
{"level":"info","ts":"2023-03-17T19:06:13.371Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"user@domain.com","admissionreview/uid":"37f8a257-53c0-4d40-aaeb-8e88ebbd7126","admissionreview/allowed":true,"admissionreview/result":"nil"}
{"level":"info","ts":"2023-03-17T19:06:13.490Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000d57dd0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"4125\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc001398e00), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:4125, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.security.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.0.3.9:59594\", RequestURI:\"/validations?timeout=10s\", TLS:(*tls.ConnectionState)(0xc000d5d3f0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc0013999c0)}","commit":"89ef904-dirty"}
{"level":"error","ts":"2023-03-17T19:06:14.350Z","logger":"policy-controller","caller":"webhook/validation.go:47","msg":"error validating signatures: no matching signatures:\n","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"user@domain.com","stacktrace":"github.com/sigstore/policy-controller/pkg/webhook.valid\n\tgithub.com/sigstore/policy-controller/pkg/webhook/validation.go:47\ngithub.com/sigstore/policy-controller/pkg/webhook.ValidatePolicySignaturesForAuthority\n\tgithub.com/sigstore/policy-controller/pkg/webhook/validator.go:752\ngithub.com/sigstore/policy-controller/pkg/webhook.ValidatePolicy.func1\n\tgithub.com/sigstore/policy-controller/pkg/webhook/validator.go:529"}
{"level":"warn","ts":"2023-03-17T19:06:14.350Z","logger":"policy-controller","caller":"webhook/validator.go:1115","msg":"Failed to validate at least one policy for index.docker.io/library/alpine@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc001dc8517 wanted 1 policies, only validated 0","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"user@domain.com"}
{"level":"error","ts":"2023-03-17T19:06:14.350Z","logger":"policy-controller","caller":"validation/validation_admit.go:180","msg":"Failed the resource specific validation","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"user@domain.com","stacktrace":"knative.dev/pkg/webhook/resourcesemantics/validation.validate\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/resourcesemantics/validation/validation_admit.go:180\nknative.dev/pkg/webhook/resourcesemantics/validation.(*reconciler).Admit\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/resourcesemantics/validation/validation_admit.go:79\nknative.dev/pkg/webhook.admissionHandler.func1\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/admission.go:123\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2109\nnet/http.(*ServeMux).ServeHTTP\n\tnet/http/server.go:2487\nknative.dev/pkg/webhook.(*Webhook).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/webhook/webhook.go:262\nknative.dev/pkg/network/handlers.(*Drainer).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20221027143007-728dfd8e2862/network/handlers/drain.go:113\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991"}
{"level":"info","ts":"2023-03-17T19:06:14.350Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"89ef904-dirty","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"cspservice","knative.dev/name":"testimg","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"user@domain.com","admissionreview/uid":"58fe6f28-2251-406b-a1e6-73d48fd2dd7c","admissionreview/allowed":false,"admissionreview/result":"&Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:Failure,Message:validation failed: failed policy: evblocal: spec.containers[0].image\nindex.docker.io/library/alpine@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc001dc8517 signature key validation failed for authority authority-0 for index.docker.io/library/alpine@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc001dc8517: no matching signatures:\n,Reason:BadRequest,Details:nil,Code:400,}"}

The log message for warning mode looks pretty strange for me anyway

[hectorj2f]

@prudnitskiy This log output makes sense to me:

 signature key validation failed for authority authority-0 for index.docker.io/library/alpine@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc001dc8517: no matching signatures

I don't see any GKE KMS error fetching the key or anything else. I recommend you to try cosign verify ... against that image and your key to be sure everything is working fine.

You could also use the policy-tester component to test the policy against the image before creation.

[mathieu-benoit]

Out of curiosity @prudnitskiy, how did you configure your Workload Identity with Policy-controller KSAs?

FYI: I recently got this working with this setup:

gcloud container clusters create ${CLUSTER_NAME} \
    --workload-pool=${PROJECT_ID}.svc.id.goog \
    --zone ${ZONE} \
    --scopes "gke-default,https://www.googleapis.com/auth/cloudkms"

gcloud artifacts repositories create ${REGISTRY_NAME} \
    --repository-format docker \
    --location ${REGION}

PC_GSA_NAME=policy-controller-sa
gcloud iam service-accounts create ${PC_GSA_NAME}

# Enable workload identity for both policy-controller-policy-webhook and policy-controller-webhook KSAs:
gcloud iam service-accounts add-iam-policy-binding ${PC_GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:${PROJECT_ID}.svc.id.goog[cosign-system/policy-controller-policy-webhook]"
gcloud iam service-accounts add-iam-policy-binding ${PC_GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:${PROJECT_ID}.svc.id.goog[cosign-system/policy-controller-webhook]"

# Grant the associated GSA the 3 roles cloudkms.verifier, cloudkms.viewer and artifactregistry.reader:
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --role roles/cloudkms.verifier \
    --member serviceAccount:${PC_GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --role roles/cloudkms.viewer \
    --member serviceAccount:${PC_GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
gcloud artifacts repositories add-iam-policy-binding ${REGISTRY_NAME} \
    --location ${REGION} \
    --member "serviceAccount:${PC_GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role roles/artifactregistry.reader

# Annotate both policy-controller-policy-webhook and policy-controller-webhook KSAs:
helm upgrade policy-controller \
    --install \
    -n cosign-system sigstore/policy-controller \
    --create-namespace \
    --set "policywebhook.serviceAccount.annotations.iam\.gke\.io/gcp-service-account=${PC_GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
    --set "webhook.serviceAccount.annotations.iam\.gke\.io/gcp-service-account=${PC_GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"

cat << EOF | kubectl apply -f -
apiVersion: policy.sigstore.dev/v1alpha1
kind: ClusterImagePolicy
metadata:
  name: private-signed-images-cip
spec:
  images:
  - glob: "**"
  authorities:
  - key:
      kms: gcpkms://projects/${PROJECT_ID}/locations/${REGION}/keyRings/${KEY_RING}/cryptoKeys/${KEY_NAME}/cryptoKeyVersions/1
EOF

Curious to know what you are doing on your end, and if this above could help you.