Verify only pods

[dszakallas]

Description

Make policy-controller configurable to only consider pods, not higher level resources.

Use case

We are using a continuous reconciliation (GitOps) solution, Flux, to maintain cluster state. The mutation of the image reference by the mutating admission webhook has undesired effects for us when used in conjunction with Flux:

• Jobs are immutable. It can happen that digest changes for the same tag between reconciliations, causing mutation to happen to the job, which is rejected by kube api server, as it targets an immutable field.
• even for deployments, resolving a digest every sync cycle causes a tight control that forces redeployment whenever the digest changes behind a tag, which we find undesirable.

Workarounds

While the scope of resources to validate can be controlled on the Policy level, this is not sufficient for our use, as mutations still get applied to all recognized resources and this seems to be hard coded. The only workaround I found was to fork the code and remove these.

Related work

Correct me if I am wrong, but I believe this change would not deteriorate the integrity of the cluster, since the pods owned by these unvalidated resources would still go through admission control, and their images will be verified. As an example of this practice, PSA (Pod Security Admission) only operates on the pod resource. Recently, Connaisseur also added a flag to opt out of higher level resource validation.