Set vpa webhook reinvocationPolicy: IfNeeded

[voelzmo]

How to categorize this PR?

/area auto-scaling
/kind enhancement

What this PR does / why we need it:
Under normal circumstances, VPA is ignoring injected sidecar Containers. This is achieved by ensuring that the VPA admission-controller MutatingWebhook is called before all other webhooks which inject containers. The VPA admission-controller will add an annotation vpaObservedContainers with a list of all Containers found in a Pod and later on only apply recommendations to the Containers found in this annotation. This feature was introduced specifically to avoid endless eviction loops in the case where controllers injecting sidecar Containers are overwriting resources set by the VPA admission-controller.

In gardener, however, we need the VPA admission-controller to be called after other webhooks inject containers, as we're e.g. putting containers injected by the provider extensions under control of VPA.

In this scenario, problems can arise if the injecting webhook specifies reinvocationPolicy: IfNeeded and unconditionally reconciles the resources section of the injected container. This way, it will overwrite the resources set by VPA, resulting in an endless eviction loop. This became visible after we introduced the app-oidc-controller, which injects additional sidecar Containers and also adjusts their resources.

By setting vpa webhook reinvocationPolicy: IfNeeded for the VPA admission-controller webhook, we ensure that the VPA admission-controller is invoked last, making sure that the requests are set correctly also for injected Containers.

Release note:

Set `reinvocationPolicy: IfNeeded` for VPA admission-controller webhook to ensure that webhooks injecting sidecar containers will not trigger and endless eviction loop.

[ScheererJ]

Thanks for the pull request. Could you please share more details concerning which scenario/webhook required this change?

[voelzmo]

Thanks for the pull request. Could you please share more details concerning which scenario/webhook required this change?

adjusted the PR description to contain this additional sentence

This became visible after we introduced the app-oidc-controller, which injects additional sidecar Containers and also adjusts their resources.

[voelzmo]

/retest

[voelzmo]

/retest

[ScheererJ]

/lgtm

[gardener-prow]

LGTM label has been added.

Git tree hash: a517e99d8a7a8ec45fcb362a8431cd1409a6e7a8

[rfranzke]

/lgtm
/approve

[gardener-prow]

LGTM label has been added.

Git tree hash: 5d9d9a1c51f7fb7260a7ad4142f2d4f5beff55ac

[gardener-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rfranzke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rfranzke]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rfranzke]

/lgtm

[gardener-prow]

LGTM label has been added.

Git tree hash: b61caf2b8f721f7ae2130fcf880c49804231686c