Drop nginx-ingress load balancer in favor of Istio

[ScheererJ]

How to categorize this PR?

/area networking
/kind enhancement
/kind api-change

What this PR does / why we need it:
This change moves nginx-ingress behind istio. The consequence is that the load balancer of nginx-ingress becomes an ordinary cluster service without direct exposure.

nginx-ingress is exposed via istio resources (Gateway, VirtualService & DestinationRule) similar as kube-apiserver is exposed.

TLS is still terminated at nginx-ingress. The same holds true for authentication of the observability components.

As a side effect of this change, this PR also brings an improvements to the Gardener Operator: It adds a new ingress.domains field to replace ingress.domain. This allows to expose Ingress resources via multiple domains.

Which issue(s) this PR fixes:
Fixes #7232

Special notes for your reviewer:
Please note that the forwarding currently only works for domain names using the wildcard ingress domain of seed/garden. Additional names need to be added to the istio configuration.

Release note:

Istio is now used as the single entry point on seed clusters. The load balancer of nginx-ingress is removed and traffic goes through istio before being handled by nginx if necessary.

A new field `.spec.runtimeCluster.ingress.domains` was added to the `Garden` API. This field allows to use multiple ingress domains for components of the runtime cluster. All domains are assumed to be wildcard domains. Earlier, the API only accepted one domain name via `.spec.runtimeCluster.ingress.domain`.
⚠️ With this change `.spec.runtimeCluster.ingress.domain` is deprecated and will be removed in the next release. Please update your `Garden` resource to the new `.spec.runtimeCluster.ingress.domains` field by removing the existing domain configuration from `ingress.domain` and add it as the first entry of `ingress.domains`.

[ScheererJ]

Error in pull-gardener-e2e-kind-migration-ha-single-zone seems unrelated => retrying

  [FAILED] Expected success, but got an error:
      <*fmt.wrapError | 0xc0005831a0>: 
      failed updating binding for shoot "garden-local/e2e-migrate": Operation cannot be fulfilled on shoots.core.gardener.cloud "e2e-migrate": the object has been modified; please apply your changes to the latest version and try again
      {
          msg: "failed updating binding for shoot \"garden-local/e2e-migrate\": Operation cannot be fulfilled on shoots.core.gardener.cloud \"e2e-migrate\": the object has been modified; please apply your changes to the latest version and try again",
          err: <*errors.StatusError | 0xc0005b8140>{

/test pull-gardener-e2e-kind-migration-ha-single-zone

[ScheererJ]

Issue in pull-gardener-e2e-kind-upgrade does not seem to be a general issue as local test succeeded => retrying

/test pull-gardener-e2e-kind-upgrade

[ScheererJ]

Will add support for multiple ingress domains to Garden.

/hold

[rfranzke]

/assign

[ScheererJ]

/cla

[gardener-prow]

Successfully reached out to cla-assistant.io to initialize recheck of PR #9038

[ScheererJ]

Rebased and adapted validation_test.go to fix unit test.

[rfranzke]

/lgtm
/approve

[gardener-prow]

LGTM label has been added.

Git tree hash: 710fa1c0e3b3dd4d98843ccd24509d3f67b50c62

[gardener-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rfranzke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rfranzke]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gardener-prow]

@ScheererJ: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gardener-apidiff	7f42bcd	link	false	/test pull-gardener-apidiff

Full PR test history. Your PR dashboard. Command help for this repository.
Please help us cut down on flakes by linking this test failure to an open flake report or filing a new flake report if you can't find an existing one. Also see our testing guideline for how to avoid and hunt flakes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[ScheererJ]

/retest-required

[ScheererJ]

/test pull-gardener-e2e-kind