Add the ability to force the location of the Seed to be in the same region as Kyma cluster [EPIC]

[TorstenD-SAP]

Description

The user who creates a Kyma cluster in the BTP cockpit should be able to enforce the location of the Control Plane to be in the same region as the Hyperscaler account where the Worker Nodes of the cluster are deployed. If it is not possible to have the Control Plane in the same region, the user should see an error message allowing him to proceed without this enforcement. In all cases it has to be transparent to the customer in which region the Control Plane is hosted.

Reasons

The region of the Control Plane is automatically chosen by Gardener (https://gardener.cloud/docs/gardener/concepts/scheduler/). Because of this the Control Plane could sometimes be deployed in a different region than the worker nodes, among others because Gardener doesn't have Seed clusters in all the regions Kyma can be deployed. This can lead to a violation of the law because the Control Plane could be in another legal area than the Worker Nodes and the customer is storing personal data (e. g. names, email addresses) on the Control Plane. We also have customers which are very sensitive regarding the regions where sensitive data is stored.

AC (Added by PK)

• Phase 1: do the full investigation where we need to put implementation efforts, areas: (Gophers estimation: Size/M)
  • 1.1) Check exactly what gardener can do about it
  • 1.2) BTP Cockpit. New input parameter/s in KEB schema for plans: aws, azure, gcp, azure_lite, sap-converged-cloud. Proper validation/error messages if user wants to have seed in the same region as Kyma cluster
  • 1.3) KEB implementation and contract with Provisioner
    • KEB
    • Provisioner (effort is time boxed to 1-3 man days): Extend Provisioner to support the configuration of seeds within the same region as the shoot has control-plane#3441
    • KIM: Support the configuration of seeds within the same region as the shoot infrastructure-manager#241
• Phase 2: Feature delivery
  • 2.1) KEB implementation + feature flag for dev, stage, prod
  • 2.2) Provisioner implementation
  • 2.3) Set KEB feature flag on DEV to true and write new SKR e2e integration test
  • 2.4) Register new schema on CIS - DEV
  • 2.5) Register new schema on CIS - STAGE
  • 2.6) Register new schema on CIS - PROD
  • 2.7) Update sap help sap portal, docs
  • 2.8) RN
  • 2.9) Synchronise timeouts between Provisioner and KEB

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

[TorstenD-SAP]

A label seed.gardener.cloud/region was added to each Gardener seed. This label can be used to restrict the seeds allowed for a shoot cluster by using the spec.seedSelector in the shoot spec.

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[tobiscr]

We agreed with @kyma-project/gopher to offer this feature under following constraints:

• We know that not all regions have yet their own seed, thats why we will show in the UI already a link to the documentation that this feature is not in all regions supported and can lead to failed Kyma clusters (because Gardener rejected the cluster creation)
• KIM and KEB will not check up-front if a seed exists in the requested regions and follow a "trail and error" approach: if Gardener could create the cluster all is fine otherwise the customer get's an error replied.

[ralikio]

I have tested seedSelector field mentioned in #18182 (comment). @PK85 suggested that for our test scenario we should select a shoot region that does not contain any seeds in it - ap-northeast-1. Just by creating a shoot with default configuration it got assigned a aws-ha-us2 region. Creation of another shoot with seedCluster set to ap-northeast-1 resulted in the following status:

*Status*
Create Pending

*Last Message*
Failed to schedule Shoot: none out of the ... seeds has the matching labels required by 
seed selector of 'Shoot' (selector: 'seed.gardener.cloud/region=ap-northeast-1')

Status: Create Pending seems counterintuitive to @kyma-project/gopher and @kyma-project/framefrog and will be consulted with Gardener Team.

[ralikio]

Proposed request sent to Provisioner's graphql API with new field shootAndSeedSameRegion:

{
	runtimeInput: {
		...	
	},
	clusterConfig:{
		gardenerConfig: {
			...	
			shootAndSeedSameRegion: false (default) | true,
		},
		...
	},
}

[ralikio]

Schema generated out of kyma-project/kyma-environment-broker#781: https://gist.github.com/ralikio/32138d957c9886a9e182494f39e6078d

[tobiscr]

JFYI - added a draft PR for Gardener to extract the Seed determining logic into separate struct to make it reusable for other apps over their API:

gardener/gardener#9843

[ralikio]

Two additional tests cases conducted regarding Gardener's spec.controlPlane.highAvailability.failureTolerance.type: zone and seedSelector. From the gardener documentation https://gardener.cloud/docs/gardener/high-availability/ we learn that:

Regarding the seed cluster selection, the only constraint is that shoot clusters with failure tolerance type zone are only allowed to run on seed clusters with at least three zones. All other shoot clusters (non-HA or those with failure tolerance type node) can run on seed clusters with any number of zones.

Case I - Creating a non-HA shoot on a region that only contains HA seeds - contains HA in its name

Provider: aws
Seed Selector: eu-north-1 - a region with two HA seeds
HA options: spec.controlPlane.highAvailability.failureTolerance.type: zone not set
Result: shoot gets created successfully.

Case II - Creating a HA shoot on a region that only contains non-HA seeds - no HA in its name

Provider: gcp
Seed Selector: europe-west-3 - a region with one non-HA seed
HA options: spec.controlPlane.highAvailability.failureTolerance.type: zone enabled
Result:

Create Pending - Failed to schedule Shoot: 0/1 seed cluster candidate(s) are eligible for scheduling: {*** => shoot does not tolerate the seed's taints}

Case III - Creating a HA shot in the region that contains one HA seed - contains HA in its name

Provider: gcp
Seed Selector: me-central2 - a region with one HA seed
HA options: spec.controlPlane.highAvailability.failureTolerance.type: zone enabled
Result:

Create Pending - Failed to schedule Shoot: 0/1 seed cluster candidate(s) are eligible for scheduling: {*** => shoot does not tolerate the seed's taints}

[ralikio]

Rendering of schema changes: