Add the ability to force the location of the Seed to be in the same region as Kyma cluster [EPIC]

[TorstenD-SAP]

Description

The user who creates a Kyma cluster in the BTP cockpit should be able to enforce the location of the Control Plane to be in the same region as the Hyperscaler account where the Worker Nodes of the cluster are deployed. If it is not possible to have the Control Plane in the same region, the user should see an error message allowing him to proceed without this enforcement. In all cases it has to be transparent to the customer in which region the Control Plane is hosted.

Reasons

The region of the Control Plane is automatically chosen by Gardener (https://gardener.cloud/docs/gardener/concepts/scheduler/). Because of this the Control Plane could sometimes be deployed in a different region than the worker nodes, among others because Gardener doesn't have Seed clusters in all the regions Kyma can be deployed. This can lead to a violation of the law because the Control Plane could be in another legal area than the Worker Nodes and the customer is storing personal data (e. g. names, email addresses) on the Control Plane. We also have customers which are very sensitive regarding the regions where sensitive data is stored.

AC (Added by PK)

• Phase 1: do the full investigation where we need to put implementation efforts, areas: (Gophers estimation: Size/M)
  • 1.1) Check exactly what gardener can do about it
  • 1.2) BTP Cockpit. New input parameter/s in KEB schema for plans: aws, azure, gcp, azure_lite, sap-converged-cloud. Proper validation/error messages if user wants to have seed in the same region as Kyma cluster
  • 1.3) KEB implementation and contract with Provisioner
    • KEB
    • Provisioner (effort is time boxed to 1-3 man days): Extend Provisioner to support the configuration of seeds within the same region as the shoot has control-plane#3441, ShootAndSeedSameRegion Field control-plane#3440, Correction for #344 - shootAndSeedSameRegion in GardenerConfigInput control-plane#3442
    • KIM: Support the configuration of seeds within the same region as the shoot [KIM/feature] infrastructure-manager#241,
• Phase 2: Feature delivery
  • 2.1) KEB implementation + feature flag for dev, stage, prod: KEB Parameter to Enforce One Region for a Shoot and a Seed kyma-environment-broker#781, ShootAndSeedSameRegion - Provisioner & KEB kyma-environment-broker#804, ShootAndSeedSameRegion - Schema & Feature Flag kyma-environment-broker#796
    • 2.1.1) Provisioner Variant
    • 2.1.2) KIM Variant
  • 2.2) Provisioner implementation
  • 2.3) Set KEB feature flag on DEV to true and write new SKR e2e integration test
  • 2.4) Register new schema on CIS - DEV
  • 2.5) Register new schema on CIS - STAGE
  • 2.6) Register new schema on CIS - PROD
  • 2.7) Update sap help sap portal, docs
  • 2.8) RN
  • 2.9) Synchronise timeouts between Provisioner and KEB

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

[TorstenD-SAP]

A label seed.gardener.cloud/region was added to each Gardener seed. This label can be used to restrict the seeds allowed for a shoot cluster by using the spec.seedSelector in the shoot spec.

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[tobiscr]

We agreed with @kyma-project/gopher to offer this feature under following constraints:

• We know that not all regions have yet their own seed, thats why we will show in the UI already a link to the documentation that this feature is not in all regions supported and can lead to failed Kyma clusters (because Gardener rejected the cluster creation)
• KIM and KEB will not check up-front if a seed exists in the requested regions and follow a "trail and error" approach: if Gardener could create the cluster all is fine otherwise the customer get's an error replied.

[ralikio]

I have tested seedSelector field mentioned in #18182 (comment). @PK85 suggested that for our test scenario we should select a shoot region that does not contain any seeds in it - ap-northeast-1. Just by creating a shoot with default configuration it got assigned a aws-ha-us2 region. Creation of another shoot with seedCluster set to ap-northeast-1 resulted in the following status:

*Status*
Create Pending

*Last Message*
Failed to schedule Shoot: none out of the ... seeds has the matching labels required by 
seed selector of 'Shoot' (selector: 'seed.gardener.cloud/region=ap-northeast-1')

Status: Create Pending seems counterintuitive to @kyma-project/gopher and @kyma-project/framefrog and will be consulted with Gardener Team.

[ralikio]

Proposed request sent to Provisioner's graphql API with new field shootAndSeedSameRegion:

{
	runtimeInput: {
		...	
	},
	clusterConfig:{
		gardenerConfig: {
			...	
			shootAndSeedSameRegion: false (default) | true,
		},
		...
	},
}

[ralikio]

Schema generated out of kyma-project/kyma-environment-broker#781: https://gist.github.com/ralikio/32138d957c9886a9e182494f39e6078d

[tobiscr]

JFYI - added a draft PR for Gardener to extract the Seed determining logic into separate struct to make it reusable for other apps over their API:

gardener/gardener#9843

[ralikio]

No relevant, see #18182 (comment).

---

Two additional tests cases conducted regarding Gardener's spec.controlPlane.highAvailability.failureTolerance.type: zone and seedSelector. From the gardener documentation https://gardener.cloud/docs/gardener/high-availability/ we learn that:

Regarding the seed cluster selection, the only constraint is that shoot clusters with failure tolerance type zone are only allowed to run on seed clusters with at least three zones. All other shoot clusters (non-HA or those with failure tolerance type node) can run on seed clusters with any number of zones.

Case I - Creating a non-HA shoot on a region that only contains HA seeds - contains HA in its name

Provider: aws
Seed Selector: eu-north-1 - a region with two HA seeds
HA options: spec.controlPlane.highAvailability.failureTolerance.type: zone not set
Result: shoot gets created successfully.

Case II - Creating a HA shoot on a region that only contains non-HA seeds - no HA in its name

Provider: gcp
Seed Selector: europe-west-3 - a region with one non-HA seed
HA options: spec.controlPlane.highAvailability.failureTolerance.type: zone enabled
Result:

Create Pending - Failed to schedule Shoot: 0/1 seed cluster candidate(s) are eligible for scheduling: {*** => shoot does not tolerate the seed's taints}

Case III - Creating a HA shot in the region that contains one HA seed - contains HA in its name

Provider: gcp
Seed Selector: me-central2 - a region with one HA seed
HA options: spec.controlPlane.highAvailability.failureTolerance.type: zone enabled
Result:

Create Pending - Failed to schedule Shoot: 0/1 seed cluster candidate(s) are eligible for scheduling: {*** => shoot does not tolerate the seed's taints}

[ralikio]

Rendering of schema changes:

[ralikio]

Tests for seed selection process when provisioning shoots in high availability configuration (documented in #18182 (comment)) assumed that seeds that contained ha in their name (e.g. aws-ha-eu3) are specially designed to serve HA configuration. This is incorrect. Seeds with such names are just results of old naming conventions. All seeds with at least three zones are able to handle ha control plane deployments. Additionally, there is also visible property that restricts number of seeds available for scheduling. At time of writing the comment all of seeds were deployed across three zones.

[ralikio]

As of today we have implemented KEB part for Provisioner. @kyma-project/gopher are waiting for KIM implementation.

[tobiscr]

Appendix - some more background information related to this issue:

Customer reported bug
Slack Thread on #kyma-team
Slack Thread on #sap-tech-gardener-live

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[tobiscr]

@kyma-project/gopher : enabling "seed in region as shoot"-flag is only supported for new clusters - customers cannot enable it after the cluster exist - is this correct?

[tobiscr]

Feedback from @PK85 (via Slack):

Yes, only for new ones, cause as I remember seed cannot be updated
Maybe your team can double check that

@kyma-project/gopher : enabling "seed in region as shoot"-flag is only supported for new clusters - customers cannot enable it after the cluster exist - is this correct?