Provide tooling for automated Kyma lifecycle and subscription management

[varbanv]

Description

Users should be able to fully automate Kyma related tasks in their CI/CD pipelines. This starts with Kyma instance provisioning, includes deployment of necessary artifacts and test executions, and ends with Kyma instance removal and associated service instance cleanup. All of this should not require much additional effort from the end user and should be as easy as possible to configure.

Context

Problem

Right now, users can automate the Kyma instance creation using the new Terraform provider, the cloud orchestrator tools, or the btpcli directly, however, the next step of accessing the cluster requires user interaction.

And finally, there are a number of scenarios that could prevent a cluster from being deleted and would require user interaction to complete.
While the second and third problems can be solved with some coding and additional resources, we want to provide ease-of-use in order to improve the perception and adoption of Kyma.

Benefits

For customers:

• speed up development and release cycles
• implement best practices around CI/CD
• reduce configuration drift
• reduce dependency on long-living Kyma instances
• reduce cost by only using Kyma instances when needed
  Example usecase scenario that should be possible to run in automated way

For us:

• increase adoption
• enable customers to treat Kyma instances as disposable assets and reduce risk
• enable users to reduce cost

Proposed solution

Design and implement a new set of kyma CLI commands that helps developers who use managed kyma runtimes within BTP ecosystem with development of their CAP applications. CLI commands should be designed with automation flow in mind (No user context should be required),

The commands should cover:

• automated lifecycle management of kyma runtime lifecycle (provisioning, deprovisioning)
• getting access to freshly created cluster
• attaching BTP service instances to kyma runtimes via service manager
• eliminating manual actions (i.e mapping hana instance, establishing trust for custom IAS tenant, etc..)

Acceptance criteria

• Users can create a Kyma instance and access it without any user interaction.
  • Support additional OIDC configuration with shoot-oidc-service extension #18305
  • POC: Command to access btp kyma runtime with workflow OIDC token cli#2093
  • Allow passing json configuration when calling provision command cli#2115
• Users can completely remove an existing Kyma instance without any user interaction, even if there are modules "stuck" in deletion.
  • Set force-deletion flag when creating shoot-cluster [KIM/feature] infrastructure-manager#103 - KIM dependant
  • otherwise, this could be done via btp cli subaccount force deletionsubaccount
• Documentation and guides that cover the full end-to-end CI/CD scenario
  • https://help.sap.com/docs/btp/btp-developers-guide-internal/terraform-module-for-kyma?locale=en-US&state=DRAFT&version=Internal

Attachments

Idea for assisted kyma provisioning

[Disper]

What would be the exact/examples of scenarios around provisioning?

[kwiatekus]

Similar request to log into kyma in headless mode
https://github.tools.sap/kyma/backlog/issues/2518

[kwiatekus]

https://github.tools.sap/kyma/backlog/issues/2660#issuecomment-2104608

[pbochynski]

One possible solution is: #18305

[kwiatekus]

What would be the exact/examples of scenarios around provisioning?

@Disper
Additional config decribing a "system issuer" should be collected from user and sent to KEB. Further on, provisioner uses this data to deploy the OICD config object and enable the shoot-oidc-service extension.
https://github.tools.sap/kyma/backlog/issues/2660#issuecomment-3658944

Provisioner should add cluster role binding to principal represented by tokens issued by the system issuer

[kwiatekus]

Provisioning kyma runtime via BTP CLI
https://blogs.sap.com/2022/02/24/creating-sap-btp-kyma-runtime-via-the-sap-btp-cli/

Cloud orchestrator:
https://pages.github.tools.sap/cloud-orchestration/

[kwiatekus]

For users to be able to deploy their code w/o additional subscription (in the DEV mode) we aim to separate deocker-regisrty into separate DEV module to be used also outside of serverless usage
#18555 (comment)

[kwiatekus]

As of today, the new prototype commands (developed in https://github.com/kyma-project/cli/tree/v3) allow to

• provision managed kyma runtime within SAP BTP subaccount
• create hana db instance in an automated way (incl. binding and mapping to kyma).
• create references to shared instances

Getting access to new kyma runtime vi kyma CLI (in 100% automated scenario) will be possible after implementig #18305

[kwiatekus]

Customizable OIDC configurations (required for this epic) are being discussed within
kyma-project/infrastructure-manager#134

[kwiatekus]

We started working on cli part:

• kyma cli command that generates kubeconfig for CI/CD workflow
  POC: Command to access btp kyma runtime with workflow OIDC token cli#2093
• enhancing cli provisioning command to onclude extra oidc config
  Allow passing json configuration when calling provision command cli#2115 (dependency to KEB Extend Kyma Provisioning Parameters to allow multiple OIDC definitions kyma-environment-broker#423)
• kyma cli command that helps creating a serviceaccount based kubeconfig
  Produce token based kubeconfig originating from ServiceAccount via kyma cli cli#2036

ETA - on track

[kwiatekus]

Currently @tobiscr and the team is planning a migration from provisioner to KIM.
To start initial testing of the tooling (provisioning and getting access in automated scenario) it would be necessary to switch provisioning of new clusters via kyma infrastructure manager.

[tobiscr]

Plan is to have a working KIM version on DEV latest on end of July 2024

[kwiatekus]

We have synced with @tobiscr : The latest estimate for KIM delivery on prod is end of August 2024.

We could start testing the e2e provisioning (with additional oidc) via kyma cli on DEV only after necessary integration of KEB to KIM is implemented on DEV. @kyma-project/gopher PTAL
kyma-project/kyma-environment-broker#423 (comment)

[kwiatekus]

Status update 15.07.2024

• we have planned additional work related to building the kyma cli binary using SAP hyperspace - to ensure compliant binary. (@kyma-project/otters ). We will try out the template for pipelines designed for go code and hope 🤞 there will be no complications.
• the work on KIM side is on going (@kyma-project/framefrog )
• the work on KEB side (changing schema for provisioning params) was refined and planned (@kyma-project/gopher )

[kwiatekus]

Status update 29.07.2024

• @ptesny has came up with (and documented) a recipe how to use custom SAP IAS tenant for automated BTP platform resources creation (incl kyma runtime) and getting access to the newly provisioned cluster in automated scenario
• @otters are working on a template for CI pipeline for automated application integration tests that run on automatically created kyma runtime in btp environment
• @otters are aiming to use btp terraform provider for managing btp resources in creating test resources

[kwiatekus]

Status update 26.08.2024

@kyma-project/otters team continue working on the example repository that contains:

• *.tf (terraform manifests) for all necessary btp resources that need to be provisioned for an automated test scenario
• make targets for provisioning resources, building and deploying sample application, testing sample application
• definition for ADO pipeline (integrated with hyperspace tooling)

Once the pipeline is in place we want to document the sample repo as a working recipe for managing kyma lifecycle in the CI scenario. We estimate that we need extra week.

New delivery date: 06.09.2024

[kwiatekus]

Status update 03.09.2024

• a sample demonstration hyperspace pipeline running integration test of sample app is ready.
• (in progress) providing a dedicated reusable terraform module (not a kyma module!) - Extract kyma-on-btp terraform resources into its own reusable module terraform-module#1

New delivery date: 13.09.2024

[kwiatekus]

status update 09.09.2024

• demo hyperspace pipeline is using extracted terraform module
• polishing new terraform module :
  • stability and automated CI
  • usage examples

[kwiatekus]

Status update 04.10.2024

Working with technical writers on the documentation (as part of btp development guides)
https://help.sap.com/docs/btp/btp-developers-guide-internal/terraform-module-for-kyma?locale=en-US&state=DRAFT&version=Internal

[kwiatekus]

Status update 21.10.2024

• public documentation - simply documenting the usage of tf module
• internal documentation (in progress)

[kwiatekus]

Status update 04.11.2024

• added section about sap btp terraform provider into internal documentation
• documentation on help portal is in review