Module catalogue improvements

[janmedrek]

Description

Module catalogue (in this case - collection of Module Templates setup) should be adapted to the new requirements:

• A given module should be available in multiple different versions, users should be able to enable a specific version in their cluster. This requirement does not contradict the channel concept. It should still be possible to "subscribe" to a given channel (which will be one of the versions marked with that channel).
• Some module versions/channels use the same OCM descriptors, right now OCM part is embedded in the Module Template CR, which means there is no way to change/reference it across different resources. We should consider referencing that part instead of having that as a part of the Module Template spec. (We can use either K8S resources or OCM repo URLs)
• For the restricted markets there should be an option to configure the repository from which manifests are fetched (i.e. China market will use the China-restricted repository).
• For each module, there should be a list of "domain CRDs" that this module brings (i.e. Serverless will have Functions, Istio will have Gateways, Virtual Services...) that can be used to see which CRDs correspond to which module.

As for the implementation options, there are two main ones:

• We could continue using Module Templates and have them extended according to the requirements. This, however, may be problematic from the "multiple versions" point of view.
• Have a central "registry" of the modules, just like in the Community Modules repo. We could aggregate them in a single list (with all the fields that we require) and then use a Config Map or a Secret mounted to the KLM pod.

Reasons

We should adapt to the changing requirements and Community Modules concept.

Acceptance Criteria

• Describe the Module Catalogue
  • When a new Kyma Module is added to the catalogue
    • Then a new entry in Module Catalogue is created
      • And the Kyma Module entry contains all available versions of that module
      • And each version of that Kyma Module references a template for the OCM Descriptor URL
      • And the Kyma Module entry contains a list of all Module Domain CRDs managed by that Kyma Module
      • And Module Distribution Channels are assigned to specific versions of that Kyma Module
• Describe the Lifecycle Manager behaviour
  • When a new KCP Cluster is created
  • And a configuration for Manifest Repository is created
    • Then Lifecycle Manager uses Manifest Repository to fill in OCM Descriptor templates in Module Catalogue

Open Questions

• What to do with the restricted OCM repository authentication regarding fetching manifest/resources
• Take into consideration the following tickets which may influence overall API shape:
  • [Deletion Modes] Implement logic for "unmanaging" a Module #1427
  • Delete modes for modules #1211
  • Separate business and module statuses #1471

Related Tickets

• Decouple ModuleTemplates from Channels #1815
• Decouple Templates for Kyma- and Community Modules? #1816
• Phased Rollout for Support of Module Versions #1817
• Use explicit ModuleTemplate .spec.manager info to determine the module manager #1831
• [ModuleCatalog] Deprecate .spec.channel in ModuleTemplate #1839
• [ModuleCatalog] Add .spec.manager info to ModuleTemplate #1840
• [ModuleCatalog] Add .spec.info to ModuleTemplate #1841
• [ModuleCatalog] Add .spec.resources to ModuleTemplate #1842
• [ModuleCatalog] Add .spec.associatedResources to ModuleTemplate #1843
• Add Channel field to Manifest.Spec instead of the Channel label #1477
• Community Modules documentation #1812
• [ModuleCatalog] Support installation of specific module versions #1847
• [ModuleCatalog] Introduce ModuleReleaseMeta CRD #1848
• [ModuleCatalog] Implement Logic for ModuleReleaseMeta CRD in Kyma Reconcile Loop #1849
• [ModuleCatalog] Adapt Watch Mechanism to Monitor ModuleReleaseMeta CR Changes #1850
• [ModuleCatalog] Extend Catalog sync to handle both ModuleTemplate and ModuleReleaseMeta CRs #1851
• [ModuleCatalog] Support .spec.resources modulectl#53
• [ModuleCatalog] Support .spec.associatedResources modulectl#54
• [ModuleCatalog] Support .spec.info modulectl#55
• [ModuleCatalog] Support .spec.manager modulectl#57
• [ModuleCatalog] ModuleTemplate's metadata.name is <module-name>-<version> modulectl#62

[jeremyharisch]

Acceptance Criteria for EPIC:

• Research about the topic
• Think about a proper issue split for this EPIC
• Create Sub-Issues to cover the implementation
• Regarding business requirements talk to @janmedrek if needed

Timebox: 2 Days

Sub Issues:

• [Module catalogue improvements ] Extend Kyma CR module list with an explicit version #1589
• [Module catalogue improvements] Update ModuleTemplate API version to support fixed module versions #1590
• [Module catalogue improvements] Implement new Module Catalog support #1591
• Add ModuleTemplate reconciler #1599

[Tomasz-Smelcerz-SAP]

I think we should use K8s explicit types and avoid obscure solutions (like mounted ConfigMaps/Secrets) for the Module Catalogue.
That's why I propose to have the Module Catalogue defined as:
Module Catalogue is a collection of Module Templates in the cluster the Lifecycle-Manger is deployed.

[Tomasz-Smelcerz-SAP]

Requirement 1:

A given module should be available in multiple different versions, users should be able to enable a specific version in their cluster. This requirement does not contradict the channel concept. It should still be possible to "subscribe" to a given channel (which will be one of the versions marked with that channel).

Use case 1: User wants to enable a specific version of a module in their cluster.

[...]
- modules:
  - name: my-module
    version: 1.0.0
[...]

Use case 2: User wants to enable a specific channel of a module in their cluster.

[...]
- modules:
  - name: my-module
    channel: fast
[...]

Analysis:

This requirement implies there are many versions of the given module to choose from.
It is best modelled with many ModuleTemplate CRs - one for each version of the module.
It's better to have multiple ModuleTemplates (one for a version) instead of one "big" ModuleTemplate for all versions, because the ModuleTemplate should be immutable. The benefits of immutability is, for example, much easier caching of the data.
If we have a single ModuleTemplate CR for a given module that describes all of it's versions, then when a new version appears this ModuleTemplate CR must be edited (mutated). That's why it's better to have a single ModuleTemplate CR for a given version of a given module.

Another requirement is that versions may "belong to" a set of channels, so there is a mapping between versions and channels.
A version may be assigned to a (possibly empty) set of channels. It seems reasonable then to add this mapping to the ModuleTemplate CR.

API Changes:

• Kyma CR type: We need a new optional version attribute in the Module list. The channel attribute should be optional as well. User must provide either the version or the channel attribute.
• ModuleTemplate CR type:
  • version: Currently the module version is defined using an annotation: "operator.kyma-project.io/module-version". We need a new optional version attribute in the ModuleTemplate type. The reason for that is to make the version explicit in the ModuleTemplate type. In later API versions, this attribute should be mandatory.
  • channels: We also need to add the "channels" attribute in order to configure channels available for given module version.
  • name: for now the module name is just a label on the ModuleTemplate CR object. Now there is an opportunity to improve the API and make the name an explicit attribute of ModuleTemplate spec. This should be optional in the next API version, eventually it should be required.

Implementation details:

ModuleTemplate:

An example of current ModuleTemplate CR type (version is an annotation):

kind: ModuleTemplate
metadata:
  name: template-operator-regular
  namespace: kcp-system
  labels:
    "operator.kyma-project.io/module-name": "template-operator"
  annotations:
    "operator.kyma-project.io/doc-url": "https://kyma-project.io"
    "operator.kyma-project.io/is-cluster-scoped": "false"
    "operator.kyma-project.io/module-version": "0.1.2" 
spec:
  channel: regular
  mandatory: false 
  data:
  [...]
  descriptor:
  [...]

An example of the "new" ModuleTemplate CR type with the version attribute:

kind: ModuleTemplate
metadata:
  name: template-operator-0-1-2
  namespace: kcp-system
  labels:
    "operator.kyma-project.io/module-name": "template-operator"
  annotations:
    "operator.kyma-project.io/doc-url": "https://kyma-project.io"
    "operator.kyma-project.io/is-cluster-scoped": "false"
spec:
  name: "template-operator"
  channels: ["regular", "fast"]
  version: 0.1.2
  mandatory: false 
  data:
  [...]
  descriptor:
  [...]

In addition I would like to have the module name be an explicit attribute in the ModuleTemplate CR. The label may remain for easier searching, but the module name should be explicit.

[Tomasz-Smelcerz-SAP]

Modified channel/version design proposal:

At this moment we only allow users to select the channel.
After the change, the user will also be able to select the version.
for that to work, the version must refer to a unique ModuleTemplate.

I propose to make the following changes in the ModuleTemplate API:

• drop the "channel" attribute
• add the "channels" attribute
• add the "version" attribute

Example:

1. A module version defined in the "regular" channel

kind: ModuleTemplate
[...]
spec:
  channels: ["regular"]
  version: 0.1.1

2. A module version defined in two channels:

kind: ModuleTemplate
[...]
spec:
  channels: ["regular", "fast"]
  version: 0.1.2

3. A standalone module version (no channel):

kind: ModuleTemplate
[...]
spec:
  channels: []
  version: 0.1.3

Channels mutability issues:

What if the team releases a module version as standalone, and later wants to publish it in the fast channel? Similarly - what if the team initially publishes a module version in fast channel and later on decides to publish it also in the regular channel?
We have two options here:

1. Somehow we allow the channels attribute to be modified in the ModuleTemplate
2. The team must release a new version for that purpose, referring to the same OCM artifact(s)

I opt for option 2. as it simplifies module management - as ModuleTemplates are immutable, it's simpler to distribute/cache them.

[Tomasz-Smelcerz-SAP]

Requirement 4

For each module, there should be a list of "domain CRDs" that this module brings (i.e. Serverless will have Functions, Istio will have Gateways, Virtual Services...) that can be used to see which CRDs correspond to which module.

Option 1:
This can be done during the ModuleTemplate creation. We can extends modulectl create module command (not yet existing) to parse the resources in order to find the CRDs. In addition we may allow user to explicitly add some CRDs manually.

The modified ModuleTemplate CR example:

kind: ModuleTemplate
metadata:
  name: kyma-istio
  namespace: kcp-system 
spec:
  version: 1.2.3
  mandatory: false
  managedResources:
  - apiVersion: "networking.istio.io/v1beta1"
    kind: "Gateway"
  - apiVersion: "networking.istio.io/v1beta1"
    kind: "VirtualService"
  [...]

Option 2:
The managedResources list is just a part of the module and is stored as a separate layer in the module artifact. The advantage of this approach is that the managedResources is bound to the module definition at the module creation time, and it cannot be easily changed. This eliminates all sorts of errors related to invalid module configuration in the control-plane.

[Tomasz-Smelcerz-SAP]

Requirement 3

For the restricted markets there should be an option to configure the repository from which manifests are fetched (i.e. China market will use the China-restricted repository).

For now there is not enough information provided about the possible scenarios here. But it looks like we can implement this in the Lifecycle-Manager code, without affecting the API. One possible solution would be to add additional metadata to some ModuleTemplate CRs that would define the alternative repository URL. It could be an annotation.
The Lifecycle-Manager would then use this metadata instead of the original repository URLs (taken from OCM descriptor) to fetch the relevant manifests.

Note: If all the repository URLs are to be replaced with a single value, this can also be implemented as the CLI argument to the Lifecycle-Manager process. This would then affect ALL Modules in given environment. We can implement both options anyway.

[ruanxin]

Hi @pbochynski ,

We have some questions here, based on the current shipping kcp concept, we are versioning release channel, with the current approach, KLM is possible to use the latest channel version to get a list of latest module version.

However, in order to support the following request:

A given module should be available in multiple different versions, users should be able to enable a specific version in their cluster. This requirement does not contradict the channel concept. It should still be possible to "subscribe" to a given channel (which will be one of the versions marked with that channel).

How it aligned with the channel version then, to support multiple module versions, KLM must know all history versions of the channel first.
e.g:
channel/fast:1.0.0 contains istio:1.0.0
channel/fast:1.0.1 contains istio:1.1.0
to provide istio 1.0.0 and istio 1.1.0 to user in fast channel, KLM must know OCM component channel/fast have two versions(1.0.0, 1.0.1) exists. Which means KLM have to introduce a process to scan all channel/fast in OCM registry to build a reference between module and channel upfront. Is it necessary? But then, is it better we consider channel as a label attribute to kyma module?

Other questions:

1. Although here requested to support multiple versions, the version downgrade should still prevented, right?
2. Does this feature also means we allow user to always stay in an old version without upgrading?
3. What's the value behind this feature? consider the challenges may increase maintenance efforts and compatibility issue,
   • for example, an old istio version maybe not supported by other newer module which depends on it.
   • and allow user to freely choose different version, especially between major version, it might bring breaking change, especially during the CRD API upgrade. e.g: 1.0.0 contains v1alpha1, in 1.1.0 introduce v1alpha1, v1alpha2, in 1.2.0 deprecated v1alpha1, but this version should make sure handle the version upgrade properly, then in 1.3.0 only keep v1alpha2, if user directly jump from 1.0.0 to 1.3.0, it's not possible, API server will not allow apply the CRD without v1alpha1

[pbochynski]

You can have a look at my PoC where I describe all component versions available in all channels:
https://raw.githubusercontent.com/kyma-project/community-modules/main/model.json
I use channels as a version attribute there.
You do not need to know the history of the channel: you just apply the version for the channel or apply the version selected by the user. As in the current solution, you prevent downgrades based on semantic versioning. The only difference to the current logic is that if the current version is older than the oldest version listed in the shipment you automatically upgrade it (the user can't stay behind). This behavior is the same as what we have now with the regular channel (you can't stay behind regular channel also in the future)

The value of the feature is that you can decide about timing. You can still have a test cluster pinned to the fast channel, but your production channel you will upgrade when your SRE/dev team is available, and customers won't be affected. You can switch your production cluster to the version from the fast channel the next day after you successfully tested it on another cluster. So it is more control for the user. Of course, if you don't upgrade in 2 weeks (or whatever time we still support that version) you will be upgraded automatically.

[ruanxin]

After clarified with @pbochynski:

1. Mutiple version means user are free to subscribe module within channel or choose one version specifically.
2. Module team able to publish version to channel or optionally offer some module version which allow user to subscribe with.
3. Version downgrade is still not supported
4. During iteration, if the version subscribed by user is not managed by module team (not appears in channel or offered by module team specifically), this module becomes unmanaged.

[Tomasz-Smelcerz-SAP]

@pbochynski Hi Piotr,
I have a question related to "managedResources". In your POC this list is common to all module versions. Is there a chance that this list changes from version to version? I think it's a valid use case. What then? The team must release a new module, with a different name?

[c-pius]

Regarding the following statement from PB (see #1472 (comment)):

You do not need to know the history of the channel: you just apply the version for the channel or apply the version selected by the user. As in the current solution, you prevent downgrades based on semantic versioning. The only difference to the current logic is that if the current version is older than the oldest version listed in the shipment you automatically upgrade it (the user can't stay behind). This behavior is the same as what we have now with the regular channel (you can't stay behind regular channel also in the future)

Not exactly sure what "shipment" means, but I would interpret this as follows:

• support for a module version may be sunset at a given point in time ("the current version is older than the oldest version listed in the shipment")
• when it has been sunset, an automatic update is forced on the version

If this understanding is correct, we need to clarify:

• a) what version to update to? Newest one, oldest still supported one, ...
• b) how to display the forced update to the user (see [Module catalogue improvements] Update ModuleTemplate API version to support fixed module versions #1590 (comment))
• c) how to determine the module version is sunset (e.g., when no module template/version resource is found for a given version identifier)

EDIT: discussed the above with Xin. Basically, this is already described in point 4. here: #1472 (comment). If the user configured a channel, e.g., regular, the module version will be auto updated with the evolution in the channel. If the user configured a fixed version, and the support for this version is dropped, we are not auto updating it and the module becomes unmanaged.

[janmedrek]

TODO: Consider introducing module version in MT's spec field