Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Explain some quirks with clusterCIDRIPv4 for Calico CNI #281

Open
moshiurH opened this issue Jan 6, 2025 · 0 comments · May be fixed by #284
Open

Explain some quirks with clusterCIDRIPv4 for Calico CNI #281

moshiurH opened this issue Jan 6, 2025 · 0 comments · May be fixed by #284
Assignees
@KoryKessel-Mirantis
Labels
Docs-B Should Do

Comments

@moshiurH
Copy link
Collaborator

moshiurH commented Jan 6, 2025

Testing for the Calico CIDR option (see sample yaml at https://docs.mirantis.com/mke-docs/docs/concepts/cni/#providers-configuration) was done. It was found that when configuring the Calico spec.network.providers.extraConfig.clusterCIDRIPv4 value, it is easily configurable before creating the cluster. However when trying to modify the clusterCIDRIPv4 value after the cluster has been created, it does not work. Based on Calico's official documentation, it sounds like or is implied that the clusterCIDRIPv4 can be modified via some sort of migration strategy (by creating a new IPPool - basically a rabbit hole).

For the sake of simplicity, maybe we should state clusterCIDRIPv4 can be configured before deployment. For modifying clusterCIDRIPv4 after cluster deployment, consult Mirantis Support.

Image

Some Test Notes:

Case 1 - Modify a default mke4.yaml's:

$ cat mke4.yaml
apiVersion: mke.mirantis.com/v1alpha1
kind: MkeConfig
...
    providers:
    - enabled: true
      extraConfig:
        CALICO_DISABLE_FILE_LOGGING: "true"
        CALICO_STARTUP_LOGLEVEL: DEBUG
        FELIX_LOGSEVERITYSCREEN: DEBUG
        clusterCIDRIPv4: 10.100.0.0/16 #originally 192.168.0.0/16
        enableWireguard: "false"
        ipAutodetectionMethod: ""
        mode: vxlan
        overlay: Always
        vxlanPort: "4789"
        vxlanVNI: "10000"
      provider: calico
...

Deploy with mkectl apply.

Confirm:

$ export KUBECONFIG=./mke.kubeconf
$ calicoctl get ippool -o wide --allow-version-mismatch
NAME                  CIDR            NAT    IPIPMODE   VXLANMODE   DISABLED   DISABLEBGPEXPORT   SELECTOR   
default-ipv4-ippool   10.100.0.0/16   true   Never      Always      false      false              all()      
$ kubectl get ippools.projectcalico.org default-ipv4-ippool -o yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  creationTimestamp: "2025-01-06T21:47:57Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: tigera-operator
  name: default-ipv4-ippool
  resourceVersion: "2037"
  uid: 48a009df-febe-4280-ab7c-111c93d34558
spec:
  allowedUses:
  - Workload
  - Tunnel
  blockSize: 26
  cidr: 10.100.0.0/16
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Always

Case 2 - Deploy a default mke4.yaml then reapply with new CIDR:

$ cat mke4.yaml
apiVersion: mke.mirantis.com/v1alpha1
kind: MkeConfig
...
    providers:
    - enabled: true
      extraConfig:
        CALICO_DISABLE_FILE_LOGGING: "true"
        CALICO_STARTUP_LOGLEVEL: DEBUG
        FELIX_LOGSEVERITYSCREEN: DEBUG
        clusterCIDRIPv4: 192.168.0.0/16
        enableWireguard: "false"
        ipAutodetectionMethod: ""
        mode: vxlan
        overlay: Always
        vxlanPort: "4789"
        vxlanVNI: "10000"
      provider: calico
...

Deploy with mkectl apply.

Change clusterCIDRIPv4: 192.168.0.0/16 to clusterCIDRIPv4: 10.100.0.0/16 then run mkectl apply again.

Verify if the CIDR has updated:

$ calicoctl get ippool -o wide --allow-version-mismatch
NAME                  CIDR             NAT    IPIPMODE   VXLANMODE   DISABLED   DISABLEBGPEXPORT   SELECTOR   
default-ipv4-ippool   192.168.0.0/16   true   Never      Always      false      false              all()      
$ kubectl get ippools.projectcalico.org default-ipv4-ippool -o yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  creationTimestamp: "2025-01-06T22:49:08Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: tigera-operator
  name: default-ipv4-ippool
  resourceVersion: "2006"
  uid: 9888d319-6949-405c-85db-98d4eaa78fa3
spec:
  allowedUses:
  - Workload
  - Tunnel
  blockSize: 26
  cidr: 192.168.0.0/16
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Always
$ mkectl config get
apiVersion: mke.mirantis.com/v1alpha1
kind: MkeConfig
...
    providers:
    - enabled: true
      extraConfig:
        CALICO_DISABLE_FILE_LOGGING: "true"
        CALICO_STARTUP_LOGLEVEL: DEBUG
        FELIX_LOGSEVERITYSCREEN: DEBUG
        clusterCIDRIPv4: 10.100.0.0/16
        enableWireguard: "false"
        ipAutodetectionMethod: ""
        mode: vxlan
        overlay: Always
        vxlanPort: "4789"
        vxlanVNI: "10000"
      provider: calico
    - enabled: false
      provider: custom

Notice how mkectl believe that the clusterCIDRIPv4 value is 10.100.0.0/16 but calicoctl still has it set to 192.168.0.0/16.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KoryKessel-Mirantis KoryKessel-Mirantis

Labels
Docs-B Should Do
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Insert configuration note for clusterCIDRIPv4 (CNI). Mirantis/mke-docs
2 participants
@KoryKessel-Mirantis @moshiurH