-
Notifications
You must be signed in to change notification settings - Fork 366
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add scrips for migrating from other CNI to Antrea
Add scrtip and antctl subcommand "migrate" to migrate from other CNI(Calico, flannel) to Antrea. It also supports conditional NetworkPolicy conversion. Signed-off-by: hjiajing <[email protected]>
- Loading branch information
Showing
14 changed files
with
1,253 additions
and
8 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # Copyright 2022 Antrea Authors | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| set -e | ||
|
|
||
| SANDBOX_ID_ANNOTATION="io.kubernetes.cri.sandbox-id" | ||
| CONTAINERD_BINARIES_URL="https://github.com/containerd/containerd/releases/download/v1.7.7/containerd-1.7.7-linux-amd64.tar.gz" | ||
|
|
||
| # Remove rules of other CNI in CHAIN CNI-HOSTPORT-DNAT | ||
| iptables -t nat -F CNI-HOSTPORT-DNAT || true | ||
| chains=$(iptables -t nat -L | grep CNI-DN | grep -v "antrea" | awk '{print $2}') | ||
| for chain in $chains; do | ||
| iptables -t nat -X "$chain" || true | ||
| done | ||
|
|
||
| wget $CONTAINERD_BINARIES_URL | ||
| tar xvf containerd-1.7.7-linux-amd64.tar.gz | ||
| mv ./bin/ctr /usr/local/bin/ctr | ||
|
|
||
| pause_container_ids=$(ctr -n k8s.io containers ls | grep -v "CONTAINER" | grep "registry.k8s.io/pause" | awk '{print $1}') | ||
| # If the container's linux.namespaces fields contains "network", it means that the container uses CNI network. | ||
| # In order to switch CNI, we need to kill the corresponding tasks to restart the Pods. | ||
| for container_id in $pause_container_ids; do | ||
| container_info=$(ctr -n k8s.io containers info "$container_id" --spec) | ||
| sandbox_id=$(echo "$container_info" | jq -r .annotations.\"$SANDBOX_ID_ANNOTATION\") | ||
| namespaces=$(echo "$container_info" | jq .linux.namespaces | jq -c -r '.[]') | ||
| for namespace in $namespaces; do | ||
| if [[ "$(echo "$namespace" | jq .type)" == "\"network\"" ]]; then | ||
| echo "Container $container_id uses CNI network, kill the corresponding task" | ||
| ctr -n k8s.io tasks kill "$sandbox_id" || true | ||
| ctr -n k8s.io container remove "$container_id" || true | ||
| fi | ||
| done | ||
| done | ||
|
|
||
| # After restart, some containers may be dangling without a pause container, remove them | ||
| tasks=$(ctr -n k8s.io tasks list -q) | ||
| container_ids=$(ctr -n k8s.io containers ls | grep -v "CONTAINER" | grep -v "registry.k8s.io/pause" | awk '{print $1}') | ||
| for container_id in $container_ids; do | ||
| sandbox_id=$(ctr -n k8s.io containers info "$container_id" --spec | jq -r .annotations.\"$SANDBOX_ID_ANNOTATION\") | ||
| if [[ ! "$tasks" =~ $sandbox_id ]]; then | ||
| echo "Container $container_id has no corresponding task, remove it" | ||
| ctr -n k8s.io tasks kill "$container_id" || true | ||
| fi | ||
| done | ||
|
|
||
| rm /host/etc/cni/net.d/10-calico.conflist || true | ||
| rm /host/etc/cni/net.d/10-flannel.conflist || true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| kind: DaemonSet | ||
| apiVersion: apps/v1 | ||
| metadata: | ||
| labels: | ||
| app: antrea | ||
| component: antrea-migrator | ||
| name: antrea-migrator | ||
| namespace: kube-system | ||
| spec: | ||
| selector: | ||
| matchLabels: | ||
| app: antrea | ||
| component: antrea-migrator | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: antrea | ||
| component: antrea-migrator | ||
| spec: | ||
| hostPID: true | ||
| hostNetwork: true | ||
| nodeSelector: | ||
| kubernetes.io/os: linux | ||
| tolerations: | ||
| - key: CriticalAddonsOnly | ||
| operator: Exists | ||
| - effect: NoSchedule | ||
| operator: Exists | ||
| - effect: NoExecute | ||
| operator: Exists | ||
| serviceAccountName: antrea-agent | ||
| volumes: | ||
| - name: containerd | ||
| hostPath: | ||
| path: /run/containerd | ||
| initContainers: | ||
| # initContainer will kill all sandboxes container | ||
| - name: antrea-migrator-init | ||
| image: antrea/antrea-ubuntu:latest | ||
| imagePullPolicy: IfNotPresent | ||
| securityContext: | ||
| privileged: true | ||
| capabilities: | ||
| add: | ||
| # SYS_MODULE is required to load the OVS kernel module. | ||
| - SYS_MODULE | ||
| command: | ||
| - restart_sandbox | ||
| volumeMounts: | ||
| - mountPath: /run/containerd | ||
| name: containerd | ||
| containers: | ||
| # If the all antrea-migrator Pods are running, it means that all initContainers have been completed | ||
| - image: antrea/antrea-ubuntu:latest | ||
| imagePullPolicy: IfNotPresent | ||
| name: antrea-migrator | ||
| command: | ||
| - "sleep" | ||
| - "infinity" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| # Migrate from other CNI to Antrea | ||
|
|
||
| This document describes how to migrate from other CNIs to Antrea. Currently, we support migration from Calico and | ||
| Flannel. | ||
| As for Calico, NetworkPolicy conversion is conditional supported. As for Flannel, only Pod networking migration is | ||
| supported. | ||
|
|
||
| With the help of migration [script](../hack/migrate-to-antrea.sh) ant `antctl` subcommand `migrate`, the migration | ||
| process is fully automated. | ||
| The migration process is divided into four steps: | ||
|
|
||
| 1. Install Antrea in the cluster. | ||
| 2. If necessary, convert NetworkPolicy from other CNI to Antrea. | ||
| 3. Restart all Pods in the cluster in-place. | ||
| 4. Uninstall the old CNI. | ||
|
|
||
| ## Install Antrea | ||
|
|
||
| The script will check the requirement of CNI migration at first. If we are migrating from Calico, the script will list | ||
| all Calico NetworkPolicy to check if there is any unsupported feature. If we are migrating from Flannel, the script will | ||
| check if Antrea is installed in the cluster. If Antrea is already installed, the script will exit with an error message. | ||
|
|
||
| ## Convert NetworkPolicy | ||
|
|
||
| If the old CNI is Calico, the script will convert Calico NetworkPolicy to Antrea NetworkPolicy. The conversion is based | ||
| on | ||
| Calico NetworkPolicy CRD, so the script will check if the Calico APIServer is running in the cluster. If the Calico | ||
| APIServer | ||
| is not running, the script will exit with an error message. If the Calico APIServer is running, the script will convert | ||
| Calico | ||
| NetworkPolicy to Antrea NetworkPolicy using `antctl migrate convert-networkpolicy`. If the conversion fails, the script | ||
| will exit | ||
| with an error message. | ||
|
|
||
| Please Note: Not all Calico NetworkPolicy features are supported by Antrea NetworkPolicy. Please refer to the following | ||
| table for | ||
| the unsupported features: | ||
|
|
||
| | Calico NetworkPolicy Feature | Antrea NetworkPolicy Support | | ||
| |---------------------------------------------------------|------------------------------| | ||
| | "or" expression in any selector | NOT | | ||
| | "()" expression in any selector | NOT | | ||
| | "starts with" expression in any selector | NOT | | ||
| | "ends with" expression in any selector | NOT | | ||
| | `spec.PreDNAT` | NOT | | ||
| | `spec.ApplyOnForward` | NOT | | ||
| | `spec.DoNotTrack` | NOT | | ||
| | `spec.Ingress.ICMP` or `spec.Egress.ICMP` | NOT | | ||
| | `spec.Ingress.NotICMP` or `spec.Egress.NotICMP` | NOT | | ||
| | `spec.Ingress.NotProtocol` or `spec.Egress.NotProtocol` | NOT | | ||
| | `spec.Ingress.NotPorts` or `spec.Egress.NotPorts` | NOT | | ||
| | `spec.Ingress.Metadata` or `spec.Egress.Metadata` | NOT | | ||
| | `spec.Ingress.NotSelector` or `spec.Egress.NotSelector` | NOT | | ||
| | `spec.Ingress.HTTP` or `spec.Egress.HTTP` | NOT | | ||
| | `spec.Ingress.Nets` or `spec.Egress.Nets` | NOT | | ||
| | `spec.Ingress.NotNets` or `spec.Egress.NotNets` | NOT | | ||
| | All other features | YES | | ||
|
|
||
| ## Restart Pods | ||
|
|
||
| After Antrea is installed in the cluster, the script will restart all Pods in the cluster in-place by deploying a | ||
| DaemonSet named `antrea-migrator`, which will run a Pod on each Node. The Pod will kill all Pods' containerd task on the | ||
| Node, and the containerd task will be restarted by the containerd service. In this way, all Pods in the cluster will be | ||
| restarted | ||
| in-place and do not need to be rescheduled and recreated. | ||
|
|
||
| The restart result is as follows: | ||
|
|
||
| ```bash | ||
| $ kubectl get pod -n migrate-test -o wide | ||
| NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES | ||
| migrate-exmaple-6d6b97f96b-29qbq 1/1 Running 1 (24s ago) 2m5s 10.10.1.3 test-worker <none> <none> | ||
| migrate-exmaple-6d6b97f96b-dqx2g 1/1 Running 1 (23s ago) 2m5s 10.10.1.6 test-worker <none> <none> | ||
| migrate-exmaple-6d6b97f96b-jpflg 1/1 Running 1 (23s ago) 2m5s 10.10.1.5 test-worker <none> <none> | ||
| ``` | ||
|
|
||
| ## Uninstall old CNI | ||
|
|
||
| After all Pods are restarted, the script will uninstall the old CNI by using `kubectl delete -f <old-cni-yaml>`. If the | ||
| old CNI | ||
| is Calico, the script will also delete the Calico iptables rules on each Node by following commands: | ||
|
|
||
| ```bash | ||
| kubectl exec -n kube-system {ANTREA_AGENT} -- /bin/bash -c 'iptables-save | grep -v cali | iptables-restore' --kubeconfig $KUBECONFIG | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.