Upgrade Ceph CSI along with K8s upgrades

[nickatnceas]

https://github.com/ceph/ceph-csi/blob/devel/docs/ceph-csi-upgrade.md

[nickatnceas]

k8s-dev:

outin@Bluey:~$ helm list -A
NAME              	NAMESPACE      	REVISION	UPDATED                                	STATUS  	CHART                   	APP VERSION
...
ceph-csi-cephfs   	ceph-csi-cephfs	1       	2021-10-22 13:24:42.069667472 -0700 PDT	deployed	ceph-csi-cephfs-3.4.0   	v3.4.0
ceph-csi-rbd      	ceph-csi-rbd   	1       	2021-11-02 15:50:24.488735501 -0700 PDT	deployed	ceph-csi-rbd-3.4.0      	v3.4.0
...

[nickatnceas]

Looks like Ceph-CSI 3.8 to 3.11 are all tested options for K8s 1.26:

From https://github.com/ceph/ceph-csi/commits/devel/README.md

[nickatnceas]

We will need to do step upgrades, without skipping versions, according to the docs (ie 3.4 to 3.5 to 3.6 to 3.7 to 3.8):

Also, we do not recommend any direct upgrades to 3.5 except from 3.4 to 3.5. For example, upgrading from 3.3 to 3.5 is not recommended.

https://github.com/ceph/ceph-csi/blob/v3.5.1/docs/ceph-csi-upgrade.md

[nickatnceas]

ceph-csi snapshots will also need to be updated or reinstalled via Helm, they are not part of the current Helm ceph-csi install: #41

[nickatnceas]

Helm installation commands:

RBD: https://github.com/DataONEorg/k8s-cluster/blob/main/storage/Ceph/RBD/helm/helm-start.sh
CephFS: https://github.com/DataONEorg/k8s-cluster/blob/main/storage/Ceph/CephFS/helm/helm-start.sh

[nickatnceas]

Planned upgrade commands for k8s-dev:

For both:

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm repo update ceph-csi

RBD:
https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-rbd/README.md#upgrade-chart

helm upgrade --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd \
  --version 3.5.1 \
  --set rbac.create=true \
  --set serviceAccounts.nodeplugin.create=true \
  --set serviceAccounts.provisioner.create=true \
  --set configMapName=ceph-csi-config \
  --set externallyManagedConfigmap=true \
  --set nodeplugin.name=csi-cephrbdplugin \
  --set logLevel=5 \
  --set nodeplugin.registrar.image.tag="v2.2.0" \
  --set nodeplugin.plugin.image.tag="v3.5.1" \
  --set secret.create=false \
  --set secret.name=csi-cephrbd-secret \
  --set storageClass.create=false \
  --dry-run --debug

helm upgrade --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd \
  --version 3.6.2 \
  --set rbac.create=true \
  --set serviceAccounts.nodeplugin.create=true \
  --set serviceAccounts.provisioner.create=true \
  --set configMapName=ceph-csi-config \
  --set externallyManagedConfigmap=true \
  --set nodeplugin.name=csi-cephrbdplugin \
  --set logLevel=5 \
  --set nodeplugin.registrar.image.tag="v2.2.0" \
  --set nodeplugin.plugin.image.tag="v3.6.2" \
  --set secret.create=false \
  --set secret.name=csi-cephrbd-secret \
  --set storageClass.create=false \
  --dry-run --debug

helm upgrade --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd \
  --version 3.7.2 \
  --set rbac.create=true \
  --set serviceAccounts.nodeplugin.create=true \
  --set serviceAccounts.provisioner.create=true \
  --set configMapName=ceph-csi-config \
  --set externallyManagedConfigmap=true \
  --set nodeplugin.name=csi-cephrbdplugin \
  --set logLevel=5 \
  --set nodeplugin.registrar.image.tag="v2.5.1" \
  --set nodeplugin.plugin.image.tag="v3.7.2" \
  --set secret.create=false \
  --set secret.name=csi-cephrbd-secret \
  --set storageClass.create=false \
  --dry-run --debug

helm upgrade --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd \
  --version 3.8.1 \
  --set rbac.create=true \
  --set serviceAccounts.nodeplugin.create=true \
  --set serviceAccounts.provisioner.create=true \
  --set configMapName=ceph-csi-config \
  --set externallyManagedConfigmap=true \
  --set nodeplugin.name=csi-cephrbdplugin \
  --set logLevel=5 \
  --set nodeplugin.registrar.image.tag="v2.6.2" \
  --set nodeplugin.plugin.image.tag="v3.8.1" \
  --set secret.create=false \
  --set secret.name=csi-cephrbd-secret \
  --set storageClass.create=false \
  --dry-run --debug

CephFS:
https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-cephfs/README.md#upgrade-chart

helm upgrade --namespace ceph-csi-cephfs ceph-csi-cephfs ceph-csi/ceph-csi-cephfs \
  --version 3.5.1 \
  --set rbac.create=true \
  --set serviceAccounts.nodeplugin.create=true \
  --set serviceAccounts.provisioner.create=true \
  --set configMapName=ceph-csi-config \
  --set externallyManagedConfigmap=true \
  --set nodeplugin.name=csi-cephfsplugin \
  --set logLevel=5 \
  --set nodeplugin.registrar.image.tag="v2.2.0" \
  --set nodeplugin.plugin.image.tag="v3.5.1" \
  --set secret.create=false \
  --set secret.name=csi-cephfs-secret \
  --set storageClass.create=false \
  --dry-run --debug

helm upgrade --namespace ceph-csi-cephfs ceph-csi-cephfs ceph-csi/ceph-csi-cephfs \
  --version 3.6.2 \
  --set rbac.create=true \
  --set serviceAccounts.nodeplugin.create=true \
  --set serviceAccounts.provisioner.create=true \
  --set configMapName=ceph-csi-config \
  --set externallyManagedConfigmap=true \
  --set nodeplugin.name=csi-cephfsplugin \
  --set logLevel=5 \
  --set nodeplugin.registrar.image.tag="v2.2.0" \
  --set nodeplugin.plugin.image.tag="v3.6.2" \
  --set secret.create=false \
  --set secret.name=csi-cephfs-secret \
  --set storageClass.create=false \
  --dry-run --debug

helm upgrade --namespace ceph-csi-cephfs ceph-csi-cephfs ceph-csi/ceph-csi-cephfs \
  --version 3.7.2 \
  --set rbac.create=true \
  --set serviceAccounts.nodeplugin.create=true \
  --set serviceAccounts.provisioner.create=true \
  --set configMapName=ceph-csi-config \
  --set externallyManagedConfigmap=true \
  --set nodeplugin.name=csi-cephfsplugin \
  --set logLevel=5 \
  --set nodeplugin.registrar.image.tag="v2.5.1" \
  --set nodeplugin.plugin.image.tag="v3.7.2" \
  --set secret.create=false \
  --set secret.name=csi-cephfs-secret \
  --set storageClass.create=false \
  --dry-run --debug

helm upgrade --namespace ceph-csi-cephfs ceph-csi-cephfs ceph-csi/ceph-csi-cephfs \
  --version 3.8.1 \
  --set rbac.create=true \
  --set serviceAccounts.nodeplugin.create=true \
  --set serviceAccounts.provisioner.create=true \
  --set configMapName=ceph-csi-config \
  --set externallyManagedConfigmap=true \
  --set nodeplugin.name=csi-cephfsplugin \
  --set logLevel=5 \
  --set nodeplugin.registrar.image.tag="v2.6.2" \
  --set nodeplugin.plugin.image.tag="v3.8.1" \
  --set secret.create=false \
  --set secret.name=csi-cephfs-secret \
  --set storageClass.create=false \
  --dry-run --debug

Snapshots:
https://github.com/ceph/ceph-csi/blob/devel/docs/snap-clone.md

wget https://github.com/ceph/ceph-csi/archive/refs/tags/v3.8.1.tar.gz
tar xvfz v3.8.1.tar.gz
./scripts/install-snapshot.sh install

Snapshot version appears to be the same: SNAPSHOT_VERSION=${SNAPSHOT_VERSION:-"v5.0.1"}, so we can skip this step.

[nickatnceas]

Preparing for the upgrade with steps taken from #24 (comment)

outin@Bluey:~/helm$ helm history --namespace ceph-csi-rbd ceph-csi-rbd
REVISION	UPDATED                 	STATUS  	CHART             	APP VERSION	DESCRIPTION
1       	Tue Nov  2 15:50:24 2021	deployed	ceph-csi-rbd-3.4.0	v3.4.0     	Install complete

outin@Bluey:~/helm$ helm get manifest --namespace ceph-csi-rbd ceph-csi-rbd > ceph-csi-rbd_3.4.0.yaml

outin@Bluey:~/helm$ helm history --namespace ceph-csi-cephfs ceph-csi-cephfs
REVISION	UPDATED                 	STATUS  	CHART                	APP VERSION	DESCRIPTION
1       	Fri Oct 22 13:24:42 2021	deployed	ceph-csi-cephfs-3.4.0	v3.4.0     	Install complete

outin@Bluey:~/helm$ helm get manifest --namespace ceph-csi-cephfs ceph-csi-cephfs > ceph-csi-cephfs_3.4.0.yaml

[nickatnceas]

If something goes wrong with the update, we can attempt a helm rollback, or use the yaml files exported above.

helm rollback --namespace ceph-csi-rbd ceph-csi-rbd 3.4.0
helm rollback --namespace ceph-csi-cephfs ceph-csi-cephfs 3.4.0