Merge pull request #224 from red-hat-storage/sync_us--devel

Syncing latest changes from upstream devel for ceph-csi

PendingReleaseNotes.md

@@ -13,3 +13,8 @@
   ConfigMap in [PR](https://github.com/ceph/ceph-csi/pull/4165)
 
 - Support for CephFS kernel and fuse mount options per cluster in [PR](https://github.com/ceph/ceph-csi/pull/4245)
+
+## NOTE
+
+- Support is limited to only the active Ceph releases. Support for EOLed Ceph
+  releases are removed [PR](https://github.com/ceph/ceph-csi/pull/4262)

README.md

@@ -81,39 +81,39 @@ for its support details.
 
 | Plugin | Features                                                  | Feature Status | CSI Driver Version | CSI Spec Version | Ceph Cluster Version | Kubernetes Version |
 | ------ | --------------------------------------------------------- | -------------- | ------------------ | ---------------- | -------------------- | ------------------ |
-| RBD    | Dynamically provision, de-provision Block mode RWO volume | GA             | >= v1.0.0          | >= v1.0.0        | Octopus (>=15.0.0)  | >= v1.14.0         |
-|        | Dynamically provision, de-provision Block mode RWX volume | GA             | >= v1.0.0          | >= v1.0.0        | Octopus (>=15.0.0)  | >= v1.14.0         |
-|        | Dynamically provision, de-provision Block mode RWOP volume| Alpha          | >= v3.5.0          | >= v1.5.0        | Octopus (>=15.0.0)  | >= v1.22.0         |
-|        | Dynamically provision, de-provision File mode RWO volume  | GA             | >= v1.0.0          | >= v1.0.0        | Octopus (>=15.0.0)  | >= v1.14.0         |
-|        | Dynamically provision, de-provision File mode RWOP volume | Alpha          | >= v3.5.0          | >= v1.5.0        | Octopus (>=15.0.0)  | >= v1.22.0         |
-|        | Provision File Mode ROX volume from snapshot              | Alpha          | >= v3.0.0          | >= v1.0.0        | Octopus (>=v15.0.0) | >= v1.17.0         |
-|        | Provision File Mode ROX volume from another volume        | Alpha          | >= v3.0.0          | >= v1.0.0        | Octopus (>=v15.0.0) | >= v1.16.0         |
-|        | Provision Block Mode ROX volume from snapshot             | Alpha          | >= v3.0.0          | >= v1.0.0        | Octopus (>=v15.0.0) | >= v1.17.0         |
-|        | Provision Block Mode ROX volume from another volume       | Alpha          | >= v3.0.0          | >= v1.0.0        | Octopus (>=v15.0.0) | >= v1.16.0         |
-|        | Creating and deleting snapshot                            | GA             | >= v1.0.0          | >= v1.0.0        | Octopus (>=15.0.0)  | >= v1.17.0         |
-|        | Provision volume from snapshot                            | GA             | >= v1.0.0          | >= v1.0.0        | Octopus (>=15.0.0)  | >= v1.17.0         |
-|        | Provision volume from another volume                      | GA             | >= v1.0.0          | >= v1.0.0        | Octopus (>=15.0.0)  | >= v1.16.0         |
-|        | Expand volume                                             | Beta           | >= v2.0.0          | >= v1.1.0        | Octopus (>=15.0.0)  | >= v1.15.0         |
-|        | Volume/PV Metrics of File Mode Volume                     | GA             | >= v1.2.0          | >= v1.1.0        | Octopus (>=15.0.0)  | >= v1.15.0         |
-|        | Volume/PV Metrics of Block Mode Volume                    | GA             | >= v1.2.0          | >= v1.1.0        | Octopus (>=15.0.0)  | >= v1.21.0         |
-|        | Topology Aware Provisioning Support                       | Alpha          | >= v2.1.0          | >= v1.1.0        | Octopus (>=15.0.0)  | >= v1.14.0         |
-| CephFS | Dynamically provision, de-provision File mode RWO volume  | GA             | >= v1.1.0          | >= v1.0.0        | Octopus (>=15.0.0)  | >= v1.14.0         |
-|        | Dynamically provision, de-provision File mode RWX volume  | GA             | >= v1.1.0          | >= v1.0.0        | Octopus (>=v15.0.0) | >= v1.14.0         |
-|        | Dynamically provision, de-provision File mode ROX volume  | Alpha          | >= v3.0.0          | >= v1.0.0        | Octopus (>=v15.0.0) | >= v1.14.0         |
-|        | Dynamically provision, de-provision File mode RWOP volume | Alpha          | >= v3.5.0          | >= v1.5.0        | Octopus (>=15.0.0)  | >= v1.22.0         |
-|        | Creating and deleting snapshot                            | GA             | >= v3.1.0          | >= v1.0.0        | Octopus (>=v15.2.4)  | >= v1.17.0         |
-|        | Provision volume from snapshot                            | GA             | >= v3.1.0          | >= v1.0.0        | Octopus (>=v15.2.4)  | >= v1.17.0         |
-|        | Provision volume from another volume                      | GA             | >= v3.1.0          | >= v1.0.0        | Octopus (>=v15.2.4)  | >= v1.16.0         |
-|        | Expand volume                                             | Beta           | >= v2.0.0          | >= v1.1.0        | Octopus (>=v15.0.0) | >= v1.15.0         |
-|        | Volume/PV Metrics of File Mode Volume                     | GA             | >= v1.2.0          | >= v1.1.0        | Octopus (>=v15.0.0) | >= v1.15.0         |
-| NFS    | Dynamically provision, de-provision File mode RWO volume  | Alpha          | >= v3.6.0          | >= v1.0.0        | Pacific (>=16.2.0)   | >= v1.14.0         |
-|        | Dynamically provision, de-provision File mode RWX volume  | Alpha          | >= v3.6.0          | >= v1.0.0        | Pacific (>=16.2.0)   | >= v1.14.0         |
-|        | Dynamically provision, de-provision File mode ROX volume  | Alpha          | >= v3.6.0          | >= v1.0.0        | Pacific (>=16.2.0)   | >= v1.14.0         |
-|        | Dynamically provision, de-provision File mode RWOP volume | Alpha          | >= v3.6.0          | >= v1.5.0        | Pacific (>=16.2.0)   | >= v1.22.0         |
-|        | Expand volume                                             | Alpha          | >= v3.7.0          | >= v1.1.0        | Pacific (>=16.2.0)   | >= v1.15.0         |
-|        | Creating and deleting snapshot                            | Alpha          | >= v3.7.0          | >= v1.1.0        | Pacific (>=16.2.0)   | >= v1.17.0         |
-|        | Provision volume from snapshot                            | Alpha          | >= v3.7.0          | >= v1.1.0        | Pacific (>=16.2.0)   | >= v1.17.0         |
-|        | Provision volume from another volume                      | Alpha          | >= v3.7.0          | >= v1.1.0        | Pacific (>=16.2.0)   | >= v1.16.0         |
+| RBD    | Dynamically provision, de-provision Block mode RWO volume | GA             | >= v1.0.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.14.0         |
+|        | Dynamically provision, de-provision Block mode RWX volume | GA             | >= v1.0.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.14.0         |
+|        | Dynamically provision, de-provision Block mode RWOP volume| Alpha          | >= v3.5.0          | >= v1.5.0        | Pacific (>=v16.2.0)  | >= v1.22.0         |
+|        | Dynamically provision, de-provision File mode RWO volume  | GA             | >= v1.0.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.14.0         |
+|        | Dynamically provision, de-provision File mode RWOP volume | Alpha          | >= v3.5.0          | >= v1.5.0        | Pacific (>=v16.2.0)  | >= v1.22.0         |
+|        | Provision File Mode ROX volume from snapshot              | Alpha          | >= v3.0.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.17.0         |
+|        | Provision File Mode ROX volume from another volume        | Alpha          | >= v3.0.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.16.0         |
+|        | Provision Block Mode ROX volume from snapshot             | Alpha          | >= v3.0.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.17.0         |
+|        | Provision Block Mode ROX volume from another volume       | Alpha          | >= v3.0.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.16.0         |
+|        | Creating and deleting snapshot                            | GA             | >= v1.0.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.17.0         |
+|        | Provision volume from snapshot                            | GA             | >= v1.0.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.17.0         |
+|        | Provision volume from another volume                      | GA             | >= v1.0.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.16.0         |
+|        | Expand volume                                             | Beta           | >= v2.0.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.15.0         |
+|        | Volume/PV Metrics of File Mode Volume                     | GA             | >= v1.2.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.15.0         |
+|        | Volume/PV Metrics of Block Mode Volume                    | GA             | >= v1.2.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.21.0         |
+|        | Topology Aware Provisioning Support                       | Alpha          | >= v2.1.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.14.0         |
+| CephFS | Dynamically provision, de-provision File mode RWO volume  | GA             | >= v1.1.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.14.0         |
+|        | Dynamically provision, de-provision File mode RWX volume  | GA             | >= v1.1.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.14.0         |
+|        | Dynamically provision, de-provision File mode ROX volume  | Alpha          | >= v3.0.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.14.0         |
+|        | Dynamically provision, de-provision File mode RWOP volume | Alpha          | >= v3.5.0          | >= v1.5.0        | Pacific (>=v16.2.0)  | >= v1.22.0         |
+|        | Creating and deleting snapshot                            | GA             | >= v3.1.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.17.0         |
+|        | Provision volume from snapshot                            | GA             | >= v3.1.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.17.0         |
+|        | Provision volume from another volume                      | GA             | >= v3.1.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.16.0         |
+|        | Expand volume                                             | Beta           | >= v2.0.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.15.0         |
+|        | Volume/PV Metrics of File Mode Volume                     | GA             | >= v1.2.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.15.0         |
+| NFS    | Dynamically provision, de-provision File mode RWO volume  | Alpha          | >= v3.6.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.14.0         |
+|        | Dynamically provision, de-provision File mode RWX volume  | Alpha          | >= v3.6.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.14.0         |
+|        | Dynamically provision, de-provision File mode ROX volume  | Alpha          | >= v3.6.0          | >= v1.0.0        | Pacific (>=v16.2.0)  | >= v1.14.0         |
+|        | Dynamically provision, de-provision File mode RWOP volume | Alpha          | >= v3.6.0          | >= v1.5.0        | Pacific (>=v16.2.0)  | >= v1.22.0         |
+|        | Expand volume                                             | Alpha          | >= v3.7.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.15.0         |
+|        | Creating and deleting snapshot                            | Alpha          | >= v3.7.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.17.0         |
+|        | Provision volume from snapshot                            | Alpha          | >= v3.7.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.17.0         |
+|        | Provision volume from another volume                      | Alpha          | >= v3.7.0          | >= v1.1.0        | Pacific (>=v16.2.0)  | >= v1.16.0         |
 
 `NOTE`: The `Alpha` status reflects possible non-backward
 compatible changes in the future, and is thus not recommended

api/deploy/kubernetes/cephfs/csi-config-map.go

@@ -22,6 +22,7 @@ import (
 	"fmt"
 	"text/template"
 
+	"github.com/ceph/ceph-csi/api/deploy/kubernetes"
 	"github.com/ghodss/yaml"
 	v1 "k8s.io/api/core/v1"
 )
@@ -30,11 +31,13 @@ import (
 var csiConfigMap string
 
 type CSIConfigMapValues struct {
-	Name string
+	Name        string
+	ClusterInfo []kubernetes.ClusterInfo
 }
 
 var CSIConfigMapDefaults = CSIConfigMapValues{
-	Name: "ceph-csi-config",
+	Name:        "ceph-csi-config",
+	ClusterInfo: []kubernetes.ClusterInfo{},
 }
 
 // NewCSIConfigMap takes a name from the CSIConfigMapValues struct and relaces

api/deploy/kubernetes/cephfs/csi-config-map.yaml

@@ -5,4 +5,4 @@ metadata:
   name: "{{ .Name }}"
 data:
   config.json: |-
-    []
+    {{ .ClusterInfo }}

api/deploy/kubernetes/cephfs/csidriver.yaml

@@ -7,3 +7,4 @@ spec:
   attachRequired: false
   podInfoOnMount: false
   fsGroupPolicy: File
+  seLinuxMount: true

api/deploy/kubernetes/csi-config-map.go

@@ -0,0 +1,59 @@
+/*
+Copyright 2023 The Ceph-CSI Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package kubernetes
+
+type ClusterInfo struct {
+	// ClusterID is used for unique identification
+	ClusterID string
+	// Monitors is monitor list for corresponding cluster ID
+	Monitors []string
+	// CephFS contains CephFS specific options
+	CephFS CephFS
+	// RBD Contains RBD specific options
+	RBD RBD
+	// NFS contains NFS specific options
+	NFS NFS
+	// Read affinity map options
+	ReadAffinity ReadAffinity
+}
+
+type CephFS struct {
+	// symlink filepath for the network namespace where we need to execute commands.
+	NetNamespaceFilePath string
+	// SubvolumeGroup contains the name of the SubvolumeGroup for CSI volumes
+	SubvolumeGroup string
+	// KernelMountOptions contains the kernel mount options for CephFS volumes
+	KernelMountOptions string
+	// FuseMountOptions contains the fuse mount options for CephFS volumes
+	FuseMountOptions string
+}
+type RBD struct {
+	// symlink filepath for the network namespace where we need to execute commands.
+	NetNamespaceFilePath string
+	// RadosNamespace is a rados namespace in the pool
+	RadosNamespace string
+}
+
+type NFS struct {
+	// symlink filepath for the network namespace where we need to execute commands.
+	NetNamespaceFilePath string
+}
+
+type ReadAffinity struct {
+	Enabled             bool
+	CrushLocationLabels []string
+}

api/deploy/kubernetes/nfs/csi-config-map.go

@@ -22,6 +22,7 @@ import (
 	"fmt"
 	"text/template"
 
+	"github.com/ceph/ceph-csi/api/deploy/kubernetes"
 	"github.com/ghodss/yaml"
 	v1 "k8s.io/api/core/v1"
 )
@@ -30,11 +31,13 @@ import (
 var csiConfigMap string
 
 type CSIConfigMapValues struct {
-	Name string
+	Name        string
+	ClusterInfo []kubernetes.ClusterInfo
 }
 
 var CSIConfigMapDefaults = CSIConfigMapValues{
-	Name: "ceph-csi-config",
+	Name:        "ceph-csi-config",
+	ClusterInfo: []kubernetes.ClusterInfo{},
 }
 
 // NewCSIConfigMap takes a name from the CSIConfigMapValues struct and relaces

api/deploy/kubernetes/nfs/csi-config-map.yaml

@@ -5,4 +5,4 @@ metadata:
   name: "{{ .Name }}"
 data:
   config.json: |-
-    []
+    {{ .ClusterInfo }}

api/deploy/kubernetes/nfs/csidriver.yaml

@@ -6,5 +6,6 @@ metadata:
 spec:
   attachRequired: false
   fsGroupPolicy: File
+  seLinuxMount: true
   volumeLifecycleModes:
     - Persistent

api/deploy/kubernetes/rbd/csi-config-map.go

@@ -22,6 +22,7 @@ import (
 	"fmt"
 	"text/template"
 
+	"github.com/ceph/ceph-csi/api/deploy/kubernetes"
 	"github.com/ghodss/yaml"
 	v1 "k8s.io/api/core/v1"
 )
@@ -30,11 +31,13 @@ import (
 var csiConfigMap string
 
 type CSIConfigMapValues struct {
-	Name string
+	Name        string
+	ClusterInfo []kubernetes.ClusterInfo
 }
 
 var CSIConfigMapDefaults = CSIConfigMapValues{
-	Name: "ceph-csi-config",
+	Name:        "ceph-csi-config",
+	ClusterInfo: []kubernetes.ClusterInfo{},
 }
 
 // NewCSIConfigMap takes a name from the CSIConfigMapValues struct and relaces

api/deploy/kubernetes/rbd/csi-config-map.yaml

@@ -5,4 +5,4 @@ metadata:
   name: "{{ .Name }}"
 data:
   config.json: |-
-    []
+    {{ .ClusterInfo }}

api/deploy/kubernetes/rbd/csidriver.yaml

@@ -6,4 +6,5 @@ metadata:
 spec:
   attachRequired: true
   podInfoOnMount: false
+  seLinuxMount: true
   fsGroupPolicy: File

charts/ceph-csi-cephfs/templates/extra-deploy.yaml

@@ -0,0 +1,4 @@
+{{- range .Values.extraDeploy }}
+---
+  {{- tpl ( . | toYaml) $ }}
+{{- end }}

charts/ceph-csi-cephfs/values.yaml

@@ -323,6 +323,9 @@ cephconf: |
     # see https://github.com/ceph/ceph-csi/issues/1928
     fuse_big_writes = true
 
+# Array of extra objects to deploy with the release
+extraDeploy: []
+
 #########################################################
 # Variables for 'internal' use please use with caution! #
 #########################################################

charts/ceph-csi-rbd/templates/extra-deploy.yaml

@@ -0,0 +1,4 @@
+{{- range .Values.extraDeploy }}
+---
+  {{- tpl ( . | toYaml) $ }}
+{{- end }}

charts/ceph-csi-rbd/values.yaml

@@ -501,6 +501,9 @@ cephconf: |
     auth_service_required = cephx
     auth_client_required = cephx
 
+# Array of extra objects to deploy with the release
+extraDeploy: []
+
 #########################################################
 # Variables for 'internal' use please use with caution! #
 #########################################################

deploy/nfs/kubernetes/csi-nfsplugin-provisioner.yaml

@@ -65,6 +65,7 @@ spec:
             - "--leader-election"
             - "--retry-interval-start=500ms"
             - "--handle-volume-inuse-error=false"
+            - "--feature-gates=RecoverVolumeExpansionFailure=true"
           env:
             - name: ADDRESS
               value: unix:///csi/csi-provisioner.sock

docs/deploy-cephfs.md

@@ -77,6 +77,7 @@ you're running it inside a k8s cluster and find the config itself).
 | `csi.storage.k8s.io/provisioner-secret-namespace`, `csi.storage.k8s.io/node-stage-secret-namespace` | for Kubernetes | Namespaces of the above Secret objects                                                                                                                                                                                  |
 | `encrypted`                                                                                         | no             | disabled by default, use `"true"` to enable fscrypt encryption on PVC and `"false"` to disable it. **Do not change for existing storageclasses**                                                                          |
 | `encryptionKMSID`                                                                                   | no             | required if encryption is enabled and a kms is used to store passphrases                                                                                                                                                |
+| `extraDeploy` | no | array of extra objects to deploy with the release |
 
 **NOTE:** An accompanying CSI configuration file, needs to be provided to the
 running pods. Refer to [Creating CSI configuration](../examples/README.md#creating-csi-configuration)