From fc39996bfb2412b80d77b7322ba4ec7efb70fee6 Mon Sep 17 00:00:00 2001 From: coulof Date: Thu, 29 Aug 2024 19:21:44 +0200 Subject: [PATCH 1/7] Add tabs to improve readability --- .../csmoperator/drivers/powerstore.md | 28 +++++++++---------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/content/docs/deployment/csmoperator/drivers/powerstore.md b/content/docs/deployment/csmoperator/drivers/powerstore.md index 256d5c0cbd..fdbf001f7e 100644 --- a/content/docs/deployment/csmoperator/drivers/powerstore.md +++ b/content/docs/deployment/csmoperator/drivers/powerstore.md @@ -20,31 +20,27 @@ kubectl get csm --all-namespaces ### Prerequisites +Depending of the protocol you use, refer to the following sections to configure the requirements for the CSI Driver. -#### Fibre Channel requirements - -Dell PowerStore supports Fibre Channel communication. If you use the Fibre Channel protocol, ensure that the -following requirement is met before you install the CSI Driver for Dell PowerStore: -- Zoning of the Host Bus Adapters (HBAs) to the Fibre Channel port must be done. - - -#### Set up the iSCSI Initiator -The CSI Driver for Dell PowerStore v1.4 and higher supports iSCSI connectivity. +> **NOTE:** with RedHat Openshift all the software pre-requisites are already installed. +{{< tabpane name="protocol-prereqs" text=true >}} +{{% tab header="Fiber Channel" %}} +The zoning of the Host Bus Adapters (HBAs) to the Fibre Channel port must be done, **before** installing the CSI Driver, if you use the Fibre Channel protocol. +{{% /tab %}} +{{% tab header="iSCSI" %}} If you use the iSCSI protocol, set up the iSCSI initiators as follows: - Ensure that the iSCSI initiators are available on both Controller and Worker nodes. - Kubernetes nodes must have access (network connectivity) to an iSCSI port on the Dell PowerStore array that -has IP interfaces. Manually create IP routes for each node that connects to the Dell PowerStore. +has IP interfaces. - All Kubernetes nodes must have the _iscsi-initiator-utils_ package for CentOS/RHEL or _open-iscsi_ package for Ubuntu installed, and the _iscsid_ service must be enabled and running. To do this, run the `systemctl enable --now iscsid` command. - Ensure that the unique initiator name is set in _/etc/iscsi/initiatorname.iscsi_. For information about configuring iSCSI, see _Dell PowerStore documentation_ on Dell Support. - - -#### Set up the NVMe Initiator - -If you want to use the protocol, set up the NVMe initiators as follows: +{{% /tab %}} +{{% tab header="NVMe" %}} +Set up the NVMe initiators as follows: - The driver requires NVMe management command-line interface (nvme-cli) to use configure, edit, view or start the NVMe client and target. The nvme-cli utility provides a command-line and interactive shell option. The NVMe CLI tool is installed in the host using the below command. ```bash sudo apt install nvme-cli @@ -63,6 +59,8 @@ modprobe nvme_tcp *NOTE:* - Do not load the nvme_tcp module for NVMeFC +{{% /tab %}} +{{< /tabpane >}} #### Linux multipathing requirements Dell PowerStore supports Linux multipathing. Configure Linux multipathing before installing the CSI Driver for Dell From 3a633f25f456aadab0e47d39d8450b957e9d369f Mon Sep 17 00:00:00 2001 From: coulof Date: Fri, 6 Sep 2024 10:23:30 +0200 Subject: [PATCH 2/7] Add tabs for PowerMax --- .../csmoperator/drivers/powermax.md | 147 ++++++++++-------- 1 file changed, 85 insertions(+), 62 deletions(-) diff --git a/content/docs/deployment/csmoperator/drivers/powermax.md b/content/docs/deployment/csmoperator/drivers/powermax.md index 95e3bf8bf6..63b263538a 100644 --- a/content/docs/deployment/csmoperator/drivers/powermax.md +++ b/content/docs/deployment/csmoperator/drivers/powermax.md @@ -10,6 +10,12 @@ To deploy the Operator, follow the instructions available [here](../../#installa Note that the deployment of the driver using the operator does not use any Helm charts and the installation and configuration parameters will be slightly different from the one specified via the Helm installer. +## Listing installed drivers with the ContainerStorageModule CRD +User can query for all Dell CSI drivers using this command: +```bash +kubectl get csm --all-namespaces +``` + ## Prerequisites The CSI Driver for Dell PowerMax can create PVC with different storage protocols access : * direct Fiber Channel @@ -20,21 +26,20 @@ The CSI Driver for Dell PowerMax can create PVC with different storage protocols In most cases, you will use one protocol only; therefore you should comply with the according prerequisites and not the others. -### Listing installed drivers with the ContainerStorageModule CRD -User can query for all Dell CSI drivers using this command: -```bash -kubectl get csm --all-namespaces -``` +Depending of the protocol you use, refer to the following sections to configure the requirements for the CSI Driver. +> **NOTE:** with RedHat Openshift all the software pre-requisites are already installed. -### Fibre Channel Requirements +{{< tabpane name="protocol-prereqs" text=true >}} +{{% tab header="Fiber Channel" %}} CSI Driver for Dell PowerMax supports Fibre Channel communication. Ensure that the following requirements are met before you install CSI Driver: - Zoning of the Host Bus Adapters (HBAs) to the Fibre Channel port director must be completed. - Ensure that the HBA WWNs (initiators) appear on the list of initiators that are logged into the array. - If the number of volumes that will be published to nodes is high, then configure the maximum number of LUNs for your HBAs on each node. See the appropriate HBA document to configure the maximum number of LUNs. +{{% /tab %}} +{{% tab header="NVMe" %}} -### NVMeTCP requirements If you want to use the protocol, set up the NVMe initiators as follows: - Setup on Array
Once the NVMe endpoint is created on the array, follow the following step to update endpoint name to adhere with CSI driver. @@ -54,8 +59,8 @@ modprobe nvme_tcp ``` - The NVMe modules may not be available after a node reboot. Loading the modules at startup is recommended. - Generate and update the _/etc/nvme/hostnqn_ with hostNQN details. - -### iSCSI Requirements +{{% /tab %}} +{{% tab header="iSCSI" %}} The CSI Driver for Dell PowerMax supports iSCSI connectivity. These requirements are applicable for the nodes that use iSCSI initiator to connect to the PowerMax arrays. @@ -68,9 +73,38 @@ Set up the iSCSI initiators as follows: For more information about configuring iSCSI, see [Dell Host Connectivity guide](https://www.delltechnologies.com/asset/zh-tw/products/storage/technical-support/docu5128.pdf). -##### iscsi-daemon `MachineConfig` -To configure iSCSI in Red Hat OpenShift clusters, you can create a `MachineConfig` object using the console or `oc` to ensure that the iSCSI daemon starts on all the Red Hat CoreOS nodes. Here is an example of a `MachineConfig` object: +{{% /tab %}} +{{% tab header="NFS" %}} +CSI Driver for Dell PowerMax supports NFS communication. Ensure that the following requirements are met before you install CSI Driver: +- Configure the NFS network. Please refer [here](https://dl.dell.com/content/manual57826791-dell-powermax-file-protocol-guide.pdf?language=en-us&ps=true) for more details. +- PowerMax Embedded Management guest to access Unisphere for PowerMax. +- Create the NAS server. Please refer [here](https://dl.dell.com/content/manual55638050-dell-powermax-file-quick-start-guide.pdf?language=en-us&ps=true) for more details. + +{{% /tab %}} +{{% tab header="Auto RDM for vSphere over FC" %}} + +The CSI Driver for Dell PowerMax supports auto RDM for vSphere over FC. These requirements are applicable for the clusters deployed on ESX/ESXi using virtualized environement. + +Set up the environment as follows: + +- Requires VMware vCenter management software to manage all ESX/ESXis where the cluster is hosted. + +- Add all FC array ports zoned to the ESX/ESXis to a port group where the cluster is hosted . +- Add initiators from all ESX/ESXis to a host(initiator group)/host group(cascaded initiator group) where the cluster is hosted. +- Create a secret which contains vCenter privileges. Follow the steps [here](#support-for-auto-rdm-for-vsphere-over-fc) to create the same. + + +{{% /tab %}} +{{< /tabpane >}} + +### `MachineConfig` +To configure services in Red Hat OpenShift clusters, you can create a `MachineConfig` object +using the console or `oc` to ensure that the daemons start automatically on all the Red Hat CoreOS nodes. +Here are examples of `MachineConfig` objects: + +{{< tabpane name="machineconfig" text=true >}} +{{% tab header="iscsid" %}} ```yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig @@ -87,12 +121,11 @@ spec: - name: "iscsid.service" enabled: true ``` -Once the `MachineConfig` object has been deployed, CoreOS will ensure that `iscsid.service` starts automatically. -Alternatively, you can check the status of the iSCSI service by entering the following command on each worker node in the cluster: +You can check the status of the iSCSI service by entering the following command for each worker node in the cluster: ```bash -sudo systemctl status iscsid +oc debug node/[node-name] -- chroot /host systemctl status iscsid ``` The service should be up and running (i.e. should be active state). @@ -103,40 +136,13 @@ If the `iscsid.service` is not running, then perform the following steps on each 3. Ensure that iscsid service is running - Enable ```sudo systemctl enable iscsid``` & restart ```sudo systemctl restart iscsid``` iscsid if necessary. Note: If your worker nodes are running Red Hat CoreOS, make sure that automatic ISCSI login at boot is configured. Please contact RedHat for more details. -### NFS requirements - -CSI Driver for Dell PowerMax supports NFS communication. Ensure that the following requirements are met before you install CSI Driver: -- Configure the NFS network. Please refer [here](https://dl.dell.com/content/manual57826791-dell-powermax-file-protocol-guide.pdf?language=en-us&ps=true) for more details. -- PowerMax Embedded Management guest to access Unisphere for PowerMax. -- Create the NAS server. Please refer [here](https://dl.dell.com/content/manual55638050-dell-powermax-file-quick-start-guide.pdf?language=en-us&ps=true) for more details. - -### Linux multipathing requirements - -CSI Driver for Dell PowerMax supports Linux multipathing. Configure Linux multipathing before installing the CSI Driver. - -Set up Linux multipathing as follows: - -- All the nodes must have the _Device Mapper Multipathing_ package installed. - *NOTE:* When this package is installed it creates a multipath configuration file which is located at `/etc/multipath.conf`. Please ensure that this file always exists. -- Enable multipathing using `mpathconf --enable --with_multipathd y` -- Enable `user_friendly_names` and `find_multipaths` in the `multipath.conf` file. - -As a best practice, use these options to help the operating system and the mulitpathing software detect path changes efficiently: -```text -path_grouping_policy multibus -path_checker tur -features "1 queue_if_no_path" -path_selector "round-robin 0" -no_path_retry 10 -``` - -#### multipathd `MachineConfig` - -If you are installing a CSI Driver which requires the installation of the Linux native Multipath software - _multipathd_, please follow the below instructions +{{% /tab %}} +{{% tab header="multipathd" %}} To enable multipathd on RedHat CoreOS nodes you need to prepare a working configuration encoded in base64. -```bash echo 'defaults { +```bash +echo 'defaults { user_friendly_names yes find_multipaths yes } @@ -165,9 +171,13 @@ spec: mode: 400 path: /etc/multipath.conf ``` + After deploying this`MachineConfig` object, CoreOS will start multipath service automatically. -Alternatively, you can check the status of the multipath service by entering the following command in each worker nodes. -`sudo multipath -ll` +You can check the status of the multipath service by entering the following command in each worker nodes. + +```bash +oc debug node/[node-name] -- chroot /host multipath -ll +``` If the above command is not successful, ensure that the /etc/multipath.conf file is present and configured properly. Once the file has been configured correctly, enable the multipath service by running the following command: `sudo /sbin/mpathconf –-enable --with_multipathd y` @@ -176,33 +186,46 @@ Finally, you have to restart the service by providing the command `sudo systemctl restart multipathd` For additional information refer to official documentation of the multipath configuration. +{{% /tab %}} +{{< /tabpane >}} -#### PowerPath for Linux requirements -CSI Driver for Dell PowerMax supports PowerPath for Linux. Configure Linux PowerPath before installing the CSI Driver. +### Multipathing requirements -Follow this procedure to set up PowerPath for Linux: +CSI Driver for Dell PowerMax supports Linux multipathing PowerPath. Configure multipathing before installing the CSI Driver. -- All the nodes must have the PowerPath package installed . Download the PowerPath archive for the environment from [Dell Online Support](https://www.dell.com/support/home/en-in/product-support/product/powerpath-for-linux/drivers). -- `Untar` the PowerPath archive, Copy the RPM package into a temporary folder and Install PowerPath using `rpm -ivh DellEMCPower.LINUX--..x86_64.rpm` -- Start the PowerPath service using `systemctl start PowerPath` - ->Note: Do not install Dell PowerPath if multi-path software is already installed, as they cannot co-exist with native multi-path software. +{{< tabpane name="multipathing" text=true >}} +{{% tab header="multipathd" %}} +Set up Linux multipathing as follows: -### Auto RDM for vSphere over FC requirements +- All the nodes must have the _Device Mapper Multipathing_ package installed. + *NOTE:* When this package is installed it creates a multipath configuration file which is located at `/etc/multipath.conf`. Please ensure that this file always exists. +- Enable multipathing using `mpathconf --enable --with_multipathd y` +- Enable `user_friendly_names` and `find_multipaths` in the `multipath.conf` file. -The CSI Driver for Dell PowerMax supports auto RDM for vSphere over FC. These requirements are applicable for the clusters deployed on ESX/ESXi using virtualized environement. +As a best practice, use these options to help the operating system and the mulitpathing software detect path changes efficiently: +```text +path_grouping_policy multibus +path_checker tur +features "1 queue_if_no_path" +path_selector "round-robin 0" +no_path_retry 10 +``` +{{% /tab %}} +{{% tab header="PowerPath" %}} -Set up the environment as follows: +Follow this procedure to set up PowerPath for Linux: -- Requires VMware vCenter management software to manage all ESX/ESXis where the cluster is hosted. +- All the nodes must have the PowerPath package installed . Download the PowerPath archive for the environment from [Dell Online Support](https://www.dell.com/support/home/en-in/product-support/product/powerpath-for-linux/drivers). +- `Untar` the PowerPath archive, Copy the RPM package into a temporary folder and Install PowerPath using `rpm -ivh DellEMCPower.LINUX--..x86_64.rpm` +- Start the PowerPath service using `systemctl start PowerPath` -- Add all FC array ports zoned to the ESX/ESXis to a port group where the cluster is hosted . +>Note: Do not install Dell PowerPath if multi-path software is already installed, as they cannot co-exist with native multi-path software. +{{% /tab %}} +{{< /tabpane >}} -- Add initiators from all ESX/ESXis to a host(initiator group)/host group(cascaded initiator group) where the cluster is hosted. -- Create a secret which contains vCenter privileges. Follow the steps [here](#support-for-auto-rdm-for-vsphere-over-fc) to create the same. ### CSI PowerMax ReverseProxy From 492676e6e8449fbd79f579144262ddf817a7b529 Mon Sep 17 00:00:00 2001 From: coulof Date: Fri, 6 Sep 2024 10:35:01 +0200 Subject: [PATCH 3/7] Simplify Helm installation and remove redundant information --- .../helm/drivers/installation/powermax.md | 60 ++++++------------- 1 file changed, 17 insertions(+), 43 deletions(-) diff --git a/content/docs/deployment/helm/drivers/installation/powermax.md b/content/docs/deployment/helm/drivers/installation/powermax.md index 28b2041d2d..bc711c3db6 100644 --- a/content/docs/deployment/helm/drivers/installation/powermax.md +++ b/content/docs/deployment/helm/drivers/installation/powermax.md @@ -55,14 +55,19 @@ Install Helm 3 on the master node before you install CSI Driver for Dell PowerMa ``` -### Fibre Channel Requirements +### Storage Protocol Requirements +Depending of the protocol you use, refer to the following sections to configure the requirements for the CSI Driver. + +{{< tabpane name="protocol-prereqs" text=true >}} +{{% tab header="Fiber Channel" %}} CSI Driver for Dell PowerMax supports Fibre Channel communication. Ensure that the following requirements are met before you install CSI Driver: - Zoning of the Host Bus Adapters (HBAs) to the Fibre Channel port director must be completed. - Ensure that the HBA WWNs (initiators) appear on the list of initiators that are logged into the array. - If the number of volumes that will be published to nodes is high, then configure the maximum number of LUNs for your HBAs on each node. See the appropriate HBA document to configure the maximum number of LUNs. -### iSCSI Requirements +{{% /tab %}} +{{% tab header="iSCSI" %}} The CSI Driver for Dell PowerMax supports iSCSI connectivity. These requirements are applicable for the nodes that use iSCSI initiator to connect to the PowerMax arrays. @@ -75,14 +80,17 @@ Set up the iSCSI initiators as follows: For more information about configuring iSCSI, see [Dell Host Connectivity guide](https://www.delltechnologies.com/asset/zh-tw/products/storage/technical-support/docu5128.pdf). -### NFS requirements +{{% /tab %}} +{{% tab header="NFS" %}} CSI Driver for Dell PowerMax supports NFS communication. Ensure that the following requirements are met before you install CSI Driver: - Configure the NFS network. Please refer [here](https://dl.dell.com/content/manual57826791-dell-powermax-file-protocol-guide.pdf?language=en-us&ps=true) for more details. - PowerMax Embedded Management guest to access Unisphere for PowerMax. - Create the NAS server. Please refer [here](https://dl.dell.com/content/manual55638050-dell-powermax-file-quick-start-guide.pdf?language=en-us&ps=true) for more details. -### NVMeTCP requirements +{{% /tab %}} +{{% tab header="NVMe" %}} + If you want to use the protocol, set up the NVMe initiators as follows: - Setup on Array
Once the NVMe endpoint is created on the array, follow the following step to update endpoint name to adhere with CSI driver. @@ -103,7 +111,8 @@ modprobe nvme_tcp - The NVMe modules may not be available after a node reboot. Loading the modules at startup is recommended. - Generate and update the _/etc/nvme/hostnqn_ with hostNQN details. -### Auto RDM for vSphere over FC requirements +{{% /tab %}} +{{% tab header="Auto RDM for vSphere over FC" %}} The CSI Driver for Dell PowerMax supports auto RDM for vSphere over FC. These requirements are applicable for the clusters deployed on ESX/ESXi using virtualized environement. @@ -127,6 +136,8 @@ Create the secret by running the below command, ```bash kubectl create -f samples/secret/vcenter-secret.yaml ``` +{{% /tab %}} +{{< /tabpane >}} ### Certificate validation for Unisphere REST API calls @@ -174,47 +185,10 @@ features "1 queue_if_no_path" path_selector "round-robin 0" no_path_retry 10 ``` -#### multipathd `MachineConfig` If you are installing a CSI Driver which requires the installation of the Linux native Multipath software - _multipathd_, please follow the below instructions -To enable multipathd on RedHat CoreOS nodes you need to prepare a working configuration encoded in base64. - -```bash echo 'defaults { -user_friendly_names yes -find_multipaths yes -} -blacklist { -}' | base64 -w0 -``` - -Use the base64 encoded string output in the following `MachineConfig` yaml file (under source section) -```yaml -apiVersion: machineconfiguration.openshift.io/v1 -kind: MachineConfig -metadata: - name: workers-multipath-conf-default - labels: - machineconfiguration.openshift.io/role: worker -spec: - config: - ignition: - version: 3.2.0 - storage: - files: - - contents: - source: data:text/plain;charset=utf-8;base64,ZGVmYXVsdHMgewp1c2VyX2ZyaWVuZGx5X25hbWVzIHllcwpmaW5kX211bHRpcGF0aHMgeWVzCn0KCmJsYWNrbGlzdCB7Cn0K - verification: {} - filesystem: root - mode: 400 - path: /etc/multipath.conf -``` -After deploying this`MachineConfig` object, CoreOS will start multipath service automatically. -Alternatively, you can check the status of the multipath service by entering the following command in each worker nodes. -`sudo multipath -ll` - -If the above command is not successful, ensure that the /etc/multipath.conf file is present and configured properly. Once the file has been configured correctly, enable the multipath service by running the following command: -`sudo /sbin/mpathconf –-enable --with_multipathd y` +> NOTE: to enable e multipathd on RedHat CoreOS you can refer to sample `MachineConfig` given [here](../../../csmoperator/drivers/powermax/#machineconfig). Finally, you have to restart the service by providing the command `sudo systemctl restart multipathd` From 7b9aae100fc31b9436c26c8c7d9d2ebf2f954a63 Mon Sep 17 00:00:00 2001 From: coulof Date: Fri, 6 Sep 2024 10:54:52 +0200 Subject: [PATCH 4/7] Simplify doc and fix typos --- .../csmoperator/drivers/powerstore.md | 3 +- .../drivers/installation/.powermax.md.swp | Bin 0 -> 69632 bytes .../helm/drivers/installation/powermax.md | 95 +++++++++--------- .../helm/drivers/installation/powerstore.md | 63 +++--------- 4 files changed, 68 insertions(+), 93 deletions(-) create mode 100644 content/docs/deployment/helm/drivers/installation/.powermax.md.swp diff --git a/content/docs/deployment/csmoperator/drivers/powerstore.md b/content/docs/deployment/csmoperator/drivers/powerstore.md index fdbf001f7e..039d6f2c30 100644 --- a/content/docs/deployment/csmoperator/drivers/powerstore.md +++ b/content/docs/deployment/csmoperator/drivers/powerstore.md @@ -79,7 +79,8 @@ If you are installing a CSI Driver which requires the installation of the Linux To enable multipathd on RedHat CoreOS nodes you need to prepare a working configuration encoded in base64. -```bash echo 'defaults { +```bash +echo 'defaults { user_friendly_names yes find_multipaths yes } diff --git a/content/docs/deployment/helm/drivers/installation/.powermax.md.swp b/content/docs/deployment/helm/drivers/installation/.powermax.md.swp new file mode 100644 index 0000000000000000000000000000000000000000..2d0283bdf616db9e380bf40ed1cc77c42d61efd9 GIT binary patch literal 69632 zcmeI53vguFS>GFsO#-Y9*pLcP)Io!HXJmEvXl9?v>=>@w+l#%vfHj8xCeI>Q0 z)!o~-TNk{34}ZdsZ>&-AU0rE97uvgC6K~Z z$?yN2bMC#}lICr*-s!ISM&0+Gd+s^kcg}ad*ZI!+@n;qj4%Mt9imZs^AKKhMt2K79GPXD^;VdHr(V>MN{ocI#QcQQvG0wubAqMt8g3 zOxx{xBipVvyN#^g=ynEaXHd5v&9v9{8ML~cdat{i z_P3L(we9Bhn=rXM!N9$Vf#G1I`sjU?lP6}63js%_zQMimE1$kMaWU!j1OpQcOfWFP zzyt#m3`{UE!N3Fq_b3Jiy*E_8jlO?j=>J0a`vqfv-!OLn%-DTQ2lja){QXPeZ>;Bw z!tZYie}8%G{%gbUKOO&u2jlavi`T>7uN=Go#bfuscI-YT4g19Uzj5q-y#6JjTy^Yz zydM8PGIsw<$LgOSyZ>cl_un>l|GselP2unI*!|as>u(KzH^=V3CR`7fD?dDTKir=D znP6aofe8jC7?@ySf`JJJCK#AtV1j`O1|}G|M=_A#FnN_^D+KZC;md%%AL zehK)y!1n^*1H1!e^f};3;2iKjQC?pL+Q21X4tN0ga^QDRe18FWFK`z4e<;e&1Ahki zYTz}%7Xm+yV*O*l2Z0X&-wI^F*8^V-d=>B};D4d0zYJUh-Ud7X+z)&Y3jV{uzs3;o ze=!#P5%9afuLA!Tcr);gz-KTt{3tL7yns>S*HN5*6XVBk05iZ_fUgH$3w#kEyj+V) z_1zvL3(Je{jKwp!jc(taAqv7RDU{(ta@A$SUa#99WbWd*Wk-<2_OR1xShz#Yo$X}9 ze(DBW$$!&XcTm4k=5}8XVvz-pydP?=F+v^Tp2%B)ltLG|}cfIS) zZjh|Itu$$-{l}+x2{xK~~QO-F~u})~z%+X;%ZT6frVEwD!zXy&=yB(J%jjb`FCjEZ0S99|aS})nx2z8|0N}F!bwE{lc#n+u) zZSjVK>)j=m#vaJ%i*=|*=f{VMQ3mR@2Cam?WU5Tf=yaPPK%=G89k|5x5^&Sqa7ky+ z*;|DaBjVA;_^^9YxH}fB7#+L5o3!BV!}YY^Ne5K#Uy^MRRMd1^$xiB$MkCEKca;0P z-ToC(ARi&egn^_pj{VT!1KvhQm<(G<<`$Re;vnsBBvfB>7n079K%qU?=%>jb)oog? z7HQ5Ijnp*%d}sng#N&VTMXil|SlkqPh0$auX|>Tu9%XRFY(afL}!u5a0oR6VtC zWPnG{R5JL(3MERDNO&7XOwOx(J!L}1sj`nTB^v1s!;C6aEBp{KLJpZ-V~r9`Ld}sp zIkU6@moekVS((*w5HqW~#SQ<`;WlGb!+odQ9*PrK10w!=!}WG6+wxP=H?Wntt=8t& zj2`dgtj|-DAhg!@aNBkATF*ay(GNA)Z{I!j#Jo+reB9OCa_S!iA^mQz&(us+%!!$a zkB5F6f4$yI4jPiG=hrjY=98Zf`NaO_ zu5n7RCc+tScw(5j`DW5%>hV5x)F!d+pKGmC*}_)B1aEt;mowDJH(BC(Z8!x3{pQBn zXr-Uq6_R%|W5$lvpBH|gw`rYjvR5zeg$<) zp@r^pNo+vMan7x+tvUT|4Et>t*Xe$;Tk~O?G_KE!6{l$nwZ&%If7?mM2&iX`eyhh~ zy_pWGx?Vl$uDSIj+X|J3ua}2@5Q=)hsMuu!8f$!B)55ddZ}x8OM-MV09y8etx8kDHX7;kiWq4F@J zmJ8h>j7t0~+4OQDOw#%e_kho6s32T1l=~EHA)^?RrO@b{~RI`xw2d#}3Vs7eo zl6I@<>k@*W?zFO=Etr?jF0Z)xrA3G2*Uos#<9>VC9<)R!)i>5W5ZZN@&v**#Ec<%8 z#1Xr53*%)kK64>mSwKiAPl>8Y;bWKl`ZRWNlq-$h92;hgvM`=Yi#mP11Y20%YHbYM zQKV@1d{|yVr9z)I@Ar)X)_gBS)Q<%EWmV==wUP9a^;R2+FU^h}vz5HyBIVZQ@@q+9 zX!+Lp;d{8Mf`q^nS~4a#Y-KHED-_A=={5}{J7gO_aOu|iUYKlJEEq*QXa`L{$#+h= z#uoA@@+s^gw8#&Oz+yDM*1OEZ5)xImz+#UuamLU#(%?DJYdTCbWk9xhOIeu-E8;v6rQSNHDizAooV0nmTDC?xp*l0 zbGc`yl^B{T2=K;m860!&bOn}kWzZV5)44*Stc+c#RC$Z^|8GX8-vOlmM^#t9hq^ZT zOfWFPzyt#m3`{UE!N3Fq6AVl+Fu}kC0}~8PFfhTuZDQc`#qI#rJa`~*Of|8iwvsl^ zLw*b#cCqbNX6DC@es|j(ahON(--h`2vA0k>txNM$_L4QwHW4TdeQ)#9f z3HGst{7u=iy~*_d573LBmEw>7|4pGszlQJr-vNIgpZyO5pTtN137`$U9rz~T7x2gb zEbtNFuL6GsXafmwAMiThwZO;m&wmWq0ycpsfpfqo@z;L>_zmD!fu9F{4)`GO0pMTZ z%l{SNjlfp|pTU3s$H1q7-v`bDKZYOw>wu5q%l}E>Zvq`)1Nai)Q~39P1Nc$kgTPBb z2lzMm_WwCB4}6-we**YL;O7A0;-`RD`UmU0%=9v-yPcNX-plPwRh-ZR%ywCAFWGKe z0ScPQ{Sl%Enk^qT&`Ec5zi1PSFiuJt&Vm6>7T95MlWGhVc0>HbxS7q*Q+c=)jgUHG z6oQr39FnugYsa10Yw>gRy10>ISc6P{4g7ysWL(7TR+x9Ir)AOVxub7A=J-Bu@dY?U zixM&i=N7OhpI)rR4%**FlhJ;lWzF4xzuUzh;i{JoM^^6Z7`Nk@mCl-a%PS5esY zswZl*v$f;(aJzh~?ylZ=O88#XF>@xA)}TQB=gVjR!9D2*kig-{b<$ zm4ab;&t@V%BgTUfHYW_8%pFG5CO-I)9uVTub=8iWDr%kbuA%hd2d!kOL~O`di8z>P zvsXXleANJ0%2^Kr+Ct<+2f-%Bmt5r^vCL7TX z3h1$U20@@~(NMIz3)sN%EFWRXKXBO053YhyXJX0kr#JdYEIb#N&txvdsokn6bVVYhL+9!I!3%uN}I+u9*+{h&y_zr!kSVkD0f(RF6FB z@;v2Erz|>KS*QsX1JjL^(FKoq-d06I@%m|P-x!IHW%VQ7@UE(lTB*icP_T#qWKiJU z9&6d68*}EWb?!g7=-bOkg>d&TA;mr134B~##4z)14yWwtG+TN&U zt<4Tog~1XAskn=eM1W1F#40uA8_g>TQAj=F-PA(8xLC?8UI6!Oc=K9zrPW&~UiwV# zVqYtk87Uu%VPsOQZY1q2UA6UysK9=@)9MoBQ=XnK84S|x9(+G8L26`=5KljJLi6y- zt5ymPaz`_~TH!unRjkGed|%a%g?VQ*gw`4n@+gNux%4Uv14BXqg298HFO%+iz1-bu zK^>+c>B0y{pf5gvDbyRVNHD04BZ$E|gR13;Bz_+Pb#nAu3j+;tFRPu`dwTSs^78re zQ+m1Pq=tSn7=0ZG-d0qKMwoi#`DAQvz-%6uQp4L$FlZmTYwfhC95m}NJmMe=uK#AM zY2?^wA<(ITajZ-JId~p;(ZI?Io5B{s2-59!ZMJp-MJc@2lO`Z7v|4ek>nYKO+eB!? zocucD$JYOl|J6qeYyGVGm=CW=6eQ6fBj`Fvzi zzIcfPd)ybOUky7H=7ssi=Bn|L%cSq5<>N%F5ERN!{%&ZN12iFBWvC5~CGsnQIUbZ0eOp8;pjv?sNCK z=H9^ycP-ZyJ%+2ZwV}5s)%vLy*hHIO9G~Y2nJ)Kw-M+dXU zt48O@ATFqhC1^Y8?;S$U(YeAPFiu!ZzNNS!YI$!Pqhr5y)h(WxaI-t6WnMv?e8-E= ztmavnFxgElP=2Ke%bP`Yiq`~DOGby$VFTMMxL#m-d8_XEFQof5zfAsqjxaES@zM{?kEQ6xV2M9dM5`Y_ z&B>*^je%>dyk-?{QnS!+(Exlk{r@t0{9hl_|DQ$w{~7fEuR{NS2L1n6@xL#i|9=?$ ze+vCy<-dEP|KFuZ!NBl(B z!-tDEFX)p>2XCh8-nIw9{gd$@+utXD2lG5>g zs=RmMzBI29t!@m`y+hPN+~|z!Y_}yC3TFE;+i*&kX_3%5!bDufrVWab6XU+`=Xtyym_# zjHKn*t0sXDC7<%j`Q?3n<$|7?o4Fxi%;kCSNoZkSO|9T2p9&NfjrN(B_OU!o9tDe( zi<-%4HCpsnixD&VcIz0hS$2(3fMDP!?ii{JM4gD?IALM&+T<}z#%U1S!DGw6GY?-D^U}N!! zo-7h=4pc(&7n!=W2ZC&6N=KulJIE1}&uw8KE=b_i8w?a>Oz`J|WiC6@>O+%oDTLeS z`8Jt4$zW@A+k@77?@BA}lEH@=r;a5>)o*+9wp#q^e<;#~{Ym|LyR}}A1ki*b?cX^I zse~}47*Q_gp_ZhgBrt=FRJKWM&P}bHeQsq+am!P0TfTI0YGOdU%`p@S6A&86aUP3) zdp|YWx@`GmM~RX4``!N3jGNj``kkq|{lT1|E6R7Jzs+=gxH%!y4XBh;!=uZV6Sf}_ zBG+F+r6Cvz&!+zp7j`txe5;VLDuv#L*zstDBL$dtKC)S(ZBm1_|0+eBI~r3ndO>4` z^tD#fJ=02e2}o9^P5vk>Anf8;w4e^6Df;BwL~770O=9p-8a6+<7`BUfsbkOfm*kQ7 z!%hq5(I}H8jF_Bita&3*2|@ILxtjD5EmUT|DvK!zjwa8d+0@b;3x0-AgE-&3Xryna zoz0L+(e?`XGBVK&w#b=gd%Ti{ww%_$xS|F5Gd2Ijf?wg$KR44Q_HitA*_tKcV5PLK z_q4&I=kHMC;s)EVA)1A+`(X1%0@d`c)UwF2HfjHZ#BryG)lcm;&|4G}PMmkrhOEty zANgJp!mcw};zSuiu9(*WYyGmt|LnzEr8VEy?CZQ`a_e?55NUJ3S}*;-jvjwyO#i=v z{{NrQ|IeZSe;ED$0rdaxK>z;(^#3nH|5y2UPW1m<(f!Sl9L>6(d1p&$|Du9tuQ@*_ zv;>F=MYdR^kq#R{jFE?3&Y0S4@sO%$$!{kz*Uirkn-kX@E`E5FG4@=mk9v2wYU_FM z5toEvEx{!9wa{@{`Y8q(wCf$B;7pENmuIwgu^-3i+D|vJ8`yS6vfs0zo|UD|?&f>- z6FJiq%Q$+v#XPcaHthOlS`eya`}8y^0`m5v11z}7kWu`?Y=n~TuEM^qC@jbZ&~B&O z4I{TUVn`rExe0**@k;Q{#fQ&yI}fm7k*oovK{mO>i&N~zD*cM7;>!bNbW_&7OW?7A z9qFy)0Ll$w@iiNn3A?%I7HRH9GmJ8JI|^{N*#O~;4edhsj9c~xr66YNR_4|&Vyl@U z=X^+KqM!tea=O%Mq|dUE{oJzbKx<=7yLW#^OXneo@)EY+d9?7<{F2Seuz@yikSz3q zO|rb;wzi`hLRxyA2ji2olg~R*GLEs6NAjJfR(;8@XUsL<^Q38QC4%$3Fqwvg(ZQQu zsvz!V6|)ZRS=DkS-E&j0<({u8dKF)>Lm*HnxTz~LMf+2N&Pa)eAZL|k9<*YUab^=R z4N(pt(jp`qq*RwphjT?XnKy%7?Ibm97F()?q1+Xx)HLFJ9`&Br4LiJmjkMVnW*YRx zqn$Aeee6`oP|g7d$@aa#?Di4FuMyJDFKT};D~n(BM~=|3&3K9eL0^1Uahf_$M8CC# zUWR}Wl0e)78;Pv~FAl1Va4wn^egl>|c0Ie4J^Fe8Zb)7mzil8|wDULScDq;X7{9}Y z?W2frD_X2X+?MSALH|q`uL1EUS0A*H2ZV~DnYe8hnEK+AKGRM#xJ0o&x4i62y{7V- zeh4i)vi5X4Gt5*SgH)0vXwBeNBn<|doQ+X6u%b>TSJ+ZKcZ7+uEL#iXS{5J7mSQfk zn0zmGf{J-$ROs!+c~Zd&7qSJAovY{PN+1|nrbrQybGvg3Oe$N5#OT5q4>LYA*DFtU zc9cIv0>L4FGxW+2rP35kipZQ`WxKH!H`~SZS{r|tC|38iX|10SJ_6&>BpxaJyelnz`LV4+J(X2+sVrqfq2zU^;oRfuUm>@kso6pbR*2NMhO!rc86`6JPyWfhfSf-;Eid$#Kzx( zBxg;fsh=4o1eu?ZAv85&YPJbB?+yl{X~vGH(_D!eIJT%aHz$+Fx0c)?d>9_t5|U2m1dy`u{JZ|JU&UzYqQYGwA=D=>NZo{;y{) z>ABKJc~BI3e!)`Q^*wbteKVx@blUh-3Qac+RMReOzQyc9rkcs;jxbPA*T<|YR4Rtt zn57L1Ts)w9Bxo%d^DkmBP&ldCwOk~9Nj}&PBD5%efWHTIA?e#PI#W@ zSC08?Ipo^I6hQ8NA7{2|8DM?h&oO+K(*#-?)0nF-EJ5Dd`2fOoVOwLd^09lnE$#DCM>{lq+;a>G_n@Ezdkx+ucpdHRZVTP||B1>^T(40r!Xb8{@A+kFbR>$_aK8xj3}~*U7CFM1`{aU|Dj#k#i)7b(Dy&Y%KlB zx9*xFODglj?`t1)iKFdq?k#wBQ&K$-HKD~vB8!srOv-rdpnUVS$stcf@8oj}7}#e* zo7gsQ0ehdtqWcIl=9ZanDtmbDlj)MQnKhZz^Q=Dk(G8!)_T6h$2YN*QRrXbi6lL$_ zgbttG!7)OL>1oY^8|wcmG|~L;E?o)@QTl(1F8@P{ohby#ew3uzLPb{Klq0 z??XTn!i`q7Afsan41~(@;Eza7iPE0vB~r%#&yc^HoDm!)hd=Js{V7=5_0wqFXv&n0 zlf-GX2w9f5Iq3g7x@`391aGm7scT-{IKG|Ug(W|CkQu%dCyPp{Kk^kHj@=1fZwKdx zJrihQ@tDT6?qF#dodqEnW3s#)hWZ;`zwXj9f^5IM%nt`!-G1vOV^8~*m?FHgu5)Ya zgr^U6NNm|YUmvNpe!Vp&;rzt%0RkM4*-G1XmTF*(IBj3h)IFNFc8-ik7I0*oa>u5IpNow z{l)o)fjp%p-L6dNa~NXp&h-*`8k`6^N**0@xol@#EpJ|4*UEf9II~FaQ6?(f=Pq|NjvB|7-F8Z=nBw1pWUg`oGG* zXQKb#72W@iRQ#ajH4-uhw>WlnB8lDEy!P%c)gcYv)Rw5=2UITvRGS>O$N{K_sOCz! z6O{9DBHd3Qdfcy+HMThVS$kq&dwg$P(XJ%inSLL8IM)HWAt$k3xRbQn7NCx9D%WQo zk74!#+xKvX(Z+qtt5b5B1VR+AE@ zYhke0lyRX(dQ6kbN(JOPCKF&zGEWwcht^@s97Ad07lev8qd!asWO||5;{M~IxnXOD zN!V8h_cRr5sq=wkwuPKLGk0nTC{cE@_}mg94laCMLqG}H-VAWLuqEG~5ujA#bxi4ezJMN8vH0H*!b zrX>}DKq`}y%x9nR7c(z8S+p{%6x;VV_O$P*lkQr^y3f0De3J;iyTt;zrkvlE;gdF% z$smcLs%UUeQ3w;FU0-e1#Jh{vTdZw_4ATFfMT`B%(q!@fzbR<61N8q-q5m(S|Gy9Y z{}1s0za9Pm2hsn30C*qp4DdGKBiI9e5_kc)41Ah|0{;#86!0-Xb$>L}dzb%IEzf7Z z9mK6wtFm`jNuJUlWYv(?YfaLB?mJZih;TX!Ic$paXniJ|jdrrh@#=d-M*Bk&r8li< zPzKw1E!$diS)*J381CJ-;zi4koPX%VF-2ec^X2V? znpU=@^(hpxj)WWHkMkJ)_^UeL+t!oDM`4`$-0Z--KfEtZYg{{tyQK; z^_tF2yGn&wvbMISKSr0>-D=fRy0Ok~@8x(U2zfA|@0m|9Uai`WRy&(1zTKx_lKfm7 zAxU@-S+2G!kJa3zz9B{nO9BQC$8t7cTqZV@O5?^;Kts^Z8DQk9{@4-MXu4YMFw(pG z?+0rcPfmC)3IIF@kdGAMV5(}NO5F1?VC$_B2FmgU4Pi= zXxPCiuDP8BMqS!pAT7+YKSEmFANk0F%m$Tk{YOv_J5$ESOR9% zAR`&c8@-7vAl7Xoy5UtKz=8NG5lcUC5e(Ta-H5B8N#UA&rUK?01K~V|87%uSXjR5;LARxY}qL zrXcHGnD(|v%AcdTxwG68r6+T-g!k*vDL>VuR6Kkw(Mc(_JT)hH-;b!fcI8ky9coy< zafIeG7dWAT<*L;m42d>-iOH09_|@%fOWq236(`f>W{C)m4v)|b%;&auk=-1zAZFq& zpSf_eO};0}L6_1lGIo8n^GsM+0LXS3w*IFqI=jRJ5;T(^!Biz?Fh7DxkNUg~VR=E_ zXrf0nV#dspqwJo*>}_r85;y@&15>~s zzP3{NB=ApwzX3c4oCLlA_?OrhJ`B7Ecn5F-cq8yy;Pn{S@zwnoVcL7^K6A&KOfxG$fYNwJ@&$(r(dGk{9Dh?I0 zKbUju5I@0>u?^yL=6mLPzDIe^q52=NRkq?U=bL2bDo03eJi2jq3Z)tqKHu4#4cE;? z5~|FfH8q(LrISw-pA*g;ekKou&YMM{vU25husNqVz+ta?B5dQO{hIhu>YsDbsu9bL zA4zEs+`;?fS(i6#Kwj`}w;=Rv_9ihfd&v8-M z)SE-A>`99B_gCs|j=5>g+g35$=GMxhnvljJt5qz`1D04BiMG*XUDDi1I2Zu2VI=#I zcmJ5=p+g1K$UIGb?&QQ|Y>DobAy{884BLZNkJM3lCiRGbTII-*BL|+PWwkC@-W#l( z#rj~U8B}Pm6lWc_CyhMwNnA6JFqG5c2{X%Ae4A#uFzvk`lvfx8$rB7G3b)P%U#k7NNLEE2Pb7+YQ)vM5v zP>sdU+gW5n3XxTgZC7gL!S%p+S{o_DX3R3Z>YvK$Ve?iz77WhV>lrJB$}ucWkz-#V&>??LkJ^L4Ct&AMj`*N9dj0r6aFpV$mymk>&IZH+_GW zLj#R~X;!IJIp)Cgb@m^0hiz@6ANZLd!rS~Ov#taiJ1Wj$gA&1Er;e}$qK$2WNa_md zk80GZ_#vN^+9!uAoJlrLB$Qn!k%kI{uydaL#J^yALjdAxeuozmCGYJ3fS&O=db&9=VV<}*7+#1Lf;W#VdeB}j)U z#rXnS!Q)v6ypZ(#@B?eJC1qh?6f0+G=?*6QQW#H&Ctcx*wfyC|cU0a$->`CZyeAA0 zKL&kP{@j8bmNw%+aYd$e3@bSY!VA@bV%^s%jT+6?f>y(C*d#2eCpT`83N*e|1_!Z3 zSNX3%RaNz^1=p~l*1QmqNP7>AVFspGa6<-lKYiio5q~3i536#i9up23&OPt)%Pksu zZ*B|ckV|wpDO{e`y!9M*aZDI62RbTR_lK6MUGqCG%+T1{ABI98Feia#9FD0MuzoZ} zGQs>%;*hzzQ%2JO+Fkd%&*)-vazJ zc7d+~y4VC(fiu_wP61QE*8^VwT)-Z17I*}BGw=<-hp`L12=sskFbf<9-i?jm+koeR zZwCGl`@sJJz90BL;Libbzz4AnJO!Krz6SUs>;bg_ z?hX&}vSfms(Aiui%BXS04j=U^N(w7*2v3aBVUM%uJHxA-<|^USKOipVrACDPiUhSY z{$Lm_7H0?}C<60>i8(u5v;Bf#-Sl5G-iq_>WQC8PCi(;+D1{Y)DD4g`_w}1ag>+ z!!0H`$6tk1|B~70_aXfeA3-2TDC7Kd#1+H}Rtg$ECUtt=X#$OTS z1m}Bwr-8nf_Jh&G7HK_b(EtnnAvMRB7GI8TeqH4-czPQm9A+5TE>L`Wdru;Ymg#BJ zFS*b|oZn@cn7q8H5btnDQCpEYR63#l@ghT2`ft< z_STY8iP(;@60wiYK}~piklX^fy6bRt4K*)d`I-?8*srMR!X!nvB$Q~~7{3v$(Dl+F z#<4=2~jIk5&<1#%9a|Ni4s^VezCl<|)dX2Pg>GVMrchLfY04SO&}KLS_l z>7Y^X>};nNV$gZ9v+C^(8Wb1*{KX(6nF1xW=@2IwNwl#Y`s|rDC_z2n8L>s%E{S?<#y`wVj9)jBbr!I2n0u_*s20d*Tg3xKth1il{;_aHIB?*kQL71|3XxYaAnIXJ$GLPjA^;H={7tSxj$hvKf zja(Jun>ECwghS9Poe(a-sA7se#VBU^LHRF+yj35qy~dqQ{ajl+g@c9(aC+_(YTM~k zVr-{RGt6r4Dai~p1bHAV5GE?HJ_N0@-R)kPajQICn|{J3;^c7gA!*2;5CM&d$2=c;eZ>Wo<$(%NQi#@$Z+RA5H0sChzZ z%SNns_*c*q*~wtuEZ3ejXbSDelGgF6TMC+ZVoSQHj?kCcMik;4^Hby=XTtj5&1V1c zQ;Tunv3i}h>)B4D)qMQH2TvR~43Cun*J`1yl%ahlXfzbC;TD%LAU6?4?Yt9fX*5!y ztn0*CdwR4BROtmNb#~z3y%6N}M3>EK$l$g|=NN_}Au8 zN`f%nKu5?*Mt)TlAJwEU)FwkCM;K!qL}XA z82$EZUsEyt{|Wx+^9;KGR{$STV&HYahp-L&2=EVp?*!fhd>Qbiz{ju={1@O4u?xHx_;%ni-~@0V@EYKg*akiZ zJPoV>%fJlq7T{;F6Z|yr-Pi@*3;ZtjfbRnSBJeWsM&J#=>jBvY{tob7;0h4!15-k00LF_W?&u{G2a@(NR|<%bo)=TYY4R8TzRPEqSKR#4A{sk^ zN*Kh_9voZC80J&60+j5hELRvY){#>X5U57n!XMvEVRU7jR){aTS%KTZV35g#$nt|d zfTJ_uds$1)2eU%NW3f3_7|oj2rQRNi#1KU|y{BxaQAcAx$LeG{tgThBb9b_=Ed>LO z8DXq?^`YaBxoWnRoOtMAS5>N`nDVIF9l|AXbddbTRhtQ!y-z+&+(N8lGwt-V#O1~G z?1|$KKB^a)7+pC)#zD9-2Waq9Qb*7sXl_f%m+VY$(Nojfc{K%;z zg}fFUCHW3Znfe-S9nzil9Hx%AD3x~NC=#OTbwak-lz)VSfJnmG-R-o|HR`xdkim|R z6dN^=RL&XgTj&D2h_Cr38dfso(Q@@c>K_r}B`zVV`PO)0MUO~C)S>4{8hl6TU5=hs z)6;iO$hqpwB>~1_xhR}P`9xYLhfHHtZ-a~iZ-sag$*#8<)p@rhc&*tbdlcgG;d@JL zLy}j9)$|Z`WF64x@LK^!4O^^wI7$?1nUpscmyT>8a@{7BnHKX6Jjc9XKRzPjq+zf; zSJTigw3Mh#HcV=N7LN zmssk}XULmM9+^&II$cWeg0wgQRTk^2Ox7GFA8ajZd|S?p zy~wT7Ce|t&*Ve^?Cckg;s_gQ9@T;y+rR(>$UA48dWft|b7Z&W9+WEzcPd``9H%3)Ytq)u6=IPog z9r3QQbGml*(T7(bez3;#p}O$;wWm8wQaq0E=b`~gzp+IJXAmoUev{(mr3)pGfFx2o z$()EJw%{{0r9_E>i^2a~)D=2$eNRMDe4pFF)gY^9YzMc}28G&Rws{%HdEN%VrH-lp z%fp_O+Ct*r-Q6uj{@|0q4umlXv7P;m%=ad4v1+^(e%E`7R?-lvf?AmthF9xe1Imsu zW=lr6qF0m*UWpNnrdScJ6UBJFz%IYsO>#gr`~4(Ep__D)6b}@Q3x*Uk%rL9Y(Q0%} zI~bUyznl}r>rjk??#4!|L9|`&2lUR+;JM8`M7f2dM?4yf79N#iiJ3qQJJG&Q2@0e+8opjKN*Jv~rdh(%e5DrmYBQ%JAj1`4nsS z0#eJRyb`{%t5ZtVi0WLS~Bp0(ySnI0ByARUq&k^Udadf}It8oT{CB>Zq8xUGI< z+gIUvp@vG(jJ4PVmSr~jlAsh;+CUe-)P|xXcs_6bxWJY!M7LbQCE)m;}!eG3SzE(H^mOEPP2-wlBMeSR3#(>%iPqg zdvTZ!)793-YN%t%KWn~t*u`Mj_ZQ5OM9)`=v~M-`{18|lW)--e^7d5bWk(zFm&G`c z*aeez!|6Ma+FarqAf6h!>qnaLDBS=Xt%15?_u^6z!uoYxK|LKDI4-pTj%d-|LtIlt zkQukve>JgQY0UWykq-JaH)X&dO%?%6E{rkTBRRytxtjG=47Y5vC1=~<59V|DaN ly;z;`2${t^f6VtIH!&cw=blMxfut-J*Ih~EwtQmp`2V3#fQ|qF literal 0 HcmV?d00001 diff --git a/content/docs/deployment/helm/drivers/installation/powermax.md b/content/docs/deployment/helm/drivers/installation/powermax.md index bc711c3db6..5839b58404 100644 --- a/content/docs/deployment/helm/drivers/installation/powermax.md +++ b/content/docs/deployment/helm/drivers/installation/powermax.md @@ -17,32 +17,13 @@ The following requirements must be met before installing CSI Driver for Dell Pow - NFS requirements - NVMeTCP requirements - Auto RDM for vSphere over FC requirements +- Linux multipathing requirements - Certificate validation for Unisphere REST API calls - Mount propagation is enabled on container runtime that is being used -- Linux multipathing requirements - If using Snapshot feature, satisfy all Volume Snapshot requirements - If enabling CSM for Authorization, please refer to the [Authorization deployment steps](../../../../../deployment/helm/modules/installation/authorization/) first - If using Powerpath , install the PowerPath for Linux requirements -### Prerequisite for CSI Reverse Proxy - -CSI PowerMax Reverse Proxy is an HTTPS server and has to be configured with an SSL certificate and a private key. - -The certificate and key are provided to the proxy via a Kubernetes TLS secret (in the same namespace). The SSL certificate must be an X.509 certificate encoded in PEM format. The certificates can be obtained via a Certificate Authority or can be self-signed and generated by a tool such as openssl. - -Starting from v2.7.0 , these secrets will be created automatically using the following tls.key and tls.cert contents provided in my-powermax-settings.yaml file. -For this , we need to install cert-manager using below command which manages the certs and secrets . - -```bash -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml -``` -Here is an example showing how to generate a private key and use that to sign an SSL certificate using the openssl tool: - -```bash -openssl genrsa -out tls.key 2048 -openssl req -new -x509 -sha256 -key tls.key -out tls.crt -days 3650 -``` - ### Install Helm 3 Install Helm 3 on the master node before you install CSI Driver for Dell PowerMax. @@ -138,28 +119,6 @@ kubectl create -f samples/secret/vcenter-secret.yaml ``` {{% /tab %}} {{< /tabpane >}} - -### Certificate validation for Unisphere REST API calls - -As part of the CSI driver installation, the CSI driver requires a secret with the name _powermax-certs_ present in the namespace _powermax_. This secret contains the X509 certificates of the CA which signed the Unisphere SSL certificate in PEM format. This secret is mounted as a volume in the driver container. In earlier releases, if the install script did not find the secret, it created an empty secret with the same name. From the 1.2.0 release, the secret volume has been made optional. The install script no longer attempts to create an empty secret. - -The CSI driver exposes an install parameter `skipCertificateValidation` which determines if the driver performs client-side verification of the Unisphere certificates. The `skipCertificateValidation` parameter is set to _true_ by default, and the driver does not verify the Unisphere certificates. - -If the `skipCertificateValidation` parameter is set to _false_ and a previous installation attempt created an empty secret, then this secret must be deleted and re-created using the CA certs. - -If the Unisphere certificate is self-signed or if you are using an embedded Unisphere, then perform the following steps: -1. To fetch the certificate, run - ```bash - openssl s_client -showcerts -connect [Unisphere IP]:8443 /dev/null | openssl x509 -outform PEM > ca_cert.pem - ``` - - *NOTE*: The IP address varies for each user. - -2. To create the secret, run - ```bash - kubectl create secret generic powermax-certs --from-file=ca_cert.pem -n powermax - ``` - ### Ports in the port group There are no restrictions to how many ports can be present in the iSCSI port groups provided to the driver. @@ -213,10 +172,56 @@ Set up the PowerPath for Linux as follows: ``` >Note: Do not install Dell PowerPath if multi-path software is already installed, as they cannot co-exist with native multi-path software. -### (Optional) Volume Snapshot Requirements + +### Prerequisite for CSI Reverse Proxy + +CSI PowerMax Reverse Proxy is an HTTPS server and has to be configured with an SSL certificate and a private key. + +The certificate and key are provided to the proxy via a Kubernetes TLS secret (in the same namespace). The SSL certificate must be an X.509 certificate encoded in PEM format. The certificates can be obtained via a Certificate Authority or can be self-signed and generated by a tool such as openssl. + +Starting from v2.7.0 , these secrets will be created automatically using the following tls.key and tls.cert contents provided in my-powermax-settings.yaml file. +For this, we need to install cert-manager using below command which manages the certs and secrets . + +```bash +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml +``` + +Here is an example showing how to generate a private key and use that to sign an SSL certificate using the openssl tool: + +```bash +openssl genrsa -out tls.key 2048 +openssl req -new -x509 -sha256 -key tls.key -out tls.crt -days 3650 +``` + +### Optional pre-requisites +Depending of your Kubernetes distribution or the level of security you have there are several pre-requisites that must be met before you can install the CSI Driver. + +#### (Optional) Certificate validation for Unisphere REST API calls + +As part of the CSI driver installation, the CSI driver requires a secret with the name _powermax-certs_ present in the namespace _powermax_. This secret contains the X509 certificates of the CA which signed the Unisphere SSL certificate in PEM format. This secret is mounted as a volume in the driver container. In earlier releases, if the install script did not find the secret, it created an empty secret with the same name. From the 1.2.0 release, the secret volume has been made optional. The install script no longer attempts to create an empty secret. + +The CSI driver exposes an install parameter `skipCertificateValidation` which determines if the driver performs client-side verification of the Unisphere certificates. The `skipCertificateValidation` parameter is set to _true_ by default, and the driver does not verify the Unisphere certificates. + +If the `skipCertificateValidation` parameter is set to _false_ and a previous installation attempt created an empty secret, then this secret must be deleted and re-created using the CA certs. + +If the Unisphere certificate is self-signed or if you are using an embedded Unisphere, then perform the following steps: +1. To fetch the certificate, run + ```bash + openssl s_client -showcerts -connect [Unisphere IP]:8443 /dev/null | openssl x509 -outform PEM > ca_cert.pem + ``` + + *NOTE*: The IP address varies for each user. + +2. To create the secret, run + ```bash + kubectl create secret generic powermax-certs --from-file=ca_cert.pem -n powermax + ``` + +#### (Optional) CSI Volume Snapshot + For detailed snapshot setup procedure, [click here.](../../../../../snapshots/#optional-volume-snapshot-requirements) -### (Optional) Replication feature Requirements +#### (Optional) CSM Replication Applicable only if you decided to enable the Replication feature in `my-powermax-settings.yaml` @@ -224,7 +229,7 @@ Applicable only if you decided to enable the Replication feature in `my-powermax replication: enabled: true ``` -#### Replication CRD's +##### Replication CRD's The CRDs for replication can be obtained and installed from the csm-replication project on Github. Use `csm-replication/deploy/replicationcrds.all.yaml` located in the csm-replication git repo for the installation. diff --git a/content/docs/deployment/helm/drivers/installation/powerstore.md b/content/docs/deployment/helm/drivers/installation/powerstore.md index bfc47203e4..9e4009ea35 100644 --- a/content/docs/deployment/helm/drivers/installation/powerstore.md +++ b/content/docs/deployment/helm/drivers/installation/powerstore.md @@ -32,17 +32,20 @@ Install Helm 3.x on the master node before you install the CSI Driver for Dell P ```bash curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash ``` -### Fibre Channel requirements +### Storage Protocol requirements +Depending of the protocol you use, refer to the following sections to configure the requirements for the CSI Driver. + +{{< tabpane name="protocol-prereqs" text=true >}} +{{% tab header="Fiber Channel" %}} Dell PowerStore supports Fibre Channel communication. If you use the Fibre Channel protocol, ensure that the following requirement is met before you install the CSI Driver for Dell PowerStore: - Zoning of the Host Bus Adapters (HBAs) to the Fibre Channel port must be done. +{{% /tab %}} +{{% tab header="iSCSI" %}} -### Set up the iSCSI Initiator -The CSI Driver for Dell PowerStore v1.4 and higher supports iSCSI connectivity. - -If you use the iSCSI protocol, set up the iSCSI initiators as follows: +Set up the iSCSI initiators as follows: - Ensure that the iSCSI initiators are available on both Controller and Worker nodes. - Kubernetes nodes must have access (network connectivity) to an iSCSI port on the Dell PowerStore array that has IP interfaces. Manually create IP routes for each node that connects to the Dell PowerStore. @@ -52,10 +55,10 @@ To do this, run the `systemctl enable --now iscsid` command. For information about configuring iSCSI, see _Dell PowerStore documentation_ on Dell Support. +{{% /tab %}} +{{% tab header="NVMe" %}} -### Set up the NVMe Initiator - -If you want to use the protocol, set up the NVMe initiators as follows: +Set up the NVMe initiators as follows: - The driver requires NVMe management command-line interface (nvme-cli) to use configure, edit, view or start the NVMe client and target. The nvme-cli utility provides a command-line and interactive shell option. The NVMe CLI tool is installed in the host using the below command. ```bash sudo apt install nvme-cli @@ -75,6 +78,9 @@ modprobe nvme_tcp *NOTE:* - Do not load the nvme_tcp module for NVMeFC +{{% /tab %}} +{{< /tabpane >}} + ### Linux multipathing requirements Dell PowerStore supports Linux multipathing. Configure Linux multipathing before installing the CSI Driver for Dell PowerStore. @@ -86,46 +92,9 @@ Set up Linux multipathing as follows: - Enable `user_friendly_names` and `find_multipaths` in the `multipath.conf` file. - Ensure that the multipath command for `multipath.conf` is available on all Kubernetes nodes. -#### multipathd `MachineConfig` - -If you are installing a CSI Driver which requires the installation of the Linux native Multipath software - _multipathd_, please follow the below instructions - -To enable multipathd on RedHat CoreOS nodes you need to prepare a working configuration encoded in base64. - -```bash echo 'defaults { -user_friendly_names yes -find_multipaths yes -} -blacklist { -}' | base64 -w0 -``` - -Use the base64 encoded string output in the following `MachineConfig` yaml file (under source section) -```yaml -apiVersion: machineconfiguration.openshift.io/v1 -kind: MachineConfig -metadata: - name: workers-multipath-conf-default - labels: - machineconfiguration.openshift.io/role: worker -spec: - config: - ignition: - version: 3.2.0 - storage: - files: - - contents: - source: data:text/plain;charset=utf-8;base64,ZGVmYXVsdHMgewp1c2VyX2ZyaWVuZGx5X25hbWVzIHllcwpmaW5kX211bHRpcGF0aHMgeWVzCn0KCmJsYWNrbGlzdCB7Cn0K - verification: {} - filesystem: root - mode: 400 - path: /etc/multipath.conf -``` -After deploying this`MachineConfig` object, CoreOS will start multipath service automatically. -Alternatively, you can check the status of the multipath service by entering the following command in each worker nodes. -`sudo multipath -ll` +> NOTE: to enable e multipathd on RedHat CoreOS you can refer to sample `MachineConfig` given [here](../../../csmoperator/drivers/powerstore/#multipathd-machineconfig). -If the above command is not successful, ensure that the /etc/multipath.conf file is present and configured properly. Once the file has been configured correctly, enable the multipath service by running the following command: +Once the file has been configured correctly, enable the multipath service by running the following command: `sudo /sbin/mpathconf –-enable --with_multipathd y` Finally, you have to restart the service by providing the command From bc48bbe499715f4a850bdcec1e26c8ebab420687 Mon Sep 17 00:00:00 2001 From: coulof Date: Fri, 6 Sep 2024 14:12:21 +0200 Subject: [PATCH 5/7] Typos --- .../helm/drivers/installation/.powermax.md.swp | Bin 69632 -> 0 bytes .../helm/drivers/installation/powermax.md | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) delete mode 100644 content/docs/deployment/helm/drivers/installation/.powermax.md.swp diff --git a/content/docs/deployment/helm/drivers/installation/.powermax.md.swp b/content/docs/deployment/helm/drivers/installation/.powermax.md.swp deleted file mode 100644 index 2d0283bdf616db9e380bf40ed1cc77c42d61efd9..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 69632 zcmeI53vguFS>GFsO#-Y9*pLcP)Io!HXJmEvXl9?v>=>@w+l#%vfHj8xCeI>Q0 z)!o~-TNk{34}ZdsZ>&-AU0rE97uvgC6K~Z z$?yN2bMC#}lICr*-s!ISM&0+Gd+s^kcg}ad*ZI!+@n;qj4%Mt9imZs^AKKhMt2K79GPXD^;VdHr(V>MN{ocI#QcQQvG0wubAqMt8g3 zOxx{xBipVvyN#^g=ynEaXHd5v&9v9{8ML~cdat{i z_P3L(we9Bhn=rXM!N9$Vf#G1I`sjU?lP6}63js%_zQMimE1$kMaWU!j1OpQcOfWFP zzyt#m3`{UE!N3Fq_b3Jiy*E_8jlO?j=>J0a`vqfv-!OLn%-DTQ2lja){QXPeZ>;Bw z!tZYie}8%G{%gbUKOO&u2jlavi`T>7uN=Go#bfuscI-YT4g19Uzj5q-y#6JjTy^Yz zydM8PGIsw<$LgOSyZ>cl_un>l|GselP2unI*!|as>u(KzH^=V3CR`7fD?dDTKir=D znP6aofe8jC7?@ySf`JJJCK#AtV1j`O1|}G|M=_A#FnN_^D+KZC;md%%AL zehK)y!1n^*1H1!e^f};3;2iKjQC?pL+Q21X4tN0ga^QDRe18FWFK`z4e<;e&1Ahki zYTz}%7Xm+yV*O*l2Z0X&-wI^F*8^V-d=>B};D4d0zYJUh-Ud7X+z)&Y3jV{uzs3;o ze=!#P5%9afuLA!Tcr);gz-KTt{3tL7yns>S*HN5*6XVBk05iZ_fUgH$3w#kEyj+V) z_1zvL3(Je{jKwp!jc(taAqv7RDU{(ta@A$SUa#99WbWd*Wk-<2_OR1xShz#Yo$X}9 ze(DBW$$!&XcTm4k=5}8XVvz-pydP?=F+v^Tp2%B)ltLG|}cfIS) zZjh|Itu$$-{l}+x2{xK~~QO-F~u})~z%+X;%ZT6frVEwD!zXy&=yB(J%jjb`FCjEZ0S99|aS})nx2z8|0N}F!bwE{lc#n+u) zZSjVK>)j=m#vaJ%i*=|*=f{VMQ3mR@2Cam?WU5Tf=yaPPK%=G89k|5x5^&Sqa7ky+ z*;|DaBjVA;_^^9YxH}fB7#+L5o3!BV!}YY^Ne5K#Uy^MRRMd1^$xiB$MkCEKca;0P z-ToC(ARi&egn^_pj{VT!1KvhQm<(G<<`$Re;vnsBBvfB>7n079K%qU?=%>jb)oog? z7HQ5Ijnp*%d}sng#N&VTMXil|SlkqPh0$auX|>Tu9%XRFY(afL}!u5a0oR6VtC zWPnG{R5JL(3MERDNO&7XOwOx(J!L}1sj`nTB^v1s!;C6aEBp{KLJpZ-V~r9`Ld}sp zIkU6@moekVS((*w5HqW~#SQ<`;WlGb!+odQ9*PrK10w!=!}WG6+wxP=H?Wntt=8t& zj2`dgtj|-DAhg!@aNBkATF*ay(GNA)Z{I!j#Jo+reB9OCa_S!iA^mQz&(us+%!!$a zkB5F6f4$yI4jPiG=hrjY=98Zf`NaO_ zu5n7RCc+tScw(5j`DW5%>hV5x)F!d+pKGmC*}_)B1aEt;mowDJH(BC(Z8!x3{pQBn zXr-Uq6_R%|W5$lvpBH|gw`rYjvR5zeg$<) zp@r^pNo+vMan7x+tvUT|4Et>t*Xe$;Tk~O?G_KE!6{l$nwZ&%If7?mM2&iX`eyhh~ zy_pWGx?Vl$uDSIj+X|J3ua}2@5Q=)hsMuu!8f$!B)55ddZ}x8OM-MV09y8etx8kDHX7;kiWq4F@J zmJ8h>j7t0~+4OQDOw#%e_kho6s32T1l=~EHA)^?RrO@b{~RI`xw2d#}3Vs7eo zl6I@<>k@*W?zFO=Etr?jF0Z)xrA3G2*Uos#<9>VC9<)R!)i>5W5ZZN@&v**#Ec<%8 z#1Xr53*%)kK64>mSwKiAPl>8Y;bWKl`ZRWNlq-$h92;hgvM`=Yi#mP11Y20%YHbYM zQKV@1d{|yVr9z)I@Ar)X)_gBS)Q<%EWmV==wUP9a^;R2+FU^h}vz5HyBIVZQ@@q+9 zX!+Lp;d{8Mf`q^nS~4a#Y-KHED-_A=={5}{J7gO_aOu|iUYKlJEEq*QXa`L{$#+h= z#uoA@@+s^gw8#&Oz+yDM*1OEZ5)xImz+#UuamLU#(%?DJYdTCbWk9xhOIeu-E8;v6rQSNHDizAooV0nmTDC?xp*l0 zbGc`yl^B{T2=K;m860!&bOn}kWzZV5)44*Stc+c#RC$Z^|8GX8-vOlmM^#t9hq^ZT zOfWFPzyt#m3`{UE!N3Fq6AVl+Fu}kC0}~8PFfhTuZDQc`#qI#rJa`~*Of|8iwvsl^ zLw*b#cCqbNX6DC@es|j(ahON(--h`2vA0k>txNM$_L4QwHW4TdeQ)#9f z3HGst{7u=iy~*_d573LBmEw>7|4pGszlQJr-vNIgpZyO5pTtN137`$U9rz~T7x2gb zEbtNFuL6GsXafmwAMiThwZO;m&wmWq0ycpsfpfqo@z;L>_zmD!fu9F{4)`GO0pMTZ z%l{SNjlfp|pTU3s$H1q7-v`bDKZYOw>wu5q%l}E>Zvq`)1Nai)Q~39P1Nc$kgTPBb z2lzMm_WwCB4}6-we**YL;O7A0;-`RD`UmU0%=9v-yPcNX-plPwRh-ZR%ywCAFWGKe z0ScPQ{Sl%Enk^qT&`Ec5zi1PSFiuJt&Vm6>7T95MlWGhVc0>HbxS7q*Q+c=)jgUHG z6oQr39FnugYsa10Yw>gRy10>ISc6P{4g7ysWL(7TR+x9Ir)AOVxub7A=J-Bu@dY?U zixM&i=N7OhpI)rR4%**FlhJ;lWzF4xzuUzh;i{JoM^^6Z7`Nk@mCl-a%PS5esY zswZl*v$f;(aJzh~?ylZ=O88#XF>@xA)}TQB=gVjR!9D2*kig-{b<$ zm4ab;&t@V%BgTUfHYW_8%pFG5CO-I)9uVTub=8iWDr%kbuA%hd2d!kOL~O`di8z>P zvsXXleANJ0%2^Kr+Ct<+2f-%Bmt5r^vCL7TX z3h1$U20@@~(NMIz3)sN%EFWRXKXBO053YhyXJX0kr#JdYEIb#N&txvdsokn6bVVYhL+9!I!3%uN}I+u9*+{h&y_zr!kSVkD0f(RF6FB z@;v2Erz|>KS*QsX1JjL^(FKoq-d06I@%m|P-x!IHW%VQ7@UE(lTB*icP_T#qWKiJU z9&6d68*}EWb?!g7=-bOkg>d&TA;mr134B~##4z)14yWwtG+TN&U zt<4Tog~1XAskn=eM1W1F#40uA8_g>TQAj=F-PA(8xLC?8UI6!Oc=K9zrPW&~UiwV# zVqYtk87Uu%VPsOQZY1q2UA6UysK9=@)9MoBQ=XnK84S|x9(+G8L26`=5KljJLi6y- zt5ymPaz`_~TH!unRjkGed|%a%g?VQ*gw`4n@+gNux%4Uv14BXqg298HFO%+iz1-bu zK^>+c>B0y{pf5gvDbyRVNHD04BZ$E|gR13;Bz_+Pb#nAu3j+;tFRPu`dwTSs^78re zQ+m1Pq=tSn7=0ZG-d0qKMwoi#`DAQvz-%6uQp4L$FlZmTYwfhC95m}NJmMe=uK#AM zY2?^wA<(ITajZ-JId~p;(ZI?Io5B{s2-59!ZMJp-MJc@2lO`Z7v|4ek>nYKO+eB!? zocucD$JYOl|J6qeYyGVGm=CW=6eQ6fBj`Fvzi zzIcfPd)ybOUky7H=7ssi=Bn|L%cSq5<>N%F5ERN!{%&ZN12iFBWvC5~CGsnQIUbZ0eOp8;pjv?sNCK z=H9^ycP-ZyJ%+2ZwV}5s)%vLy*hHIO9G~Y2nJ)Kw-M+dXU zt48O@ATFqhC1^Y8?;S$U(YeAPFiu!ZzNNS!YI$!Pqhr5y)h(WxaI-t6WnMv?e8-E= ztmavnFxgElP=2Ke%bP`Yiq`~DOGby$VFTMMxL#m-d8_XEFQof5zfAsqjxaES@zM{?kEQ6xV2M9dM5`Y_ z&B>*^je%>dyk-?{QnS!+(Exlk{r@t0{9hl_|DQ$w{~7fEuR{NS2L1n6@xL#i|9=?$ ze+vCy<-dEP|KFuZ!NBl(B z!-tDEFX)p>2XCh8-nIw9{gd$@+utXD2lG5>g zs=RmMzBI29t!@m`y+hPN+~|z!Y_}yC3TFE;+i*&kX_3%5!bDufrVWab6XU+`=Xtyym_# zjHKn*t0sXDC7<%j`Q?3n<$|7?o4Fxi%;kCSNoZkSO|9T2p9&NfjrN(B_OU!o9tDe( zi<-%4HCpsnixD&VcIz0hS$2(3fMDP!?ii{JM4gD?IALM&+T<}z#%U1S!DGw6GY?-D^U}N!! zo-7h=4pc(&7n!=W2ZC&6N=KulJIE1}&uw8KE=b_i8w?a>Oz`J|WiC6@>O+%oDTLeS z`8Jt4$zW@A+k@77?@BA}lEH@=r;a5>)o*+9wp#q^e<;#~{Ym|LyR}}A1ki*b?cX^I zse~}47*Q_gp_ZhgBrt=FRJKWM&P}bHeQsq+am!P0TfTI0YGOdU%`p@S6A&86aUP3) zdp|YWx@`GmM~RX4``!N3jGNj``kkq|{lT1|E6R7Jzs+=gxH%!y4XBh;!=uZV6Sf}_ zBG+F+r6Cvz&!+zp7j`txe5;VLDuv#L*zstDBL$dtKC)S(ZBm1_|0+eBI~r3ndO>4` z^tD#fJ=02e2}o9^P5vk>Anf8;w4e^6Df;BwL~770O=9p-8a6+<7`BUfsbkOfm*kQ7 z!%hq5(I}H8jF_Bita&3*2|@ILxtjD5EmUT|DvK!zjwa8d+0@b;3x0-AgE-&3Xryna zoz0L+(e?`XGBVK&w#b=gd%Ti{ww%_$xS|F5Gd2Ijf?wg$KR44Q_HitA*_tKcV5PLK z_q4&I=kHMC;s)EVA)1A+`(X1%0@d`c)UwF2HfjHZ#BryG)lcm;&|4G}PMmkrhOEty zANgJp!mcw};zSuiu9(*WYyGmt|LnzEr8VEy?CZQ`a_e?55NUJ3S}*;-jvjwyO#i=v z{{NrQ|IeZSe;ED$0rdaxK>z;(^#3nH|5y2UPW1m<(f!Sl9L>6(d1p&$|Du9tuQ@*_ zv;>F=MYdR^kq#R{jFE?3&Y0S4@sO%$$!{kz*Uirkn-kX@E`E5FG4@=mk9v2wYU_FM z5toEvEx{!9wa{@{`Y8q(wCf$B;7pENmuIwgu^-3i+D|vJ8`yS6vfs0zo|UD|?&f>- z6FJiq%Q$+v#XPcaHthOlS`eya`}8y^0`m5v11z}7kWu`?Y=n~TuEM^qC@jbZ&~B&O z4I{TUVn`rExe0**@k;Q{#fQ&yI}fm7k*oovK{mO>i&N~zD*cM7;>!bNbW_&7OW?7A z9qFy)0Ll$w@iiNn3A?%I7HRH9GmJ8JI|^{N*#O~;4edhsj9c~xr66YNR_4|&Vyl@U z=X^+KqM!tea=O%Mq|dUE{oJzbKx<=7yLW#^OXneo@)EY+d9?7<{F2Seuz@yikSz3q zO|rb;wzi`hLRxyA2ji2olg~R*GLEs6NAjJfR(;8@XUsL<^Q38QC4%$3Fqwvg(ZQQu zsvz!V6|)ZRS=DkS-E&j0<({u8dKF)>Lm*HnxTz~LMf+2N&Pa)eAZL|k9<*YUab^=R z4N(pt(jp`qq*RwphjT?XnKy%7?Ibm97F()?q1+Xx)HLFJ9`&Br4LiJmjkMVnW*YRx zqn$Aeee6`oP|g7d$@aa#?Di4FuMyJDFKT};D~n(BM~=|3&3K9eL0^1Uahf_$M8CC# zUWR}Wl0e)78;Pv~FAl1Va4wn^egl>|c0Ie4J^Fe8Zb)7mzil8|wDULScDq;X7{9}Y z?W2frD_X2X+?MSALH|q`uL1EUS0A*H2ZV~DnYe8hnEK+AKGRM#xJ0o&x4i62y{7V- zeh4i)vi5X4Gt5*SgH)0vXwBeNBn<|doQ+X6u%b>TSJ+ZKcZ7+uEL#iXS{5J7mSQfk zn0zmGf{J-$ROs!+c~Zd&7qSJAovY{PN+1|nrbrQybGvg3Oe$N5#OT5q4>LYA*DFtU zc9cIv0>L4FGxW+2rP35kipZQ`WxKH!H`~SZS{r|tC|38iX|10SJ_6&>BpxaJyelnz`LV4+J(X2+sVrqfq2zU^;oRfuUm>@kso6pbR*2NMhO!rc86`6JPyWfhfSf-;Eid$#Kzx( zBxg;fsh=4o1eu?ZAv85&YPJbB?+yl{X~vGH(_D!eIJT%aHz$+Fx0c)?d>9_t5|U2m1dy`u{JZ|JU&UzYqQYGwA=D=>NZo{;y{) z>ABKJc~BI3e!)`Q^*wbteKVx@blUh-3Qac+RMReOzQyc9rkcs;jxbPA*T<|YR4Rtt zn57L1Ts)w9Bxo%d^DkmBP&ldCwOk~9Nj}&PBD5%efWHTIA?e#PI#W@ zSC08?Ipo^I6hQ8NA7{2|8DM?h&oO+K(*#-?)0nF-EJ5Dd`2fOoVOwLd^09lnE$#DCM>{lq+;a>G_n@Ezdkx+ucpdHRZVTP||B1>^T(40r!Xb8{@A+kFbR>$_aK8xj3}~*U7CFM1`{aU|Dj#k#i)7b(Dy&Y%KlB zx9*xFODglj?`t1)iKFdq?k#wBQ&K$-HKD~vB8!srOv-rdpnUVS$stcf@8oj}7}#e* zo7gsQ0ehdtqWcIl=9ZanDtmbDlj)MQnKhZz^Q=Dk(G8!)_T6h$2YN*QRrXbi6lL$_ zgbttG!7)OL>1oY^8|wcmG|~L;E?o)@QTl(1F8@P{ohby#ew3uzLPb{Klq0 z??XTn!i`q7Afsan41~(@;Eza7iPE0vB~r%#&yc^HoDm!)hd=Js{V7=5_0wqFXv&n0 zlf-GX2w9f5Iq3g7x@`391aGm7scT-{IKG|Ug(W|CkQu%dCyPp{Kk^kHj@=1fZwKdx zJrihQ@tDT6?qF#dodqEnW3s#)hWZ;`zwXj9f^5IM%nt`!-G1vOV^8~*m?FHgu5)Ya zgr^U6NNm|YUmvNpe!Vp&;rzt%0RkM4*-G1XmTF*(IBj3h)IFNFc8-ik7I0*oa>u5IpNow z{l)o)fjp%p-L6dNa~NXp&h-*`8k`6^N**0@xol@#EpJ|4*UEf9II~FaQ6?(f=Pq|NjvB|7-F8Z=nBw1pWUg`oGG* zXQKb#72W@iRQ#ajH4-uhw>WlnB8lDEy!P%c)gcYv)Rw5=2UITvRGS>O$N{K_sOCz! z6O{9DBHd3Qdfcy+HMThVS$kq&dwg$P(XJ%inSLL8IM)HWAt$k3xRbQn7NCx9D%WQo zk74!#+xKvX(Z+qtt5b5B1VR+AE@ zYhke0lyRX(dQ6kbN(JOPCKF&zGEWwcht^@s97Ad07lev8qd!asWO||5;{M~IxnXOD zN!V8h_cRr5sq=wkwuPKLGk0nTC{cE@_}mg94laCMLqG}H-VAWLuqEG~5ujA#bxi4ezJMN8vH0H*!b zrX>}DKq`}y%x9nR7c(z8S+p{%6x;VV_O$P*lkQr^y3f0De3J;iyTt;zrkvlE;gdF% z$smcLs%UUeQ3w;FU0-e1#Jh{vTdZw_4ATFfMT`B%(q!@fzbR<61N8q-q5m(S|Gy9Y z{}1s0za9Pm2hsn30C*qp4DdGKBiI9e5_kc)41Ah|0{;#86!0-Xb$>L}dzb%IEzf7Z z9mK6wtFm`jNuJUlWYv(?YfaLB?mJZih;TX!Ic$paXniJ|jdrrh@#=d-M*Bk&r8li< zPzKw1E!$diS)*J381CJ-;zi4koPX%VF-2ec^X2V? znpU=@^(hpxj)WWHkMkJ)_^UeL+t!oDM`4`$-0Z--KfEtZYg{{tyQK; z^_tF2yGn&wvbMISKSr0>-D=fRy0Ok~@8x(U2zfA|@0m|9Uai`WRy&(1zTKx_lKfm7 zAxU@-S+2G!kJa3zz9B{nO9BQC$8t7cTqZV@O5?^;Kts^Z8DQk9{@4-MXu4YMFw(pG z?+0rcPfmC)3IIF@kdGAMV5(}NO5F1?VC$_B2FmgU4Pi= zXxPCiuDP8BMqS!pAT7+YKSEmFANk0F%m$Tk{YOv_J5$ESOR9% zAR`&c8@-7vAl7Xoy5UtKz=8NG5lcUC5e(Ta-H5B8N#UA&rUK?01K~V|87%uSXjR5;LARxY}qL zrXcHGnD(|v%AcdTxwG68r6+T-g!k*vDL>VuR6Kkw(Mc(_JT)hH-;b!fcI8ky9coy< zafIeG7dWAT<*L;m42d>-iOH09_|@%fOWq236(`f>W{C)m4v)|b%;&auk=-1zAZFq& zpSf_eO};0}L6_1lGIo8n^GsM+0LXS3w*IFqI=jRJ5;T(^!Biz?Fh7DxkNUg~VR=E_ zXrf0nV#dspqwJo*>}_r85;y@&15>~s zzP3{NB=ApwzX3c4oCLlA_?OrhJ`B7Ecn5F-cq8yy;Pn{S@zwnoVcL7^K6A&KOfxG$fYNwJ@&$(r(dGk{9Dh?I0 zKbUju5I@0>u?^yL=6mLPzDIe^q52=NRkq?U=bL2bDo03eJi2jq3Z)tqKHu4#4cE;? z5~|FfH8q(LrISw-pA*g;ekKou&YMM{vU25husNqVz+ta?B5dQO{hIhu>YsDbsu9bL zA4zEs+`;?fS(i6#Kwj`}w;=Rv_9ihfd&v8-M z)SE-A>`99B_gCs|j=5>g+g35$=GMxhnvljJt5qz`1D04BiMG*XUDDi1I2Zu2VI=#I zcmJ5=p+g1K$UIGb?&QQ|Y>DobAy{884BLZNkJM3lCiRGbTII-*BL|+PWwkC@-W#l( z#rj~U8B}Pm6lWc_CyhMwNnA6JFqG5c2{X%Ae4A#uFzvk`lvfx8$rB7G3b)P%U#k7NNLEE2Pb7+YQ)vM5v zP>sdU+gW5n3XxTgZC7gL!S%p+S{o_DX3R3Z>YvK$Ve?iz77WhV>lrJB$}ucWkz-#V&>??LkJ^L4Ct&AMj`*N9dj0r6aFpV$mymk>&IZH+_GW zLj#R~X;!IJIp)Cgb@m^0hiz@6ANZLd!rS~Ov#taiJ1Wj$gA&1Er;e}$qK$2WNa_md zk80GZ_#vN^+9!uAoJlrLB$Qn!k%kI{uydaL#J^yALjdAxeuozmCGYJ3fS&O=db&9=VV<}*7+#1Lf;W#VdeB}j)U z#rXnS!Q)v6ypZ(#@B?eJC1qh?6f0+G=?*6QQW#H&Ctcx*wfyC|cU0a$->`CZyeAA0 zKL&kP{@j8bmNw%+aYd$e3@bSY!VA@bV%^s%jT+6?f>y(C*d#2eCpT`83N*e|1_!Z3 zSNX3%RaNz^1=p~l*1QmqNP7>AVFspGa6<-lKYiio5q~3i536#i9up23&OPt)%Pksu zZ*B|ckV|wpDO{e`y!9M*aZDI62RbTR_lK6MUGqCG%+T1{ABI98Feia#9FD0MuzoZ} zGQs>%;*hzzQ%2JO+Fkd%&*)-vazJ zc7d+~y4VC(fiu_wP61QE*8^VwT)-Z17I*}BGw=<-hp`L12=sskFbf<9-i?jm+koeR zZwCGl`@sJJz90BL;Libbzz4AnJO!Krz6SUs>;bg_ z?hX&}vSfms(Aiui%BXS04j=U^N(w7*2v3aBVUM%uJHxA-<|^USKOipVrACDPiUhSY z{$Lm_7H0?}C<60>i8(u5v;Bf#-Sl5G-iq_>WQC8PCi(;+D1{Y)DD4g`_w}1ag>+ z!!0H`$6tk1|B~70_aXfeA3-2TDC7Kd#1+H}Rtg$ECUtt=X#$OTS z1m}Bwr-8nf_Jh&G7HK_b(EtnnAvMRB7GI8TeqH4-czPQm9A+5TE>L`Wdru;Ymg#BJ zFS*b|oZn@cn7q8H5btnDQCpEYR63#l@ghT2`ft< z_STY8iP(;@60wiYK}~piklX^fy6bRt4K*)d`I-?8*srMR!X!nvB$Q~~7{3v$(Dl+F z#<4=2~jIk5&<1#%9a|Ni4s^VezCl<|)dX2Pg>GVMrchLfY04SO&}KLS_l z>7Y^X>};nNV$gZ9v+C^(8Wb1*{KX(6nF1xW=@2IwNwl#Y`s|rDC_z2n8L>s%E{S?<#y`wVj9)jBbr!I2n0u_*s20d*Tg3xKth1il{;_aHIB?*kQL71|3XxYaAnIXJ$GLPjA^;H={7tSxj$hvKf zja(Jun>ECwghS9Poe(a-sA7se#VBU^LHRF+yj35qy~dqQ{ajl+g@c9(aC+_(YTM~k zVr-{RGt6r4Dai~p1bHAV5GE?HJ_N0@-R)kPajQICn|{J3;^c7gA!*2;5CM&d$2=c;eZ>Wo<$(%NQi#@$Z+RA5H0sChzZ z%SNns_*c*q*~wtuEZ3ejXbSDelGgF6TMC+ZVoSQHj?kCcMik;4^Hby=XTtj5&1V1c zQ;Tunv3i}h>)B4D)qMQH2TvR~43Cun*J`1yl%ahlXfzbC;TD%LAU6?4?Yt9fX*5!y ztn0*CdwR4BROtmNb#~z3y%6N}M3>EK$l$g|=NN_}Au8 zN`f%nKu5?*Mt)TlAJwEU)FwkCM;K!qL}XA z82$EZUsEyt{|Wx+^9;KGR{$STV&HYahp-L&2=EVp?*!fhd>Qbiz{ju={1@O4u?xHx_;%ni-~@0V@EYKg*akiZ zJPoV>%fJlq7T{;F6Z|yr-Pi@*3;ZtjfbRnSBJeWsM&J#=>jBvY{tob7;0h4!15-k00LF_W?&u{G2a@(NR|<%bo)=TYY4R8TzRPEqSKR#4A{sk^ zN*Kh_9voZC80J&60+j5hELRvY){#>X5U57n!XMvEVRU7jR){aTS%KTZV35g#$nt|d zfTJ_uds$1)2eU%NW3f3_7|oj2rQRNi#1KU|y{BxaQAcAx$LeG{tgThBb9b_=Ed>LO z8DXq?^`YaBxoWnRoOtMAS5>N`nDVIF9l|AXbddbTRhtQ!y-z+&+(N8lGwt-V#O1~G z?1|$KKB^a)7+pC)#zD9-2Waq9Qb*7sXl_f%m+VY$(Nojfc{K%;z zg}fFUCHW3Znfe-S9nzil9Hx%AD3x~NC=#OTbwak-lz)VSfJnmG-R-o|HR`xdkim|R z6dN^=RL&XgTj&D2h_Cr38dfso(Q@@c>K_r}B`zVV`PO)0MUO~C)S>4{8hl6TU5=hs z)6;iO$hqpwB>~1_xhR}P`9xYLhfHHtZ-a~iZ-sag$*#8<)p@rhc&*tbdlcgG;d@JL zLy}j9)$|Z`WF64x@LK^!4O^^wI7$?1nUpscmyT>8a@{7BnHKX6Jjc9XKRzPjq+zf; zSJTigw3Mh#HcV=N7LN zmssk}XULmM9+^&II$cWeg0wgQRTk^2Ox7GFA8ajZd|S?p zy~wT7Ce|t&*Ve^?Cckg;s_gQ9@T;y+rR(>$UA48dWft|b7Z&W9+WEzcPd``9H%3)Ytq)u6=IPog z9r3QQbGml*(T7(bez3;#p}O$;wWm8wQaq0E=b`~gzp+IJXAmoUev{(mr3)pGfFx2o z$()EJw%{{0r9_E>i^2a~)D=2$eNRMDe4pFF)gY^9YzMc}28G&Rws{%HdEN%VrH-lp z%fp_O+Ct*r-Q6uj{@|0q4umlXv7P;m%=ad4v1+^(e%E`7R?-lvf?AmthF9xe1Imsu zW=lr6qF0m*UWpNnrdScJ6UBJFz%IYsO>#gr`~4(Ep__D)6b}@Q3x*Uk%rL9Y(Q0%} zI~bUyznl}r>rjk??#4!|L9|`&2lUR+;JM8`M7f2dM?4yf79N#iiJ3qQJJG&Q2@0e+8opjKN*Jv~rdh(%e5DrmYBQ%JAj1`4nsS z0#eJRyb`{%t5ZtVi0WLS~Bp0(ySnI0ByARUq&k^Udadf}It8oT{CB>Zq8xUGI< z+gIUvp@vG(jJ4PVmSr~jlAsh;+CUe-)P|xXcs_6bxWJY!M7LbQCE)m;}!eG3SzE(H^mOEPP2-wlBMeSR3#(>%iPqg zdvTZ!)793-YN%t%KWn~t*u`Mj_ZQ5OM9)`=v~M-`{18|lW)--e^7d5bWk(zFm&G`c z*aeez!|6Ma+FarqAf6h!>qnaLDBS=Xt%15?_u^6z!uoYxK|LKDI4-pTj%d-|LtIlt zkQukve>JgQY0UWykq-JaH)X&dO%?%6E{rkTBRRytxtjG=47Y5vC1=~<59V|DaN ly;z;`2${t^f6VtIH!&cw=blMxfut-J*Ih~EwtQmp`2V3#fQ|qF diff --git a/content/docs/deployment/helm/drivers/installation/powermax.md b/content/docs/deployment/helm/drivers/installation/powermax.md index 5839b58404..441264228d 100644 --- a/content/docs/deployment/helm/drivers/installation/powermax.md +++ b/content/docs/deployment/helm/drivers/installation/powermax.md @@ -147,7 +147,7 @@ no_path_retry 10 If you are installing a CSI Driver which requires the installation of the Linux native Multipath software - _multipathd_, please follow the below instructions -> NOTE: to enable e multipathd on RedHat CoreOS you can refer to sample `MachineConfig` given [here](../../../csmoperator/drivers/powermax/#machineconfig). +> NOTE: to enable multipathd on RedHat CoreOS you can refer to sample `MachineConfig` given [here](../../../csmoperator/drivers/powermax/#machineconfig). Finally, you have to restart the service by providing the command `sudo systemctl restart multipathd` From ea741bafd1f8c0da41b157a6977693e244b584a3 Mon Sep 17 00:00:00 2001 From: Yamunadevi N Shanmugam <82038610+shanmydell@users.noreply.github.com> Date: Thu, 12 Sep 2024 15:40:58 +0530 Subject: [PATCH 6/7] Update powermax.md --- content/docs/deployment/helm/drivers/installation/powermax.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/deployment/helm/drivers/installation/powermax.md b/content/docs/deployment/helm/drivers/installation/powermax.md index 441264228d..b83c5ba610 100644 --- a/content/docs/deployment/helm/drivers/installation/powermax.md +++ b/content/docs/deployment/helm/drivers/installation/powermax.md @@ -147,7 +147,7 @@ no_path_retry 10 If you are installing a CSI Driver which requires the installation of the Linux native Multipath software - _multipathd_, please follow the below instructions -> NOTE: to enable multipathd on RedHat CoreOS you can refer to sample `MachineConfig` given [here](../../../csmoperator/drivers/powermax/#machineconfig). +> NOTE: To enable multipathd on RedHat CoreOS you can refer to sample `MachineConfig` given [here](../../../csmoperator/drivers/powermax/#machineconfig). Finally, you have to restart the service by providing the command `sudo systemctl restart multipathd` From 7202c0ec796dae8c3dca5d5c53044656e789743e Mon Sep 17 00:00:00 2001 From: Yamunadevi N Shanmugam <82038610+shanmydell@users.noreply.github.com> Date: Thu, 12 Sep 2024 16:13:51 +0530 Subject: [PATCH 7/7] Update powerstore.md --- content/docs/deployment/helm/drivers/installation/powerstore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/deployment/helm/drivers/installation/powerstore.md b/content/docs/deployment/helm/drivers/installation/powerstore.md index 9e4009ea35..992aae544a 100644 --- a/content/docs/deployment/helm/drivers/installation/powerstore.md +++ b/content/docs/deployment/helm/drivers/installation/powerstore.md @@ -92,7 +92,7 @@ Set up Linux multipathing as follows: - Enable `user_friendly_names` and `find_multipaths` in the `multipath.conf` file. - Ensure that the multipath command for `multipath.conf` is available on all Kubernetes nodes. -> NOTE: to enable e multipathd on RedHat CoreOS you can refer to sample `MachineConfig` given [here](../../../csmoperator/drivers/powerstore/#multipathd-machineconfig). +> NOTE: To enable multipathd on RedHat CoreOS, you can refer to sample `MachineConfig` given [here](../../../../csmoperator/drivers/powerstore/#multipathd-machineconfig). Once the file has been configured correctly, enable the multipath service by running the following command: `sudo /sbin/mpathconf –-enable --with_multipathd y`