Provider Support for External Ceph

[Nachoprojects]

Feature request to support external, pre-existing, Ceph clusters. With a potentially common provider architecture to have one existing Ceph cluster, it would make sense to have the Akash provider operator be flexible to support this. This can be done in one of two ways to rapidly support a manual configuration with a final evolution of not requiring the Ceph cluster to be connected in the rook-ceph namespace.

First, is an adjustment to https://github.com/akash-network/provider/blob/e2e52140960762b682c15d5fd67040a7a1527aa7/operator/inventory/ceph.go#L285 to support the ClusterState of "Connected" rather than just "Created" for a Ceph cluster that was manually imported to the rook-ceph namespace with the Rook documented process via https://rook.io/docs/rook/latest-release/CRDs/Cluster/external-cluster/external-cluster/ which required some modification on the auth resources to appropriately configure the client.healthchecker user that rook uses with higher permissions to appropriately create the akash-deployments pool (if desired).

Second, is a full ability to run an external ceph cluster with the standard CSI RBD enablement without the need for rook-ceph entirely. This would require the standard CSI RBD setup via https://docs.ceph.com/en/latest/rbd/rbd-kubernetes/ with the appropriate adjustment to the storage class(es) labeling and naming for beta{1..3}. This would also need to have an external user creation script to import into the k8s secrets to appropriately run the rook-ceph-tools pod inside of akash-services (or other pod with ceph cli) and connect to the external cluster with monitors configured to run "df" commands and so on.

Thanks!

[Nachoprojects]

Specific to supporting erasure-coding pools I had to make the following change to ceph.go and then there is a storageclass item.
https://github.com/akash-network/provider/blob/e2e52140960762b682c15d5fd67040a7a1527aa7/operator/inventory/ceph.go#L316 changed to "dataPool" which coincides with the following sc lines. This change is purely for the monitoring aspect as it will pickup the EC metadata pool.

parameters:
clusterID: rook-ceph
pool: ec-volumes-metadata
dataPool: ec62-ssd

These coincide with my underlying previously setup replica 3x pool on NVMe for erasure coding metadata and erasure-coding k6m2 data pools.