This is an initial, rough proof-of-concept version aimed at quickly obtaining
a demo while still in the process of learning Go and Kubernetes. It is mostly
undocumented and untested. It will remain here in a frozen state. A new
version will commence development in branch main.
Union CSI is a Container Storage Interface (CSI) plugin for Kubernetes, enabling the combination of multiple persistent volumes (branches) into single, unified hierarchies as FUSE union mounts for Pods.
- Merge existing Kubernetes persistent volumes to create a new union filesystem volume.
- Create new volumes to merge based on user configuration.
- Support generic storage composition (e.g., cached or tiered storage,
bcachefs).
Union CSI is a CSI "meta-plugin": a CSI plugin that aggregates the volumes of other plugins. This means than an underlying or lower storage driver is needed to perform the actual work (i.e., provision, attach, mount, etc.). Users point Union CSI towards a lower plugin to request and combine storage through its StorageClass object. The lower plugin does not have to be CSI compliant.
Union CSI invokes the creation and deletion of volumes from the lower plugin
through the PersistentVolumeClaim
(PVC) API. Users can create a PVC (upper) that points to the Union CSI plugin
and specify the desired number and size of branches from the lower plugin, or
indicate existing branches to collect. Union CSI will then create or adopt the
required set of PVCs (lower) that point to the lower plugin. Finally, it
utilizes mergerfs to pool the
branches together on the target node and provide a FUSE filesystem to
workloads.
The primary motivation behind Union CSI is to achieve multi-node volumes: volumes whose branches span across different disks and nodes in the cluster. This is accomplished by pooling the remote branches through the network. This enables users to achieve capacities that near the total available capacity of the cluster with just a single volume, promising massive scale-out and scale-up potential.
Union CSI currently delegates the core of this feature to the lower plugin — it is the lower plugin installed that should be able to remorely access its storage (e.g. iSCSI) in order for Union CSI to leverage and combine storage assets from different nodes.
Additionally, the only way to allocate new branches on different nodes through Union CSI is for the user to request a volume large enough so that its branches can only be placed on different nodes by the lower plugin.
The demo version of Union CSI, found in this branch, statically splits the requested capacity in half by always creating two equally sized lower PVCs. An example showcasing how this mini version can be used to yield a powerful use case is provided in Demo with Longhorn, where Longhorn is employed as the lower storage provider for Union CSI.
- Upper: Adjective for entities, components and resources related to the Union CSI volume plugin, e.g., upper plugin, upper volume, upper PVC, upper PV, etc.
- Lower: Adjective for entities, components and resources related to the underlying volume plugin used by Union CSI, e.g., lower plugin, lower volume, lower PVC, lower PV, etc.
Since Union CSI utilizes mergerfs and therefore FUSE, there is the added FUSE
overhead of transitioning back and forth from userspace to kernelspace to be
expected from Union CSI volumes. For more information, please refer to the
documentation
of mergerfs.
To create a local Docker image run
make IMAGE=union-csi:demo docker-buildTo deploy this demo version of Union CSI in a Kubernetes cluster via Kustomize using the Docker images from my personal registry on Docker Hub, run:
kubectl apply -k ./deploy/k8sUnion CSI is still work in progress and highly experimental. It is currently a personal and academic project.