Extract Kernel mount/umount parts of the cephcsi binary?

[leseb]

This is a follow-up to #2946 which has been answered. The clean takeaway (that I agree with) is the current cephcsi will never be statically compiled since we use cgo with go-ceph.

That said I'm still exploring other possibilities to get a statically compiled binary that would be responsible only for mounting and unmounting volumes using Kernel modules.
In that sense, I'm wondering if it would be possible to extract the kernel mount/umount pieces from cephcsi and put that in another binary?

Thanks!

[BlaineEXE]

Will said kernel mounter be able to create mounts in the network namespace of the long-lived pod that is running the csi binary?

[leseb]

The Kernel mounter becomes the long-lived pod in that scenario.

[Madhu-1]

Do you mean the nodeStage and NodeUnstage API which is responsible for rbd mapping and unmapping? If yes, we still use go-ceph API to talk to ceph cluster except rbd map command.

[nixpanic]

Yes, that sounds technically possible to me. As long as there is some UNIX-Domain-Socket communication possible (HostPath?), ideally over gRPC, it would work. Ceph-CSI can be adapted to call the mounting, (d)encrypting, mapping and other commands through a different interface than the local executables.

Still, a complete static linked executable with all the tools and functionalities combined seems like a maintenance headache to me.

[idryomov]

The mount/fsmount syscalls and /sys/bus/rbd/{add*,remove*} are stable interfaces for which pretty much infinite backwards compatibility is guaranteed so anyone is free to cut out rbd map and mount.ceph helpers and use the kernel interfaces directly -- subject to "re-implement everything", "have to fix bugs" and other good concerns raised by @nixpanic of course.

But I'm not sure I understand where this is coming from. It seems like the underlying issue is still the restart/upgrade of Ceph CSI containers that end up owning the network namespace. I thought that was solved with a clever use of bridging suggested by the OCP networking folks. Did that solution not pan out?

[leseb]

The mount/fsmount syscalls and /sys/bus/rbd/{add*,remove*} are stable interfaces for which pretty much infinite backwards compatibility is guaranteed so anyone is free to cut out rbd map and mount.ceph helpers and use the kernel interfaces directly -- subject to "re-implement everything", "have to fix bugs" and other good concerns raised by @nixpanic of course.

Thanks!

But I'm not sure I understand where this is coming from. It seems like the underlying issue is still the restart/upgrade of Ceph CSI containers that end up owning the network namespace. I thought that was solved with a clever use of bridging suggested by the OCP networking folks. Did that solution not pan out?

This solution was considered for a long time until we realized the number of corner cases it had. I'm sure @BlaineEXE can elaborate a bit more but from the top of my head:

• the solution would work for the cri-o runtime only (not Docker)
• risk of orphan interfaces left on the host (however the IP is released from the IPAM) which would cause L2 connectivity issues
• Multus has different primary operating modes (macvlan, ipvlan, bridge, host-device), and “moving” the interface is not the same in every case
• moved interface is completely left out of the Multus's scope
• overall over-engineered solution, no real idempotency while moving the interface on the host

You can find the latest design proposal here rook/rook#9903

[BlaineEXE]

Yes, that's a good summary from Seb. The bridging solution also might eliminate the possibility users have to use performance tweaking options for the multus interfaces without complicated caveats.

Another key downside of the previously-proposed bridging solution is that we still force a solution where we break network isolation. In my estimation, network isolation is one of the largest benefits of using multus instead of host networking. The bridging solution effectively creates a network back door on all nodes.

The solution Seb is proposing here allows us to keep CSI kernel mounts stable through upgrades and still keep network isolation.

[idryomov]

Thanks for the explanation! I have left my thoughts in the design proposal PR.