Add drive repair support

[balamurugana]

No description provided.

[Praveenrajmani]

If an inuse drive (drive with published volumes) is repaired, the job fails with the following

➜  directpv git:(xfsrepair) ✗ kubectl logs repair-c65a2d95-d140-4743-a0b1-8507ac772298-x8g5g -n directpv
I0725 08:12:04.934567       1 repair.go:70] xfs_repair: cannot open /dev/dm-1: Device or resource busy
E0725 08:12:04.934639       1 main.go:148] "unable to execute command" err="unable to run xfs_repair on device /dev/dm-1; exit status 1"

and the status of the drive will be "repairing"

➜  directpv git:(xfsrepair) ✗ kubectl directpv list drives --all 
┌────────────────────────────────┬──────┬───────────┬─────────┬─────────┬─────────┬───────────┐
│ NODE                           │ NAME │ MAKE      │ SIZE    │ FREE    │ VOLUMES │ STATUS    │
├────────────────────────────────┼──────┼───────────┼─────────┼─────────┼─────────┼───────────┤
│ praveen-thinkpad-x1-carbon-6th │ dm-0 │ vg0-lv--0 │ 1.9 GiB │ 1.8 GiB │ 4       │ Ready     │
│ praveen-thinkpad-x1-carbon-6th │ dm-2 │ vg0-lv--2 │ 1.9 GiB │ 1.8 GiB │ 4       │ Ready     │
│ praveen-thinkpad-x1-carbon-6th │ dm-3 │ vg0-lv--3 │ 1.9 GiB │ 1.8 GiB │ 4       │ Ready     │
│ praveen-thinkpad-x1-carbon-6th │ dm-1 │ vg0-lv--1 │ 1.9 GiB │ 1.8 GiB │ 4       │ Repairing │
└────────────────────────────────┴──────┴───────────┴─────────┴─────────┴─────────┴───────────┘

this error is however expected, can we consider the following?

• Can we do some checks in kubectl directpv repair to make sure the drive to be repaired doesn't have any running workloads (No published volumes)?

• Can we go ahead and umount the drive completely (umounting all the volume mounts as well) before xfs_repair? Anyway, repairing requires a minio pod restart which will bring back the volume mounts.

(OR)

Explaining in the doc to make sure to cordon the node and delete the respective pod to umount all the PV mounts before executing repair.

@balamurugana ^^

[balamurugana]

@Praveenrajmani As Kubernetes is declarative, it is not guaranteed to check mounted volumes on the drive. The repair command may see the drive is free from volume mounts, but repair job would fail with Device or resource busy because of job scheduling.

It is completely an admin responsibility when to run repair command. Documenting prerequisites for repair command is the only way to go.

[Praveenrajmani]

We can add repair doc to explain necessary pre-requisites for repairing.

Tested and LGTM otherwise.