Add an ability to delete hosted cluster state

[a13x5]

This is more like an ask and if such feature is already present please point me to the corresponding docs, because I couldn't find it.

Description

Right now if I delete k0smotron cluster (hosted) all state (PVCs) doesn't get deleted, so I should clean everything up by hands and then recreate the cluster.

This is ok, until I need to conduct some tests or have some disposable clusters. In this case 2 things happen, which aren't convenient:

1. PVCs linger in the cluster and consuming resources in the infrastructure. So this should be handled separately by some additional controller or manually.
2. If same name is used for the cluster old objects (like Nodes) are lingering. This makes cluster less stable and more confusing, since old resources (which don't exist) should be cleaned up by hands.

Proposal

If would be much more convenient to have a flag that should be explicitly set by user to delete all cluster state in case of cluster deletion.

This will make process more declarative avoiding lingering resources or confusing state in case of cluster re-deployment.

[jnummelin]

The PVC lingering is a feature of Kubernetes, unfortunately. What happens in this case is that k0smotron creates the statefulset with embedded PVC definition. So k0smotron does NOT create the PVC object directly. When we delete the statefulset, Kubernetes leaves the PVC behind.

Now looking at the statefulset object details, there's statefulset.spec.persistentVolumeClaimRetentionPolicy:

FIELDS:
  whenDeleted   <string>
    WhenDeleted specifies what happens to PVCs created from StatefulSet
    VolumeClaimTemplates when the StatefulSet is deleted. The default policy of
    `Retain` causes PVCs to not be affected by StatefulSet deletion. The
    `Delete` policy causes those PVCs to be deleted.

  whenScaled    <string>
    WhenScaled specifies what happens to PVCs created from StatefulSet
    VolumeClaimTemplates when the StatefulSet is scaled down. The default policy
    of `Retain` causes PVCs to not be affected by a scaledown. The `Delete`
    policy causes the associated PVCs for any excess pods above the replica
    count to be deleted.

I guess k0smotron could either set those to Delete by default and/or provide some field for the users to override.