☂️ Enable MCM providers to force delete machines stuck in Terminal state

[himanshu-kun]

How to categorize this issue?

/area quality
/area robustness
/kind enhancement
/priority 3

What would you like to be added:

MCM should be able to force delete machines if they are stuck in terminal state (an irrecoverable state where API calls other than force delete won't work).

Why is this needed:

We have seen issues (currently on Azure only) where the VM was stuck in terminal state (refer Live Ticket # 2946)

VM deletion failed due to - machine codes error: 
code = [Internal] message = [Code="OSProvisioningTimedOut" Message="
OS Provisioning failure has reached terminal state and is non-recoverable for VM 
'shoot--hc-can-az--prod-az-haas-hana-vsmp4-z2-8667b-5frj5'. 
Consider deleting and recreating this virtual machine. 

The above error in Azure can be reproduced possibly by

• https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/troubleshoot-deployment-new-vm-linux#issue-custom-image-provisioning-errors
• https://learn.microsoft.com/en-us/answers/questions/1112332/osprovisioningtimedout

MCM fails recovering from this situation , as we detach the disks first (an Update operation) in Azure and then go for DeleteVM() . Since disk detachment is never triggered due to terminal state, the situation becomes irrecoverable and the Delete flow of MCM keeps on repeating.

Similar situations could be seen in other providers where normal Delete won't work and a force delete might be needed.

Example ticket canary # 4358
Proposal:

• Have an alternate Force Delete flow , which is triggered if the normal Delete flow fails for a threshold number of times
  • Need to confirm the error from provider , as force delete shouldn't be triggered for errors where a backoff should be done. Ex-
    • API rate limits (often seen in CCloud)
    • invalid credentials
• If an Annotation is placed on machine obj, then we can trigger a force delete, which might vary from provider to provider.

Providers:

• mcm-provider-azure (See ☂️ Enable MCM providers to force delete machines stuck in Terminal state #810 (comment))
• mcm-provider-aws
• mcm-provider-gcp
• mcm-provider-openstack
• mcm-provider-alicloud
• mcm-provider-local

[himanshu-kun]

Post-grooming discussion

We'll consider this after the cascade delete feature is in, because there the delete call is the one which is issued first. Cascade delete issue -> gardener/machine-controller-manager-provider-azure#91

[unmarshall]

MOM

Attendees: @himanshu-kun , @rishabh-11 , @elankath , @unmarshall
MCServer today allows configuration of machine-creation-timeout but does not have a corresponding machine-deletion-timeout. We need a way to determine when is the right time to use force delete option for machine-resources (NIC, Disks and VM). When we overhaul the state transitions for machines in MCM we should think of defining timeout post which a force deletion will happen.

[unmarshall]

We must observe if force deletion of VM is required since we no longer detach the disks first but instead leverage cascade delete. So forceful deletion of the VMs is a nice feature to have, but going forward it might be required only in rare situations. How rare, we can observe and ascertain

[unmarshall]

We (me and @himanshu-kun) noticed at least in azure, that there is no use case for force deletion of the VM whose ProvisioningState is set to Failed. A simple VM delete without force deletion is sufficient. If the disks and nic have DeleteOption set to Detach then triggering the delete will dis-associate these resources to this VM. If the DeleteOption is set to Delete (cascade delete) then deleting the VM will trigger the deletion of the associated resources as well.