Consider failed machine as terminating

[himanshu-kun]

What this PR does / why we need it:
CA will now consider Failed machine as Terminating to avoid race with MCM in case of machine not joining in 20min.
Which issue(s) this PR fixes:
Fixes #118

Special notes for your reviewer:

Release note:

Race b/w MCM and Cluster Autoscaler in case of node joining timeout is fixed

/invite @unmarshall

[gardener-robot]

@hardikdr, @unmarshall You have pull request review open invite, please check

[himanshu-kun]

Logs like this would be shown for a node whose removal is skipped by CA to avoid race (machineDeployment replicas before is 2)

I0619 16:06:43.553497   21618 static_autoscaler.go:343] Some unregistered nodes were removed, skipping iteration
I0619 16:06:53.900302   21618 static_autoscaler.go:231] Starting main loop
I0619 16:06:53.902507   21618 clusterstate.go:602] Found longUnregistered Nodes [requested://shoot--i544024--ca-test-worker-1-z1-dff8d-tdcll]
I0619 16:06:53.902537   21618 static_autoscaler.go:335] 1 unregistered nodes present
I0619 16:06:53.902543   21618 static_autoscaler.go:611] Removing unregistered node requested://shoot--i544024--ca-test-worker-1-z1-dff8d-tdcll
I0619 16:06:53.902581   21618 mcm_manager.go:875] Machine "shoot--i544024--ca-test-worker-1-z1-dff8d-tdcll" is already being terminated, and hence skipping the deletion
I0619 16:06:53.902587   21618 mcm_manager.go:481] Machine "shoot--i544024--ca-test-worker-1-z1-dff8d-tdcll" priority is already set to 1, hence skipping the update
I0619 16:06:53.902600   21618 mcm_manager.go:525] MachineDeployment "shoot--i544024--ca-test-worker-1-z1" is already set to 2, skipping the update
I0619 16:06:53.902608   21618 mcm_manager.go:546] MachineDeployment shoot--i544024--ca-test-worker-1-z1 size decreased to 2
I0619 16:06:53.902629   21618 static_autoscaler.go:343] Some unregistered nodes were removed, skipping iteration

Earlier it used to be like this

I0619 16:19:56.158200   22928 static_autoscaler.go:231] Starting main loop
I0619 16:19:56.160045   22928 clusterstate.go:602] Found longUnregistered Nodes [requested://shoot--i544024--ca-test-worker-1-z1-dff8d-tdcll]
I0619 16:19:56.160208   22928 static_autoscaler.go:335] 1 unregistered nodes present
I0619 16:19:56.160225   22928 static_autoscaler.go:611] Removing unregistered node requested://shoot--i544024--ca-test-worker-1-z1-dff8d-tdcll
I0619 16:19:56.160306   22928 mcm_manager.go:481] Machine "shoot--i544024--ca-test-worker-1-z1-dff8d-tdcll" priority is already set to 1, hence skipping the update
I0619 16:19:56.343652   22928 mcm_manager.go:546] MachineDeployment shoot--i544024--ca-test-worker-1-z1 size decreased to 1
I0619 16:19:56.343686   22928 static_autoscaler.go:343] Some unregistered nodes were removed, skipping iteration

So the machineDeployment replicas is not reduced now.

[himanshu-kun]

/close as this is more of a bandaid fix which doesn't address the actual problem.