e2e: cloud-config secret can be removed before CAPO/CAPI resources being deleted

[EmilienM]

/kind bug

What steps did you take and what happened:

In e2e, when tearing down the tests, we remove the namespace that was used.
In that namespace, there is the cloud-config secret that is used to store the OpenStack credentials.
If that secret is removed, some resources will fail to be reconciled by the cluster so deleting the namespace at the end of the tests is problematic if not everything was reconciled.

This could lead to this error:

E0601 02:15:54.144814       1 controller.go:329] "Reconciler error" err="secrets \"cluster-e2e-ljvg60-cloud-config\" not found" controller="openstackserver" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackServer" OpenStackServer="e2e-ljvg60/cluster-e2e-ljvg60-control-plane-plgrj" namespace="e2e-ljvg60" name="cluster-e2e-ljvg60-control-plane-plgrj" reconcileID="588bbb5b-2d05-4391-833c-a2ef4eab3a32"

(Found via #2067)

I think until now we were lucky because all cluster resources were removed before the namespace gets deleted but with the new controller things might take a little bit more time and we found this issue again.

[EmilienM]

I'm totally unsure if we'll actually fix that situation because CAPI framework waits for a cluster to be deleted as we can see here:
https://github.com/kubernetes-sigs/cluster-api/blob/e2bf2b7d108ecd51bf7b23bf50e2432e82d8f2fe/test/framework/cluster_helpers.go#L167-L177

However it seems like nothing waits for the OpenStackServers to be deleted in my initial implementation. I'll play around that for now.

[EmilienM]

/cc @mdbooth

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[EmilienM]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale