[CI] Use Kubernetes GC to clean kubevirt VMs (packet-* jobs)

[VannTen]

What type of PR is this?
/kind feature

What this PR does / why we need it:
We regularly have CI flakes where the job failed to delete k8s namespace in the CI cluster.
It's not much, but it's a little hiccup in the PR process which I'd like to eliminate.

I'm not sure what the exact reason is, probably some race between the jobs and the time between fetching the list of namespace and the deletion.
Regardless, a simpler way to delete the VMs is to let them be dependants (in the kubernetes sense) of the job pod. This way, once the job pod is deleted, kubernetes garbage collection in the CI cluster will take care of removing the associated VMs

Special notes for your reviewer:
PR on the ci infra kubespray/kspray-infra#1 (private repo, maintainers have access)

Does this PR introduce a user-facing change?:

NONE

/label tide/merge-method-merge

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: VannTen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [VannTen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[VannTen]

/ok-to-test

[VannTen]

/cc @ant31

[VannTen]

/retest
(now that I fixed the gitlab-runner config)

[VannTen]

/retest

[VannTen]

/label ci-full

(To test it works correctly for everything)

[k8s-ci-robot]

@VannTen: The label(s) /label ci-full cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label ci-full

(To test it works correctly for everything)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[VannTen]

@ant31 What's the process to add the ci-{extended,full} label ? Can't find them with a quick grepping

[ant31]

I think the PR to add them via /label is still in review

For now you can add them manually

[VannTen]

I'll do that once the initial set of tests pass then 👍

[VannTen]

The ssh DNS errors are weird, it should not use the name of the VMI but the IP, I think from kubevirt dynamic inventory.

[VannTen]

/retest-failed This is to test whether it's something racey or systematic on those tests /hold

[tico88612]

/retest-failed

[tico88612]

@ant31 Sometimes, /retest-failed cannot retest the partially failed jobs.
Is there any other way to solve this than /retest? (/retest will refresh the status, but it wastes time.)

[tico88612]

The version of Cilium used by packet_debian11-custom-cni is outdated. (1.13.0)
Maybe fixed by #11654

[ant31]

/retest-failed

[ant31]

I guess ci-full is broken on all PR. Let's fix it on a different PR.

[ant31]

/retest

[VannTen]

Retrying a test with the error
test-vm-5mz4g | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname test-vm-5mz4g: Name or service not known",
"unreachable": true
}
got rid of the error.

Which isn't good because it seems to be random/racey

Same with:
TASK [Wait until SSH is available] *********************************************
fatal: [test-vm-7ll6t -> localhost]: FAILED! => {"changed": false, "elapsed": 240, "msg": "Timeout when waiting for localhost:22"}
fatal: [test-vm-pcmvv -> localhost]: FAILED! => {"changed": false, "elapsed": 240, "msg": "Timeout when waiting for localhost:22"}

In that case the VM have IPs 🤔

[VannTen]

It looks like the ip of VirtualMachineInstance are disappearing and I think the kubevirt dynamic inventory interpret that as using the name for ansible_host.

This might be related to kubevirt/kubevirt#12698 , the guest OS on which this happens match the one in the issue description (alma / rocky) (and opensuse, not in that issue).

[VannTen]

By doing a kubectl get vmis -A --watch during a CI runs, I see stuff like this

gitlab-runner   test-vm-8g65n                                       25s   Running      10.11.195.176   k3         True
gitlab-runner   test-vm-8g65n                                       25s   Running                      k3         True

So this really looks like the IP momentarily dissapear

[ant31]

we probably want to add some delay here and there and have some kind of retry at this step

[VannTen]

That's what the last commit does. But if the retry works and then the IP disappears, we're back where we started ^. Can't think of an effective workaround for now 🤔

[ant31]

maybe upgrading kubevirt ?

[VannTen]

Maybe. But the linked bug being still open is not making me very hopeful... (there is always a change we're not hitting that specifically but I'm not counting on it).