CNV-47438: ipam, virt: graduate persistent ips feature gate to GA

[maiqueb]

This PR graduates the PersistentIPsForVirtualization feature gate to GA.

[openshift-ci]

Hello @maiqueb! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

[maiqueb]

/test verify

[maiqueb]

@JoelSpeed o/

We have added periodic lanes to trigger the test for this feature gate in this PR, which we plan on GAing for 4.18.

The tests are quite thorough, and were added in this other PR.

Unfortunately, while the tests themselves pass, it seems the monitor tests fail constantly due to the monitor tests. We have been trying to get this sorted out, but everytime we look, another component of OpenShift Virt throws a new event / alarm / fails to adhere to conformance testing - OpenShift virt uses their own conformance testing framework, and the hypershift lanes we have in origin do not run monitor tests afaiu.

This is a list of items we have gone through in the last weeks:

• openshift/origin@9bb5aa1 (ignored virt alerts)
• virt: Skip operator unhealthy alarm origin#29305 (another virt alert to be ignored ...)
• events: Remove duplicated 'Available' events kubevirt/cluster-network-addons-operator#1922 (Remove duplicated 'Available' events from CNAO)
• Reduce frequency of Route management events kubevirt/containerized-data-importer#3490 (Reduce frequency of Route management events)
• Reduce frequency of SCC management Events kubevirt/containerized-data-importer#3477 (Reduce frequency of SCC management Events)
• Reset kubevirt_hco_system_health_status on update kubevirt/hyperconverged-cluster-operator#3170 (kubevirt_hco_system_health_status old error/warning values with different reasons still show after conditions are updated)
• Fix OperatorConditionsUnhealthy alert expression metric value kubevirt/hyperconverged-cluster-operator#3181 (Fix OperatorConditionsUnhealthy alert expression metric value; we "alerted" the cluster was healthy)

As a result, we plan on having the OpenShift Virt QE manually sign-off the feature is stable enough.

This is a heads-up so you know our intentions. Do you foresee any issues with this path ?

How does the process look like for the QE signoff ? Just setting qe-approved label ? Who overrides the failing verify lane ?

[JoelSpeed]

Looking at prow I can only see two runs of the periodics you've added, that, doesn't seem right?

Both have failed and seem to be complaining that there are no tests to run?

We normally ask to see test results in sippy, could you see if you can find any test data in sippy for the tests you've added? TRT may be able to help you if you can't work this out

[JoelSpeed]

Unfortunately, while the tests themselves pass, it seems the monitor tests fail constantly due to the monitor tests.

I can't see any evidence of this, please link

[maiqueb]

Unfortunately, while the tests themselves pass, it seems the monitor tests fail constantly due to the monitor tests.

I can't see any evidence of this, please link

Surely; if you open the Tests Passed in the periodic 4.18 execution and search for PersistentIPsForVirtualization you'll see all of them have passed.

[maiqueb]

Looking at prow I can only see two runs of the periodics you've added, that, doesn't seem right?

Both have failed and seem to be complaining that there are no tests to run?

I'd say this is on 4.19, where I am running the cluster without TP feature set.

We normally ask to see test results in sippy, could you see if you can find any test data in sippy for the tests you've added? TRT may be able to help you if you can't work this out

[JoelSpeed]

Surely; if you open the Tests Passed in the periodic 4.18 execution and search for PersistentIPsForVirtualization you'll see all of them have passed.

Ahh I see now, right, so these tests are passing, but don't show up in sippy, which means we can't actually track or visualise them. I've started some threads in slack in #forum-ocp-release-oversight. I'd like to understand why this isn't in sippy, once we have it in sippy, we can visualise your tests, and, looking at the results, it may actually satisfy the requirements to pass the verify naturally

[maiqueb]

Surely; if you open the Tests Passed in the periodic 4.18 execution and search for PersistentIPsForVirtualization you'll see all of them have passed.

Ahh I see now, right, so these tests are passing, but don't show up in sippy, which means we can't actually track or visualise them. I've started some threads in slack in #forum-ocp-release-oversight. I'd like to understand why this isn't in sippy, once we have it in sippy, we can visualise your tests, and, looking at the results, it may actually satisfy the requirements to pass the verify naturally

Oh nice. Hopefully :D

[openshift-ci-robot]

@maiqueb: This pull request references CNV-47438 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.19.0" version, but no target version was set.

In response to this:

This PR graduates the PersistentIPsForVirtualization feature gate to GA.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[maiqueb]

/jira refresh

[openshift-ci-robot]

@maiqueb: This pull request references CNV-47438 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.19.0" version, but no target version was set.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[yossisegev]

Update from CNV QE:
I managed to verify UDN persistentIP (on CNV VMs) for both migration and VM restart scenarios.
Verification was executed on an 4.18 PSI (virtual) cluster.

The scenario I ran for verifying persistent IP after restart:

1. I created 2 VMs with primary UDN interfaces.
2. On vm-a I ran an iperf3 server, which listens on a specific port:
   [fedora@vm-a ~]$ iperf3 -s -p 2345
   -----------------------------------------------------------
   Server listening on 2345 (test 1)
   -----------------------------------------------------------
3. On vm-b I ran an iperf3 client, to maintain a connection to vm-a primary interface IP (via the same port, of course):
   [fedora@vm-b ~]$ iperf3 -c 192.168.88.4 -p 2345 -t 0
   Connecting to host 192.168.88.4, port 2345
   This connection succeeded.
4. I restarted vm-a using virtctl restart vm-a
5. When vm-a came up again, I
   a. Verified eth0 still has the same IP address.
   b. Re-ran the iperf3 server and client on the VMs, and let the client connect to the same IP address (and port) as before the restart.
   Once again - a successful connection was observed in both sides.

For migration I ran almost the same scenario, with the difference of stopping the iperf3 session in both client and server before migrating the VM, and restarted it (using the same persistent IP address) after the migration was completed.
This scenario passed successfully as well.

[JoelSpeed]

/test verify

[deads2k]

/hold

If we're facing SCC pinning failures on components, that's a very serious problem that in 4.19 can cause non-functional clusters with very high bandwidth costs as components attempt to create pods with unexpected SCCs that don't conform to PSA. This happens in customer environments with custom SCCs, so passing CI doesn't mean that the bug is "safe". It just means it is latent. This has happened in the past and was extremely costly. Get PRs open to fix the SCC pinning annotations.

[deads2k]

While the desire is to GA the feature in 4.18, if it breaks on 4.19, we've simply created a problem for ourselves. Let's get a PR up adding the annotations for the failing workloads in https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/58873/rehearse-58873-pull-ci-openshift-ovn-kubernetes-release-4.19-e2e-aws-ovn-virt-techpreview/1864239266813448192

[deads2k]

We've got the ball rolling on these violations with kubevirt/cluster-network-addons-operator#1931 and more will be coming. These changes are important and lack of this pinning carries some risk in 4.18 and substantial risk in 4.19. Given 4.18 timeframes and a commitment to address the problem in 4.19, we can promote this feature once the rest of the testing functions.

/hold cancel

[JoelSpeed]

/retest

[JoelSpeed]

The periodic this morning failed on a persistent IPs test, can you please investigate and let me know here why this failed this time? https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-ovn-kubernetes-release-4.19-periodics-e2e-aws-ovn-virt-techpreview-periodic/1864460089788731392

[maiqueb]

The periodic this morning failed on a persistent IPs test, can you please investigate and let me know here why this failed this time? https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-ovn-kubernetes-release-4.19-periodics-e2e-aws-ovn-virt-techpreview-periodic/1864460089788731392

The TL;DR; is we're hitting this bug: https://issues.redhat.com/browse/OCPBUGS-42609 on the primary UDN feature.

The failing test is using the following parameters:

• provision network via NAD
• primary UDN

We create a server pod - e2e-test-network-segmentation-e2e-jljp9/testpod-ip-10-0-38-150.ec2.internal - which is supposed to have 2 network attachments:

• default cluster network
• primary UDN (which was provisioned using the NAD)

From the CNI result (which we can see on the OVN-K logs) we can see that only the default cluster network attachment was added:

Z I1205 01:17:19.211554    4193 cni.go:323] [e2e-test-network-segmentation-e2e-jljp9/testpod-ip-10-0-38-150.ec2.internal 9984a1dc2f16e647acde86605d219479cd7ce0c9b25b2f49cd8cb96b9ec48c80 network default NAD default] ADD finished CNI request [e2e-test-network-segmentation-e2e-jljp9/testpod-ip-10-0-38-150.ec2.internal 9984a1dc2f16e647acde86605d219479cd7ce0c9b25b2f49cd8cb96b9ec48c80 network default NAD default], result "{\"interfaces\":[{\"name\":\"9984a1dc2f16e64\",\"mac\":\"36:5e:57:2f:79:5b\"},{\"name\":\"eth0\",\"mac\":\"0a:58:0a:82:00:71\",\"sandbox\":\"/var/run/netns/907fbf44-3cc3-4c97-9ef1-78411ed8fa08\"}],\"ips\":[{\"interface\":1,\"address\":\"10.130.0.113/23\",\"gateway\":\"10.130.0.1\"}],\"dns\":{}}", err <nil>

The VM is trying to reach the server pod using the primary UDN, which was not plumbed into the pod.

As indicated in the bug, this race can happen.

I think I'll add some code to the tests after provisioning the NAD / UDN, and before provisioning any workload to assure that an OVN logical entity for the UDN was created. This means that the NAD/UDN was processed, and that while the network itself might not be ready, it will eventually be.

A Sleep could also do, but that sucks. I don't know for how long to sleep.

FWIW - the network segmentation feature gate is also prone to this race, and I expect to see this happening on their lanes as well.

[deads2k]

/hold

holding so that we can ensure https://issues.redhat.com/browse/OCPBUGS-42609 is resolved so that the feature works reliably before we make it accessible by default.

[deads2k]

/hold cancel

it's an issue with UDN enabled, but with something called a secondary network it seems to work reliably? It sure would be nice if everything worked at the same time....

[maiqueb]

/test verify

[JoelSpeed]

/payload-aggregate periodic-ci-openshift-ovn-kubernetes-release-4.19-periodics-e2e-aws-ovn-virt-techpreview-periodic 10

[openshift-ci]

@JoelSpeed: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-ovn-kubernetes-release-4.19-periodics-e2e-aws-ovn-virt-techpreview-periodic

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/b1380e20-b875-11ef-9100-1cf11bd6ba9c-0

[JoelSpeed]

2/10 jobs from the aggregate earlier had a failure specifically rated to this feature, see https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-api-2058-per[…]-e2e-aws-ovn-virt-techpreview-periodic/1867157953270779904 and https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-api-2058-per[…]-e2e-aws-ovn-virt-techpreview-periodic/1867157940671090688
Are these the same failures you're already looking into?

[JoelSpeed]

/payload-aggregate periodic-ci-openshift-ovn-kubernetes-release-4.19-periodics-e2e-aws-ovn-virt-periodic 10

[openshift-ci]

@JoelSpeed: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-ovn-kubernetes-release-4.19-periodics-e2e-aws-ovn-virt-periodic

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/a5b3c130-cd55-11ef-9453-12bb6b8577d4-0

[JoelSpeed]

/payload-aggregate periodic-ci-openshift-ovn-kubernetes-release-4.19-periodics-e2e-aws-ovn-virt-periodic 10

[openshift-ci]

@JoelSpeed: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-ovn-kubernetes-release-4.19-periodics-e2e-aws-ovn-virt-periodic

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/37cb4550-cd9e-11ef-850c-8b451ba1c95d-0

[openshift-ci]

@maiqueb: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	b701a1f	link	false	/test okd-scos-e2e-aws-ovn

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[JoelSpeed]

/payload-aggregate periodic-ci-openshift-ovn-kubernetes-release-4.19-periodics-e2e-aws-ovn-virt-periodic 10

[openshift-ci]

@JoelSpeed: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-ovn-kubernetes-release-4.19-periodics-e2e-aws-ovn-virt-periodic

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/d5a5f9b0-cf37-11ef-9b28-47182ea2adf8-0

[maiqueb]

Seems the tests didn't fail in https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/aggregator-periodic-ci-openshift-ovn-kubernetes-release-4.19-periodics-e2e-aws-ovn-virt-periodic/1877653489371320320/artifacts/release-analysis-prpqr-aggregator/openshift-release-analysis-prpqr-aggregator/artifacts/release-analysis-aggregator/job-run-summary.html ...

Most of the reported failures are from the missing SCC / cluster broken.

Would it make sense to temporarily add the openshift-cnv namespace to the list of namespaces that can miss the required-scc annotation ? I've pushed this PR for that a while back: openshift/origin#29341

[JoelSpeed]

Based on the aggregate run and current sippy results, I can see that the parts of this feature that do not rely on the NetworkSegmentation feature gate are stable and pass consistently.

There is an outstanding IOU from the virt folks to fix their SCC annotations, but we are not blocking this feature on that.

/override ci/prow/verify

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JoelSpeed, maiqueb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JoelSpeed]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@JoelSpeed: Overrode contexts on behalf of JoelSpeed: ci/prow/verify

In response to this:

Based on the aggregate run and current sippy results, I can see that the parts of this feature that do not rely on the NetworkSegmentation feature gate are stable and pass consistently.

There is an outstanding IOU from the virt folks to fix their SCC annotations, but we are not blocking this feature on that.

/override ci/prow/verify

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[maiqueb]

/label acknowledge-critical-fixes-only

[maiqueb]

/cherry-pick release-4.18

[openshift-cherrypick-robot]

@maiqueb: #2058 failed to apply on top of branch "release-4.18":

Applying: ipam, virt: graduate persistent ips feature gate to GA
Using index info to reconstruct a base tree...
M	features.md
M	features/features.go
M	payload-manifests/featuregates/featureGate-Hypershift-Default.yaml
M	payload-manifests/featuregates/featureGate-SelfManagedHA-Default.yaml
Falling back to patching base and 3-way merge...
Auto-merging payload-manifests/featuregates/featureGate-SelfManagedHA-Default.yaml
CONFLICT (content): Merge conflict in payload-manifests/featuregates/featureGate-SelfManagedHA-Default.yaml
Auto-merging payload-manifests/featuregates/featureGate-Hypershift-Default.yaml
CONFLICT (content): Merge conflict in payload-manifests/featuregates/featureGate-Hypershift-Default.yaml
Auto-merging features/features.go
Auto-merging features.md
CONFLICT (content): Merge conflict in features.md
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 ipam, virt: graduate persistent ips feature gate to GA

In response to this:

/cherry-pick release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

Distgit: ose-cluster-config-api
This PR has been included in build ose-cluster-config-api-container-v4.19.0-202501110011.p0.gc1a063b.assembly.stream.el9.
All builds following this will include this PR.