"Official/best" path to migrate from OKD-FCOS v4.15 to OKD-SCOS v4.16?

[titou10titou10]

A BIG thanks for the good work for the team behind the first stable version of OKD-SCOS v4.16!

Now that there is an official "stable" release of OKD-SCOS v4.16, what is the official/best plan to migrate from an installed cluster from the latest stable OKD-FCOS v4.15 to the latest stable OKD-SCOS?

Currently the latest 4.15/4.16 FCOS/SCOS versions are:

• OKD-SCOS v4.16.0-0.okd-scos-2024-08-21-155613
• OKD-SCOS v4.15.0-0.okd-scos-2024-01-18-223523
• OKD-FCOS v4.15.0-0.okd-2024-03-10-010116

Possible scenarios/paths:

• OKD-FCOS v4.15 > OKD-SCOS v4.15 > OKD-SCOS v4.16
• OKD-FCOS v4.15 > OKD-SCOS v4.16
• Install a brand new OKD-SCOS v4.16, "move(!)" the apps etc from OKD-FCOS v4.15 to it, delete cluster OKD-FCOS v4.15

At some point, upgrading from OKD-SCOS v4.15 to OKD-SCOS v4.16 was problematic and many manual steps had to be used as described in this discussion with the step-by-step instructions here

Q:

• Are the 2 first scenarios even possible?
• What is the most easy/automatic/secure way to go?
• Is there a recommended way to do it?
• Is there some work that is currently done to ease the way as I can guess we are many that will need to upgraded to OKD-SCOS v4.16+
  Thanks

[JaimeMagiera]

Just to clarify: Though there is a nightly that was moved to "stable", it's not a GM release so to speak. It just passed the E2E tests for AWS. The Working Group still has to do a few more tests, write documentation for various known issues and workarounds, make some decisions on the operator catalog, and a variety of other tasks before we make it an official release.

[titou10titou10]

oh!
that's very confusing and different from the way releases were handled for FCOS...
I even opened an issue thinking it was an official release...

[JaimeMagiera]

Yes, it is different. Overall, we're in the middle of a paradigm shift – not just switching to SCOS as the base, but also the Working Group taking over more duties from Red Hat. OKD is shifting towards being a true community project. Along those lines, the Working Group is trying to fill holes in the testing before we sign off on a GM release. For example, the builds are currently failing on vSphere due to secure boot not being supported in SCOS. We can't just release these builds as GM without some accompanying documentation on the workaround. That would frustrate a lot of people and ultimately work against the project. Additionally, the Red Hat CI only runs E2E tests on AWS and vSphere. The community needs a process of testing libvirt, gcloud, azure, and baremetal SNO if anything just to identify and document workarounds.

At next week's meeting, we're finalizing a testing template document that will provide a matrix of possible platforms and installation types. Ideally, we'd have volunteers to cover as many of the points in the matrix as possible. Interested in contributing?

[Prashanth684]

+1 to Jaime's statement. While the regular conformance e2es do pass for the stable releases to be accepted, we rely on the community for agent/assisted/baremetal install verifications.

[titou10titou10]

@JaimeMagiera, @Prashanth684

While the regular conformance e2es do pass for the stable releases to be accepted

There one thing to be flagged as "accepted" it is another to group/flag them as "stable". This has been reported weeks ago, but now I guess people now that "stable" versions are not "stable", just "accepted" with a little more than "accepted nightly builds" (what more exactly?)

we rely on the community for agent/assisted/baremetal install verifications.

OK...who's the "community" RH/IBM/FCOS/SCOS... and affiliated employees? who's involved?

Sorry to be a little bit tired, but it has been about 7 months since the last OKD GA release. Everybody knows that the move being done now is not an easy one and it will take time... but the process is so opaque from outside. Reading the weekly meeting does not give much info.. Where is the plan defined ? what are the progresses? the main milestones? the remaining tasks to go for a GA? etc.

What do you really need? I've opened many discussion/issues in the past on here and many have never been addressed. or often commented. I see very few report of the "community" about the kind of tests you are talking on here.
So now, I'm asking myself if it is worth it trying again and again to install OKD_SCOS and try to be as detailed as possible to report issues (I now test only "stable" releases where image are in quay because the nightly build seem not accessible from outside RH/FCOS/SCOS..

So what are you expecting from the "community"?
What tests?
How to report problems? to who? level of details? The last statement was to create "discussions" instead of issues...is this still correct?
What to report as problems?

At a minimum, when someone report some issue, a kind of acknowledgement should be issued by... (someone in charge?)

Thanks

[Prashanth684]

@titou10titou10 i can completely understand the frustration here. A lot of what we have done to fix OKD has happened behind the scenes and we (RedHat) have tried to communicate it as best as possible in the community meetings.

This has been reported weeks ago, but now I guess people now that "stable" versions are not "stable"

The release has been identified as "stable" from the perspective of an IPI based installation on aws with a conformance test suite run on it (details in https://github.com/openshift/origin). Assisted/agent support is not possible currently because the bootimages used are still fcos whereas we would need centos stream based bootimages. We are currently discussing a plan for that.

OK...who's the "community"

The community is anyone who is using OKD. Any feedback is appreciated and we encourage folks to join the community meeting to discuss any issues. @JaimeMagiera and others have been quite active in responding to issues filed and constantly bringing those to our attention. In the past few months the project has undergone many changes, which is why you might have noticed sparse responses to issues. We aim to correct that as well, but first, we need to make sure OKD is where we need it to be.

So what are you expecting from the "community"?
What tests?

Any tests/supported installation scenarios not covered by the basic conformance tests would help. If necessary we can also enable more test suites—things like testing community operators, samples, etc.. which are typically different for OKD and OpenShift.
I would highly recommend joining the weekly community meetings to get a good sense of where we are and what challenges we are dealing with.

[titou10titou10]

Thanks for your answer

The release has been identified as "stable" from the perspective of an IPI based installation on aws with a conformance test suite run on it (details in https://github.com/openshift/origin).
Assisted/agent support is not possible currently because the bootimages used are still fcos whereas we would need centos stream based bootimages
Any tests/supported installation scenarios not covered by the basic conformance tests would help.

So if I understand well, you are currently asking the community to test OKD on vSphere and AWS only (as those are the current automated tests) and that it is too early to test ABI/AI UPI install on baremetal as it is not supposed to work, right?

As you've maybe seen, I had success install installation OKD-SCOS v4.16.0-0.okd-scos-2024-08-21-155613 by replacing bootstrap image (and everything looked great when installed) but I'll wait until the bs image is no more FCOS

So, as said to Jaime before, when time will come, I can perform some tests installing/running OKD-SCOS with

• ABI/UPI install on baremetal (preference)
• AI/UPI install on baremetal
• SNO, full cluster (3+3) installs
• without ( fake) or with RH pull secret

[JaimeMagiera]

• As mentioned, what's ending up in stable is just what passes the AWS and vSphere E2E tests. That's how it's always been in fact. OKD has never had testing of the other platforms. We're hoping to change that by leveraging the community to test the other platforms before we sign off on something as an official release.

• We are currently asking the community to test whatever fits their use case. Since we only have E2E tests for AWS and vSphere, testing bare metal, gcloud, azure, etc. are the most helpful. In other words, we know that what passes Red Hat's E2E works. What we don't know is how a potential release works on bare metal, gcloud, azure, etc.

• It's going to be a while before the initial OS for the nodes is not FCOS. There are a lot of technical and organizational issues we're wading through to find a home for SCOS builds. It may be that our first official release is still starting out with FCOS nodes as it does now.

• As such, please test UPI now on any configuration. Use the latest FCOS for the nodes.

• Please do not use a RH pull secret. That is a violation of Red Hat's Terms of Service.

Hope that helps. Please let me know if you have any further questions. We're working on a new statement that should go out this week.

[wiqu85]

Gentlemen, I need your help because I admit that I am completely lost in what is currently happening in the OKD project.
Over a year ago I installed cluster okd on vSphere using the provided terraform playbooks. The process worked perfectly until new versions of OKD on FCOS stopped appearing.
From what I understand, it is recommended to switch to SCOS, which is constantly updated. Would any of you be so kind as to tell me how I can install okd scos on VMW? I don't see ready-made scos images that I can import into VMW, similarly to fcos. The OKD documentation in the vSpere chapter still talks about FCOS and not SCOS. Where can I download a SCOS image to import in VMW? I will be grateful for your help! Regards

[JaimeMagiera]

The OKD SCOS installers are known to work with fedora-coreos-39.20231101.3.0. There have been some issues when pivoting from newer versions. Any testing of newer versions is appreciated though. You can find more info on community testing here

[titou10titou10]

The OKD SCOS installers are known to work with fedora-coreos-39.20231101.3.0. There have been some issues when pivoting from newer versions. Any testing of newer versions is appreciated though. You can find more info on community testing here

@JaimeMagiera, FCOS v39.20231101.3.0 does not work for me for an ABI/AI UPI baremetal install as none of the more recent FCOS images I tested..(problem with the /sysroot file system read-only...)
Overriding the bs image with rhcos-4.17.0-ec.3-x86_64-live.x86_64.iso makes the installation go further and with a small tweak, have OKD-SCOS installed
Check here #2015 and here #2035
None of the FCOS version I tested worked

[JaimeMagiera]

@titou10titou10 I know, but we consider you an edge case. We have lots of successful bare metal installs working with the above scenario. We'll be documenting your edge case in the Known Issues upon release.

[titou10titou10]

we consider you an edge case.

oh? can you tell me in what it is an edge case? installing on baremetal in UPI mode with the ABI is considered an edge case? or something in the setup/config I use?

[JaimeMagiera]

An edge case in that Bare Metal + UPI + ABI is a known issue that the ABI team is now aware of. A SCOS base is needed to address the issue.

[aneagoe]

Back to the original questions, is there an upgrade path from FCOS -> SCOS? Will there even be one? Just wondering how existing clusters should be handled. Thanks!

[JaimeMagiera]

Yes, there's currently an MR to add such documentation to our website. We're gathering feedback from the community before publishing it. If you want to give it a shot and suggest any additions, that would be helpful.