Add support for IONOS Cloud #1957

mcbenjemaa · 2024-10-17T13:48:07Z

This PR will add support for the IONOS Cloud provider.
IONOS Cloud doesn't officially have a metadata service, as it's currently in development.
However, the IONOS Cloud injects user-data into /var/lib/cloud/seed/nocloud/user-data directly into the volume.

I just want help in how I can test this.

prestist · 2024-10-18T19:05:46Z

internal/providers/ionoscloud/ionoscloud.go

+)
+
+const (
+	defaultFilename = "/var/lib/cloud/seed/nocloud/user-data"


Hmm, do you have documentation for this, I am not having any luck finding the default user-data file location for IONOS

I thought cloud-init's default was something like "/var/lib/cloud/instance/user-data"

Additionally It might be good to add a reference to the in development metatdata service for this, so we can replace it when completed.

Unfortunately, we don't have that documented publicly.
Its just a internal docs.

However this docs explains how can use Userdata.
https://docs.ionos.com/cloud/compute-services/compute-engine/how-tos/boot-cloud-init

The path is /var/lib/cloud/seed
So it could be detected by the NoCloud Datasource.
So, im 100% sure of that.

But the /var/lib/cloud/instance is the information about the cloud-init run.

@prestist I can back this up as well. We don't have this publicly documented. Unfortunately, as of yet we don't have a metadata server at IONOS and therefore rely on a legacy script that injects data into this file before a VM is booted for the first time.

Ah, okay, yeah that makes sense. Ok, would it be okay to just add a comment to point to documentation?

Thank you both for answering my questions.

prestist · 2024-10-21T19:00:39Z

Okay so to help with the adding of this platform I highly recommend submitting a platform request issue to =>

https://github.com/coreos/fedora-coreos-tracker/issues/new/choose

It will ask questions and give us a tracker so that we can monitor the adding support bit.

then lets squash and update the first commit to be
providers: add support for ionos cloud (or what you feel fits it best)

with a reference to the aforementioned tracker.

internal/providers/ionoscloud/ionoscloud.go

travier · 2024-10-22T14:50:42Z

However, the IONOS Cloud injects user-data into /var/lib/cloud/seed/nocloud/user-data directly into the volume.

This is really unusual. How is this supposed to work? This would mean that Ignition would have to mount to root/var partition first to read the config and then act on it? This feels really brittle.

Okay so to help with the adding of this platform I highly recommend submitting a platform request issue to => coreos/fedora-coreos-tracker/issues/new/choose

It will ask questions and give us a tracker so that we can monitor the adding support bit.

Second this comment here. We need a clearer picture of how things work here.

mcbenjemaa · 2024-10-22T15:01:05Z

@travier

This is really unusual. How is this supposed to work? This would mean that Ignition would have to mount to root/var partition first to read the config and then act on it? This feels really brittle.

I'm afraid that's the only way, it work for now.
the file /var/lib/cloud/seed/nocloud/user-data is injected into the Image.

However, we are adding information and defining how things work as much as we can,

thanks

tuunit · 2024-10-23T09:37:32Z

However, the IONOS Cloud injects user-data into /var/lib/cloud/seed/nocloud/user-data directly into the volume.

This is really unusual. How is this supposed to work? This would mean that Ignition would have to mount to root/var partition first to read the config and then act on it? This feels really brittle.

Okay so to help with the adding of this platform I highly recommend submitting a platform request issue to => coreos/fedora-coreos-tracker/issues/new/choose
It will ask questions and give us a tracker so that we can monitor the adding support bit.

Second this comment here. We need a clearer picture of how things work here.

Yes, indeed it is brittle and we would love to have a proper metadata server in the future. Unfortunately, this will not be implemented anytime soon and we would really love to already get support for ignition especially in Flatcar Linux.

jlebon · 2024-10-23T17:01:38Z

internal/providers/ionoscloud/ionoscloud.go

+)
+
+const (
+	defaultFilename = "/var/lib/cloud/seed/nocloud/user-data"


This won't work. Ignition runs from the initramfs and the /var there is different. You could inject it in what would become /var in the real root, but you'd have to special-case code for CoreOS which feels really like a layering violation.

At least for FCOS, probably the simplest way to hack around this for now (until there's a proper metadata service) is to inject the Ignition config in the boot partition (GPT partition label boot) at /ignition/config.ign. I'm not sure if there's an equivalent on the Flatcar side. We wouldn't even need to touch Ignition for that; the QEMU qcow2 image we publish should work for now.

Yep, we just had a chat and realized this fact as well. In case of Flatcar the root partition is mounted to /sysroot and therefore the path is wrong anyway. I'll try to build a flatcar image using a patch version of ignition and we need to investigate how to make it work on other distributions and make the base path configurable

@tormath1 am I right on this on? Regarding the /sysroot being a mount point of the boot partition?

jlebon · 2024-10-23T17:14:31Z

internal/providers/ionoscloud/ionoscloud.go

+		return types.Config{}, report.Report{}, err
+	}
+
+	if util.IsCloudConfig(rawConfig) {


Why should we tolerate non-Ignition configs here? The reason this is done in Proxmox VE is because the platform itself uses cloud-init to pass through platform values which is extremely unfortunate because it makes it harder for users to provide their own config and deteriorates the UX. Is that the case for this platform as well? How can we avoid it?

Indeed this is ugly DX not going to deny that fact. But we cannot change that as of now because the IONOS Cloud fully builts on the premise that all images use cloud-init. Which is why our legacy pre-boot image injector scripts write this specific cloud-init seed file. Meanwhile our team wants to introduce ignition to support linux derivatives that use it instead of cloud-init. Therefore we need to re-purpose this seed file to get earlier support until the Compute department at IONOS introduces a proper metadata service. It is definitely on the roadmap but will take months or possibly a year. Therefore this intermediate implementation will unfortunately (as so often the case with these things) be a long living temporary solution.

tuunit · 2024-11-04T12:11:34Z

Hi @prestist, @travier, @jlebon,

please have another look at the latest state of the PR. We successfully ran this version on our test VMs.

mcbenjemaa · 2024-11-04T13:27:23Z

internal/providers/ionoscloud/ionoscloud.go

+		return nil, err
+	}
+
+	if util.IsCloudConfig(contents) {


we can get rid of this check, I believe.

no we cannot, we want to ignore the file and do nothing if the config contains the #cloud-config directive

Add support for IONOS Cloud Add check to ignore cloud-config Add mounting of root partition Add better documentation Co-authored-by: Jan Larwig <[email protected]>

jlebon · 2024-11-07T16:56:14Z

Overall, I'm very wary of getting this in. I don't think the platform should touch the rootfs, nor that Ignition should try to mount it. I would prefer to wait until there is a metadata server available.

If we really must rely on this "config injector" temporarily, then I would extend the hack on that side so that it knows if it's e.g. FCOS or Flatcar Linux and have it inject in a different place (e.g. for FCOS, as described in #1957 (comment)).

tuunit · 2024-11-07T17:34:40Z

Overall, I'm very wary of getting this in. I don't think the platform should touch the rootfs, nor that Ignition should try to mount it. I would prefer to wait until there is a metadata server available.

If we really must rely on this "config injector" temporarily, then I would extend the hack on that side so that it knows if it's e.g. FCOS or Flatcar Linux and have it inject in a different place (e.g. for FCOS, as described in #1957 (comment)).

Thanks for your review!

One remark and maybe I just misunderstand how ignition works internally but doesn't ignition mount the rootfs anyways at /sysroot every time it has to write files and change system configurations? So why would it be a problem to temporarily mount it for a quick lookup of the user-data file?

Just for context regarding the config injector: It has been like that since the inception of the company a decade or so ago and is older than ignition itself. It is managed by another department responsible for the virtual machine management and unfortunately we cannot influence how quickly they will churn out the metadata server. The current estimation is it will take a couple of months and possibly up to a year.

jlebon · 2024-11-07T17:49:05Z

One remark and maybe I just misunderstand how ignition works internally but doesn't ignition mount the rootfs anyways at /sysroot every time it has to write files and change system configurations?

The rootfs is mounted by the OS, and the files stage is run after that.

This also all happens quite late in the initrd process, not long before switching root. On the other hand, the fetch-offline stage happens quite early in the boot process.

tuunit · 2024-11-08T08:24:40Z

So for me to fully understand the phases and issue:

Stage order: fetch-offline [-> fetch] [-> kargs] -> disks -> mount -> files.

                                       dracut(initramfs)              ignition targets
-----------------------------------------------------------------------------------------------------------
                                         basic.target
                                               |
                        ______________________/|
                       /                       |                 ignition-fetch(-offline).target
                       |                       v
                       |            initrd-root-device.target
                       |                       |
                       |                       v
                       |            dracut-pre-mount.service
                       |                       |
                       |                       v
                       |                  sysroot.mount
                       |                       |                ignition-remount-sysroot.service
                       |                       v
                       |             initrd-root-fs.target
           (custom initrd services)            |
                       |                       |                ignition-mount.target
                       |                       v                ignition-files.target
                       |             dracut-mount.service
                       |                       |
                       |                       v
                       |            initrd-parse-etc.service
                       |                       |
                       |                       v
                       |            (sysroot-usr.mount and
                       |             various mounts marked
                       |               with fstab option
                       |                x-initrd.mount)
                       |                       |
                       |                       v
                       |                initrd-fs.target
                       \______________________ |
                                              \|
                                               v
                                          initrd.target
                                               |
                                               v
                                    dracut-pre-pivot.service
                                               |
                                               v
                                     initrd-cleanup.service
                                          isolates to
                                    initrd-switch-root.target

So your concern is the fetching stage of ignition could be run before the sysroot mount is done and therefore potentially fail if the disk is yet ready?

tuunit · 2024-11-08T11:34:30Z

From my point of view: This means we will have to keep the patch files for ignition inside Flatcar for now and postpone the official addition to CoreOS and Ignition until IONOS has released a working metadata server.

CC: @mcbenjemaa @tormath1

prestist reviewed Oct 18, 2024

View reviewed changes

mcbenjemaa mentioned this pull request Oct 21, 2024

Platform Request: IONOSCloud coreos/fedora-coreos-tracker#1815

Open

mcbenjemaa force-pushed the ionoscloud branch from 6cbf57a to 48c1e5e Compare October 21, 2024 19:31

tuunit mentioned this pull request Oct 22, 2024

OEM: IONOS Cloud Images flatcar/bootengine#100

Closed

2 tasks

tormath1 reviewed Oct 22, 2024

View reviewed changes

internal/providers/ionoscloud/ionoscloud.go Outdated Show resolved Hide resolved

mcbenjemaa force-pushed the ionoscloud branch from 48c1e5e to c2cdecd Compare October 22, 2024 10:02

tormath1 reviewed Oct 22, 2024

View reviewed changes

internal/providers/ionoscloud/ionoscloud.go Outdated Show resolved Hide resolved

mcbenjemaa force-pushed the ionoscloud branch from c2cdecd to c269f14 Compare October 22, 2024 13:54

tuunit mentioned this pull request Oct 23, 2024

[RFE] IONOS Cloud support flatcar/Flatcar#1570

Open

5 tasks

jlebon reviewed Oct 23, 2024

View reviewed changes

tuunit force-pushed the ionoscloud branch 2 times, most recently from 9e2a8ff to 488d302 Compare November 1, 2024 08:42

tuunit mentioned this pull request Nov 4, 2024

OEM: IONOS Cloud Images flatcar/scripts#2389

Open

8 tasks

tuunit force-pushed the ionoscloud branch from 5f56253 to 8ecbef0 Compare November 4, 2024 12:13

mcbenjemaa commented Nov 4, 2024

View reviewed changes

tuunit force-pushed the ionoscloud branch 3 times, most recently from 4d20b27 to 7b7a434 Compare November 5, 2024 16:02

providers: add support for ionos cloud

3d4bb8d

Add support for IONOS Cloud Add check to ignore cloud-config Add mounting of root partition Add better documentation Co-authored-by: Jan Larwig <[email protected]>

tuunit force-pushed the ionoscloud branch from 7b7a434 to 3d4bb8d Compare November 5, 2024 16:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for IONOS Cloud #1957

Add support for IONOS Cloud #1957

mcbenjemaa commented Oct 17, 2024

prestist Oct 18, 2024

prestist Oct 18, 2024

mcbenjemaa Oct 18, 2024

tuunit Oct 21, 2024

prestist Oct 21, 2024

prestist commented Oct 21, 2024 •

edited

Loading

travier commented Oct 22, 2024

mcbenjemaa commented Oct 22, 2024

tuunit commented Oct 23, 2024

jlebon Oct 23, 2024

tuunit Oct 24, 2024 •

edited

Loading

tuunit Oct 24, 2024

jlebon Oct 23, 2024

tuunit Oct 24, 2024

tuunit commented Nov 4, 2024

mcbenjemaa Nov 4, 2024

tuunit Nov 4, 2024 •

edited

Loading

jlebon commented Nov 7, 2024

tuunit commented Nov 7, 2024 •

edited

Loading

jlebon commented Nov 7, 2024

tuunit commented Nov 8, 2024 •

edited

Loading

tuunit commented Nov 8, 2024 •

edited

Loading

Add support for IONOS Cloud #1957

Are you sure you want to change the base?

Add support for IONOS Cloud #1957

Conversation

mcbenjemaa commented Oct 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prestist commented Oct 21, 2024 • edited Loading

travier commented Oct 22, 2024

mcbenjemaa commented Oct 22, 2024

tuunit commented Oct 23, 2024

Choose a reason for hiding this comment

tuunit Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tuunit commented Nov 4, 2024

Choose a reason for hiding this comment

tuunit Nov 4, 2024 • edited Loading

Choose a reason for hiding this comment

jlebon commented Nov 7, 2024

tuunit commented Nov 7, 2024 • edited Loading

jlebon commented Nov 7, 2024

tuunit commented Nov 8, 2024 • edited Loading

tuunit commented Nov 8, 2024 • edited Loading

prestist commented Oct 21, 2024 •

edited

Loading

tuunit Oct 24, 2024 •

edited

Loading

tuunit Nov 4, 2024 •

edited

Loading

tuunit commented Nov 7, 2024 •

edited

Loading

tuunit commented Nov 8, 2024 •

edited

Loading

tuunit commented Nov 8, 2024 •

edited

Loading