[CI:BUILD] Podman FCOS image from main #19477

lsm5 · 2023-08-01T19:52:51Z

Does this PR introduce a user-facing change?

None

lsm5 · 2023-08-01T19:54:50Z

TODO: add https://github.com/packit/wait-for-copr as a github action here.

lsm5 · 2023-08-02T17:49:30Z

@cevich could you PTAL at the included github action? (RE: #19448)

goals of the github action:

install wait-for-copr tool
use wait-for-copr tool to wait for a successful podman rpm build on podman-next copr
build an fcos image using the latest podman rpm from podman-next

$ wait-for-copr --owner rhcontainerbot --project podman-next podman `git rev-parse --short origin/main`
Waiting for podman release `cd5ce6372` in rhcontainerbot/podman-next (max_tries=180 * 10s)
Last successful: 102:4.5.0.rc1-1.20230802154918695899.main.1301.gcd5ce6372
Built found: 102:4.5.0.rc1-1.20230802154918695899.main.1301.gcd5ce6372

lsm5 · 2023-08-02T17:52:25Z

@benoitf in the github issue you were looking for this image to be published at https://quay.io/repository/podman/upstream?tab=tags ? AFAICT, the upstream is already handled by contrib/podmanimage/upstream so I'm thinking we could do an additional https://quay.io/repository/podman/fcos instead.

benoitf · 2023-08-02T18:09:24Z

hi, @lsm5 in the issue I said I'm fine with any name
What I would like is if possible to not only rolling tag but keep latest and git/sha version so it's easy to switch to a specific podman version

Also, would it include 'released' version like : latest = latest stable version like 4.6.0 and testing = current git sha

lsm5 · 2023-08-02T18:21:40Z

hi, @lsm5 in the issue I said I'm fine with any name What I would like is if possible to not only rolling tag but keep latest and git/sha version so it's easy to switch to a specific podman version

I'm thinking an explicit enough rolling tag podman-next / rhcontainerbot-podman-next.

I'm ok with commit SHAs as tags, but wondering if we'll end up with a really huge tag list, and if that's tolerable. @containers/podman-maintainers wdyt? If that's too much to be handled in PR comments, we could move this over to a discussion or something.

Also, would it include 'released' version like : latest = latest stable version like 4.6.0 and testing = current git sha

Almost never, assuming I'm reading your question correctly. podman-next copr has whatever exists on the main branch. For podman we cut releases from the release branches.

lsm5 · 2023-08-02T18:24:59Z

Also, would it include 'released' version like : latest = latest stable version like 4.6.0 and testing = current git sha

Almost never, assuming I'm reading your question correctly. podman-next copr has whatever exists on the main branch. For podman we cut releases from the release branches.

The container images built from contrib/podmanimage have stable / testing etc using rpms from Fedora updates / updates-testing. If we want fcos images tagged latest / testing etc. maybe we need a separate quay account / project to handle this.

benoitf · 2023-08-02T18:28:41Z

About tags you can set an expiration date like one month or 3 weeks on quay.io

So it won't keep all tags forever

https://docs.projectquay.io/use_quay.html#tag-expiration

quay.expires-after=3w LABEL for example

lsm5 · 2023-08-02T18:45:53Z

About tags you can set an expiration date like one month or 3 weeks on quay.io

So it won't keep all tags forever

https://docs.projectquay.io/use_quay.html#tag-expiration

quay.expires-after=3w LABEL for example

ohh my bad. I was under the impression you were talking about pushing those tags to upstream podman. tags in quay should be fine with expiration date. Thanks @benoitf .

TomSweeneyRedHat · 2023-08-02T18:57:26Z

FWIW, we could do any of:

https://quay.io/repository/containers/fcos
https://quay.io/repository/containers/podman/fcos
https://quay.io/repository/podman/fcos

or possibly others, your pick.

lsm5 · 2023-08-02T19:25:26Z

@TomSweeneyRedHat I'm thinking:

https://quay.io/repository/podman/fcos so it stays within the existing org. And then we have:

quay.io/podman/fcos:podman-next points to the latest packages from podman-next [suffices for this PR]
quay.io/podman/fcos:stable packages from Fedora stable updates
quay.io/podman/fcos:testing packages from Fedora updates-testing

and as @benoitf suggested we also keep tags named with the actual commit ID with an expiration date set.

cevich

First off the positive: You're brave for figuring out GHA enough to write this. Nice job, well done, and good for you 👍

Unfortunately, I think there are quite a few changes needed here to make it reliable/stable enough to support high-velocity development in this repo.

Taking a step back (I'm not 100% clear on what this job does), it currently only runs on every push to main, and doesn't seem to really need or touch any low-level github specific things. That makes me think maybe it would be better implemented as a script for a new cirrus-task (this would address many of my concerns above). Of course, I'm happy to help with this if it's a viable option.

Major Bias note: I've been cut, stabbed, shot, spanked, mutilated, and often barely escaped by the skin-of-my-teeth. After probably hundreds of hours fighting with GHA "stuffs" since Beta. IMHO, it's a badly architected automation system, immature (still), overly complex (in subtle ways), and fragile (breaking changes happen). Again IMHO, if/when/where possible, most automation should avoid using it in favor of our standard tried/true/boring/ancient/mature/non-fancy Cirrus-CI solution.)

.github/workflows/fcos-build.yml

lsm5 · 2023-08-03T18:22:49Z

First off the positive: You're brave for figuring out GHA enough to write this. Nice job, well done, and good for you +1

chatgpt created the structure, i filled in the blanks 😆

Unfortunately, I think there are quite a few changes needed here to make it reliable/stable enough to support high-velocity development in this repo.

Taking a step back (I'm not 100% clear on what this job does), it currently only runs on every push to main, and doesn't seem to really need or touch any low-level github specific things. That makes me think maybe it would be better implemented as a script for a new cirrus-task (this would address many of my concerns above). Of course, I'm happy to help with this if it's a viable option.

install wait-for-copr tool
use wait-for-copr tool to wait for a successful podman rpm build on podman-next copr
build an fcos image using the latest podman rpm from podman-next

Those are the 3 things we need. Packit people suggested using wait-for-copr as a GHA, that's why the GHA in this PR, but if you think Cirrus is the way to go, I'm happy to change this.

Major Bias note: I've been cut, stabbed, shot, spanked, mutilated, and often barely escaped by the skin-of-my-teeth. After probably hundreds of hours fighting with GHA "stuffs" since Beta. IMHO, it's a badly architected automation system, immature (still), overly complex (in subtle ways), and fragile (breaking changes happen). Again IMHO, if/when/where possible, most automation should avoid using it in favor of our standard tried/true/boring/ancient/mature/non-fancy Cirrus-CI solution.)

Yup, I'm aware of your thoughts on this :D . Just that I saw many other GHA's already, so I thought I'd go ahead.

TomSweeneyRedHat · 2023-08-03T18:28:06Z

@lsm5 your naming suggestion works for me.

cevich · 2023-08-03T18:56:11Z

Yup, I'm aware of your thoughts on this :D . Just that I saw many other GHA's already, so I thought I'd go ahead.

Ya, hence my positive comment up-front. Knowing how GHA works in general is a useful skill to have.

Those are the 3 things we need.

If I understand things right (maybe left, maybe not at all), I think the first two items could be valuable to do at the PR level. Confirming Cirrus (or GHA workflow) is able to successfully sync. with both the build, and confirm the artifact is "good" (i.e. maybe curl it down, do a test install, other simple checks).

For the third item, maybe we can do the build in the PR test-context but throw away the image at the end. If the run is for the main branch (i.e. post-merge), then it could do something else (if needed)?

(Please correct my misunderstandings)

lsm5 · 2023-08-04T19:12:43Z

Yup, I'm aware of your thoughts on this :D . Just that I saw many other GHA's already, so I thought I'd go ahead.

Ya, hence my positive comment up-front. Knowing how GHA works in general is a useful skill to have.

ack 😄

Those are the 3 things we need.

If I understand things right (maybe left, maybe not at all), I think the first two items could be valuable to do at the PR level. Confirming Cirrus (or GHA workflow) is able to successfully sync. with both the build, and confirm the artifact is "good" (i.e. maybe curl it down, do a test install, other simple checks).

confirm the artifact is good would likely need end-user testing on MacOS (??). Probably not something doable at the PR level I guess.

For the third item, maybe we can do the build in the PR test-context but throw away the image at the end. If the run is for the main branch (i.e. post-merge), then it could do something else (if needed)?

(Please correct my misunderstandings)

@benoitf is looking for end-user consumable fcos images, so ideally the end result should be an image pushed to quay.io/podman/fcos:podman-next (podman-next always pointing to the latest along with git tags with set expiration time). throw away the image should happen after a few days after the image has lived on quay.io.

AFAICT, all of these steps can/should run only after the PR has merged.

benoitf · 2023-08-04T19:44:04Z

@lsm5 I confirm the use case. To have an image on quay.io with tag being the git sha each time a PR is merged on the main branch.

cevich · 2023-08-07T14:44:41Z

confirm the artifact is good would likely need end-user testing on MacOS

It's just a container image no? This could be a really simple check, like does podman inspect run on it and are some relevant key/values present (i.e. labels, entrypoint, etc.).

I would suggest running the build as a new Cirrus-CI task (using a VM) with the following features:

Include in it's only_if something like $CIRRUS_CRON == '' (among any other applicable exclusions)
The version or commit of wait-for-copr can be set as task env var (In a future PR I/we can teach Renovate how to update it)
The version or commit of the https://github.com/coreos/layering-examples repo can also be set as an env var (again, a future PR can teach renovate how to update this).
In setup_environment.sh, for this new task only, download wait-for-copr and clone the coreos repo (referencing the above env vars).
In runner.sh, for this new task only:
1. If $CIRRUS_PR != '' (the task is running in a PR), trigger a scratch build of some kind and use wait-for-copr on it (or some other already built artifact) - the point is to simply confirm wait-for-copr is functioning.
2. If $CIRRUS_PR == '' && $CIRRUS_BRANCH == 'main, trigger an actual build (if necessary) and call wait-for-copr on it.
3. Check the built image conforms to some minimal standard (can be very simple checks as mentioned above).
4. If $CIRRUS_PR == '' && $CIRRUS_BRANCH == 'main, push the image to quay (this will need a robot account added to the quay namespace).

cevich · 2023-09-01T14:03:19Z

LGTM, though please remember to drop a comment re: followup PR where you're going to work on the Cirrus task. If you can put it in a code or git-commit that would be even better.
Otherwise future maintainers will be thoroughly confused why you're checking in commented-out code.

cevich · 2023-09-01T14:03:58Z

Ahh there, you go. Thanks.
#19830

lsm5 · 2023-09-01T15:07:30Z

@cevich removed draft status. mind going ahead with the final slash-lgtm ?

cevich

/lgtm

cevich · 2023-09-01T18:16:11Z

.github/workflows/fcos-podman-next-build.yml

+
+env:
+  IMAGE_NAME: fcos
+  IMAGE_TAGS: latest next podman-next ${{ github.sha }}


Minor: I don't really see a need for next or podman-next, since latest will be effectively the exact same thing.

Also Minor: Using a "naked" commit sha as a tag makes me slightly uncomfortable. It makes sense as an annotation or label for sure. But any lay-person looking at an image named "fcos" on quay, will likely have no clue what the sha belongs to. I'm not 100% on what the intended use of this image is, but I would think something like the full podman version would be more helpful.

Since you're setting an expiration of these tags, it's fine to fixup this stuff later.

Hmm, now that I think of it, someone browsing: quay.io/podman/fcos and looking at the tag latest would assume it's stable, like for example: fedora:latest points to fedora's latest release. Maybe latest should be removed. If I gotta choose between next and podman-next, I'll choose podman-next.

Ya, `podman-next' is way more meaningful than 'latest'

cevich · 2023-09-01T18:28:30Z

.github/workflows/fcos-podman-next-build.yml

+        image: ${{ env.IMAGE_REGISTRY }}/${{ env.IMAGE_NAME }}
+        tags: ${{env.IMAGE_TAGS }}
+        containerfiles: |
+          ./contrib/podman-next/fcos-podmanimage/Containerfile .


Minor-nit: There's no need to do the | line-wrapping thing for a single line.

that's what the redhat-action doc used, so I used the same. I can remove it in followup

cevich · 2023-09-01T18:30:16Z

.github/workflows/fcos-podman-next-build.yml

+        tags: ${{ steps.build-image.outputs.tags }}
+        registry: ${{ env.IMAGE_REGISTRY }}
+        username: ${{ secrets.QUAY_PODMAN_USERNAME }}
+        password: ${{ secrets.QUAY_PODMAN_PASSWORD }}


Just as a reminder: It would be really great to have some kind of notification if the job fails. Basically nobody pays any attention to the 'Actions' tab of the repo. and it seems important to know if the build fails for some reason.

remind me, do you know of a failure notification mechanism in GHA?

The only way I know to do this is like https://github.com/containers/podman/blob/main/.github/workflows/discussion_lock.yml#L57-L68 but maybe there's a way to send an in-app notification? I never looked into it.

openshift-ci · 2023-09-01T18:32:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cevich, lsm5

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [lsm5]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

lsm5 · 2023-09-01T19:18:25Z

@benoitf let's hope this works: https://quay.io/repository/podman/fcos?tab=builds

lsm5 · 2023-09-01T19:26:01Z

@benoitf let's hope this works: https://quay.io/repository/podman/fcos?tab=builds

Ignore that one. That was a manual trigger on quay and not via GHA. I still need to fix something in the GHA which I will continue in #19830

benoitf · 2023-09-04T07:38:30Z

@lsm5 I tried the command, first part is working. No error

podman machine os apply quay.io/podman/fcos:7cde6ab
Pulling manifest: ostree-unverified-image:docker://quay.io/podman/fcos:7cde6ab
Importing: ostree-unverified-image:docker://quay.io/podman/fcos:7cde6ab (digest: sha256:8bf5a428bbd000ec69d06d9d45439109275a76046401475aca6a496be8941df9)
ostree chunk layers needed: 51 (735.3 MB)
custom layers needed: 3 (33.7 MB)
Checking out tree 4c91b67...done
Inactive base removals:
  moby-engine
Staging deployment...done
...

But then I restart the podman machine

podman machine stop
podman machine start

but it's never booting

podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
Error: EOF

qemu process is at 99-100%

I also tried on a fresh machine (no container, etc)

benoitf · 2023-09-04T13:07:09Z

created #19851 to discuss the issue

Followup on containers#19477 [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <[email protected]>

Followup on containers#19477 Remove commented out cirrus task for fcos image build with podman-next and add 2 github actions: 1 for running a simple uni-arch image build on every PR and another to actually build multiarch images and push to quay after merge. `podman --version` will also include git short sha for clarity. [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <[email protected]>

openshift-ci bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none labels Aug 1, 2023

This was linked to issues Aug 1, 2023

Fix version scheme for releases inside the copr repository #19446

Closed

Update version scheme for release inside the copr repository #19447

Closed

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 1, 2023

lsm5 linked an issue Aug 1, 2023 that may be closed by this pull request

publish FCOS image of podman to easily try development versions of Podman in macOS podman machines #19448

Closed

lsm5 force-pushed the copr-rpm-fcos branch 7 times, most recently from 9e12e95 to 258a3e6 Compare August 2, 2023 17:47

cevich requested changes Aug 3, 2023

View reviewed changes

lsm5 marked this pull request as ready for review September 1, 2023 15:06

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 1, 2023

cevich approved these changes Sep 1, 2023

View reviewed changes

openshift-ci bot assigned cevich Sep 1, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 1, 2023

openshift-merge-robot merged commit 7cde6ab into containers:main Sep 1, 2023
40 checks passed

lsm5 deleted the copr-rpm-fcos branch September 1, 2023 18:39

lsm5 added a commit to lsm5/podman that referenced this pull request Sep 4, 2023

[CI:BUILD] followup PR for fcos with podman-next

fb986ce

Followup on containers#19477 [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <[email protected]>

lsm5 added a commit to lsm5/podman that referenced this pull request Sep 4, 2023

[CI:BUILD] followup PR for fcos with podman-next

e469bd9

Followup on containers#19477 [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <[email protected]>

lsm5 mentioned this pull request Sep 4, 2023

podman machine os apply command should abort if arch is different #19851

Closed

lsm5 added a commit to lsm5/podman that referenced this pull request Sep 7, 2023

[CI:BUILD] followup PR for fcos with podman-next

fb91613

Followup on containers#19477 [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <[email protected]>

lsm5 added a commit to lsm5/podman that referenced this pull request Sep 7, 2023

[CI:BUILD] followup PR for fcos with podman-next

898e6ca

Followup on containers#19477 [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <[email protected]>

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Dec 4, 2023

github-actions bot locked as resolved and limited conversation to collaborators Dec 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI:BUILD] Podman FCOS image from main #19477

[CI:BUILD] Podman FCOS image from main #19477

lsm5 commented Aug 1, 2023

lsm5 commented Aug 1, 2023

lsm5 commented Aug 2, 2023 •

edited

Loading

lsm5 commented Aug 2, 2023

benoitf commented Aug 2, 2023

lsm5 commented Aug 2, 2023

lsm5 commented Aug 2, 2023

benoitf commented Aug 2, 2023 •

edited

Loading

lsm5 commented Aug 2, 2023

TomSweeneyRedHat commented Aug 2, 2023

lsm5 commented Aug 2, 2023 •

edited

Loading

cevich left a comment

lsm5 commented Aug 3, 2023

TomSweeneyRedHat commented Aug 3, 2023

cevich commented Aug 3, 2023

lsm5 commented Aug 4, 2023

benoitf commented Aug 4, 2023

cevich commented Aug 7, 2023

cevich commented Sep 1, 2023

cevich commented Sep 1, 2023

lsm5 commented Sep 1, 2023

cevich left a comment

cevich Sep 1, 2023

lsm5 Sep 1, 2023

cevich Sep 1, 2023

cevich Sep 1, 2023

lsm5 Sep 1, 2023

cevich Sep 1, 2023

lsm5 Sep 1, 2023

cevich Sep 1, 2023

openshift-ci bot commented Sep 1, 2023

lsm5 commented Sep 1, 2023

lsm5 commented Sep 1, 2023

benoitf commented Sep 4, 2023

benoitf commented Sep 4, 2023

[CI:BUILD] Podman FCOS image from main #19477

[CI:BUILD] Podman FCOS image from main #19477

Conversation

lsm5 commented Aug 1, 2023

Does this PR introduce a user-facing change?

lsm5 commented Aug 1, 2023

lsm5 commented Aug 2, 2023 • edited Loading

lsm5 commented Aug 2, 2023

benoitf commented Aug 2, 2023

lsm5 commented Aug 2, 2023

lsm5 commented Aug 2, 2023

benoitf commented Aug 2, 2023 • edited Loading

lsm5 commented Aug 2, 2023

TomSweeneyRedHat commented Aug 2, 2023

lsm5 commented Aug 2, 2023 • edited Loading

cevich left a comment

Choose a reason for hiding this comment

lsm5 commented Aug 3, 2023

TomSweeneyRedHat commented Aug 3, 2023

cevich commented Aug 3, 2023

lsm5 commented Aug 4, 2023

benoitf commented Aug 4, 2023

cevich commented Aug 7, 2023

cevich commented Sep 1, 2023

cevich commented Sep 1, 2023

lsm5 commented Sep 1, 2023

cevich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Sep 1, 2023

lsm5 commented Sep 1, 2023

lsm5 commented Sep 1, 2023

benoitf commented Sep 4, 2023

benoitf commented Sep 4, 2023

lsm5 commented Aug 2, 2023 •

edited

Loading

benoitf commented Aug 2, 2023 •

edited

Loading

lsm5 commented Aug 2, 2023 •

edited

Loading