podman farm and registries with insecure ports

[afbjorklund]

Issue Description

It seems like the new podman farm doesn't support local registries, such as docker distribution?

Steps to reproduce the issue

Steps to reproduce the issue

1. farm build --local=false --format=docker testbuild --tag localhost:5000/testbuild

Describe the results you received

Error: build: building: 1 error occurred:
* pushing image {"063b93b4f7a0c41748e4a457d56b73afc8226682d7fe57a2ee0fbb3fc7e1fea6" "docker-archive"} to registry: Invalid image name "localhost:5000/testbuild@@unknown-digest@@", unknown transport "localhost"

Describe the results you expected

It was expected to be able to parse host:port

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

Client: v4.9.0
Server: v4.8.3, "Fedora Linux 39 (Cloud Edition)", amd64

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[rhatdan]

@umohnani8 PTAL

[umohnani8]

Yes that is because you are running your registry on your local machine and it is not something that is accessible by the farm machine on localhost. To be able to push to your local registry from your farm node, you will have to make the registry accessible to other machines. I believe steps for that are to expose the registry port and use the local machine's ip address, but I haven't tested that out.

Also was your build successful? It looks like it is trying to do a premature push and I am not able to reproduce your error.

[afbjorklund]

The issue was with the parsing, not with the actual setup of the registry. The real details can be found here:

• Podman Build Farm support podman-desktop/podman-desktop#5565 (comment)

The build goes fine, and podman push works. But not through podman farm.

I still have my old deployment, but I think that registry now moved from Docker to CNCF (much like Compose):

https://docs.docker.com/registry/ (v2 was called "distribution") -> https://distribution.github.io/distribution/

It might need an example on how to set it up, since it seems to be a requirement

---

Error: cannot create manifest list without a name, value for --tag is required

Error: "testbuild" is not a full image reference name

As far as I can tell, the farm build defaults to multi-arch and manifests and thus requires a registry to push to?

It doesn't seem to be able to do a single host build, like building on a remote server and copying back to local.

[afbjorklund]

Something like this, to run the registry on localhost (i.e. 0.0.0.0):

podman run -d -p 5000:5000 --name registry docker.io/registry:2

And to mimic the default docker settings, where localhost is HTTP:

registries.conf

[[registry]]
location = "localhost:5000"
insecure = true

---

EDIT: Note, "localhost" is just an example - could be any local host.

[umohnani8]

I think I know why you are hitting that error, what version of podman are you running on your farm machine? The version there also needs to be v4.9.0, as we updated the push semantics to support the farm build work. Can you please update that and try again.

Yes, the farm requires a registry to push to for all build. Even if you build on a single machine, it will create a manifest list and push that as well. There is no restriction on how many architectures you can build for, you can build for one to a many as you would like but we will always push the built images to a registry, create a manifest list with the digests of the images pushed to the registry and then finally push that manifest list to the registry as well.

Currently there is no way to pull the images back to local. For that you will have to do a podman pull reg/image to pull the image of your architecture back.

[afbjorklund]

Ah, OK. Didn't know that it needed a newer server as well. (It was podman-4.8.3-1.fc39.x86_64)

I figured it would work with the older server version, since Fedora CoreOS isn't synched afaik

---

It seems to work much better with the parsing, after upgrading the remote version to match

[anders@lima-podman anders]$ sudo dnf install -y https://kojipkgs.fedoraproject.org/packages/podman/4.9.0/1.fc39/x86_64/podman-4.9.0-1.fc39.x86_64.rpm

But I am not sure that is reading the registries.conf, since it doesn't find the "insecure" flag:

Error: build: building: 1 error occurred:
	* pushing image {"e1d07964853a0fce53101f3e7fbbd99820a856ca7ed04f403e7d8b4d6d4f80ee" "docker-archive"} to registry: trying to reuse blob sha256:2e112031b4b923a873c8b3d685d48037e4d5ccd967b658743d93a6e56c3064b9 at destination: pinging container registry localhost:5000: Get "https://localhost:5000/v2/": http: server gave HTTP response to HTTPS client

[umohnani8]

Can you try with the --tls-verify=false flag?

We are sending authentication information from the local machine, but I think we are not sending the registries.conf information. I will add that to my todo.

[afbjorklund]

Now it worked better, got some error output at the end but the image got uploaded.

finished build for [{linux amd64 }] at "lima-podman": built e1d07964853a0fce53101f3e7fbbd99820a856ca7ed04f403e7d8b4d6d4f80ee
Getting image source signatures
Copying blob sha256:2e112031b4b923a873c8b3d685d48037e4d5ccd967b658743d93a6e56c3064b9
Copying blob sha256:3d3ad2865b7903887ed0ec784985ec3720e06f19f01916cb072da827494e97b7
Copying config sha256:e1d07964853a0fce53101f3e7fbbd99820a856ca7ed04f403e7d8b4d6d4f80ee
Writing manifest to image destination
Error: build: error clearing list "localhost:5000/testbuild"

EDIT: Probably a result of earlier testing, and trying to overwrite an image with a manifest?

Writing manifest to image destination
INFO[0005] pushed image 408dae5cebc3b7755d4e36a34b794dd9c651e0c7ef108da0dce8dd57b5e0d308 
DEBU[0005] DoRequest Method: PUT URI: http://d/v4.9.1/libpod/manifests/localhost:5000%2Ftestbuild 
DEBU[0005] DoRequest Method: POST URI: http://d/v4.9.1/libpod/manifests/localhost:5000%2Ftestbuild/registry/localhost:5000%2Ftestbuild 
Getting image list signatures
Copying 0 images generated from 1 images in list
Writing manifest list to image destination
Storing list signatures
Saved list to "localhost:5000/testbuild"
INFO[0005] build: ok                                    
DEBU[0005] Called build.PersistentPostRunE(./bin/podman-remote --log-level=debug --connection lima-podman farm build --format=docker --local=false testbuild --tag localhost:5000/testbuild --tls-verify=false) 
DEBU[0005] Shutting down engines                        

[afbjorklund]

The workaround is good enough for now, can create a separate issue for tracking the registries.conf feature.

[umohnani8]

@afbjorklund awesome! Thank you for testing this out!

Yes the error you saw is from your earlier testing that failed and we detected an image/list with an existing name and failed to clear it. I will look into making that error make more sense.

And another issue on the registries.conf part will be great, thank you! Please assign it to me.
I think we can close this issue now with the workaround.

[afbjorklund]

Will test with two Lima, one for each arch, and then I think we can close it. (+ open a backlog issue)

It doesn't work for Podman Machine, since you can only start one machine at a time (i.e. since 4.5).

[umohnani8]

Yeah the plan is to ensure that farm build works with multiple podman machines once we are able to have multiple of them running in parallel.

[afbjorklund]

Wow, running Fedora in emulation is slooow. The selinux bootup actually made cloud-init time out, for the first boot...
I think that it really wants an external server, if you for some reason can't run qemu-user but have to use qemu-system

https://lima-vm.io/docs/config/multi-arch/

Starting amd64: 20 seconds

Startup finished in 1.005s (kernel) + 1.013s (initrd) + 9.007s (userspace) = 11.026s 
multi-user.target reached after 5.173s in userspace.

Starting arm64: 6 minutes (!)

Startup finished in 4.474s (kernel) + 54.944s (initrd) + 5min 28.050s (userspace) = 6min 27.470s 
multi-user.target reached after 3min 23.722s in userspace.

But at least the feature seems to be working:

NAME             STATUS     SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
podman-amd64     Running    127.0.0.1:33687    qemu      x86_64     2       2GiB      100GiB    ~/.lima/podman-amd64
podman-arm64     Running    127.0.0.1:40861    qemu      aarch64    2       2GiB      100GiB    ~/.lima/podman-arm64

Connecting to "lima-podman-arm64"
Connecting to "lima-podman-amd64"
Builder "lima-podman-arm64" ready
Builder "lima-podman-amd64" ready
Farm "lima" ready
Starting build for [{linux arm64 v8}] at "lima-podman-arm64"
Starting build for [{linux amd64 }] at "lima-podman-amd64"
[linux/amd64@lima-podman-amd64] STEP 1/2: FROM busybox
[linux/arm64/v8@lima-podman-arm64] STEP 1/2: FROM busybox
[linux/amd64@lima-podman-amd64] STEP 2/2: RUN true
[linux/amd64@lima-podman-amd64] --> Using cache 39a4c75616b2b790fc823ee1f7f88d2d0194099c97be12a29cca6b69ce49d999
[linux/amd64@lima-podman-amd64] --> 39a4c75616b2
[linux/amd64@lima-podman-amd64] 39a4c75616b2b790fc823ee1f7f88d2d0194099c97be12a29cca6b69ce49d999
finished build for [{linux amd64 }] at "lima-podman-amd64": built 39a4c75616b2b790fc823ee1f7f88d2d0194099c97be12a29cca6b69ce49d999
[linux/arm64/v8@lima-podman-arm64] STEP 2/2: RUN true
[linux/arm64/v8@lima-podman-arm64] --> Using cache 55fcadcce9849075f3186134d7560e025fb4243c7adf19cf9c89961222b8d549
[linux/arm64/v8@lima-podman-arm64] --> 55fcadcce984
[linux/arm64/v8@lima-podman-arm64] 55fcadcce9849075f3186134d7560e025fb4243c7adf19cf9c89961222b8d549
finished build for [{linux arm64 v8}] at "lima-podman-arm64": built 55fcadcce9849075f3186134d7560e025fb4243c7adf19cf9c89961222b8d549
Getting image source signatures
Copying blob sha256:28c83c9a0efd99200b55456e7e7e0919583996fec8240fbed0c974cfee609ca2
Copying blob sha256:2e112031b4b923a873c8b3d685d48037e4d5ccd967b658743d93a6e56c3064b9
Copying config sha256:39a4c75616b2b790fc823ee1f7f88d2d0194099c97be12a29cca6b69ce49d999
Writing manifest to image destination
Getting image source signatures
Copying blob sha256:47defa2824f38e7317c07de3d8b3752f11198c4e958ea48dde8b4ba1635e46ac
Copying blob sha256:cd322088bcfba290bcfa064b77165d73708a0a77209146be150b29b7fec4c366
Copying config sha256:55fcadcce9849075f3186134d7560e025fb4243c7adf19cf9c89961222b8d549
Writing manifest to image destination
Getting image list signatures
Copying 0 images generated from 2 images in list
Writing manifest list to image destination
Storing list signatures
Saved list to "host.lima.internal:5000/testbuild"

Now the build is available on the host, from localhost:5000/testbuild.

$ podman-remote-static --version
podman-remote-static version 4.9.0

$ limactl shell podman-amd64 podman --version
podman version 4.9.0
$ limactl shell podman-amd64 rpm -q podman
podman-4.9.0-1.fc39.x86_64
$ limactl shell podman-arm64 podman --version
podman version 4.9.0
$ limactl shell podman-arm64 rpm -q podman
podman-4.9.0-1.fc39.aarch64

There are some issues around it, though.

1. For some reason, podman-remote-static farm tries to always connect to a local podman socket first ?
   Even with --local=false, it is not possible to do things like list without specifying a v4.9.0 connection

Workaround: --connection=lima-podman-amd64 (or run local)

2. And of course, there was some kind of SELinux issue that prevented building on the aarch64 machine.
   /bin/sh: error while loading shared libraries: libm.so.6: cannot change memory protections

Workaround: limactl shell podman-arm64 sudo setenforce 0

[afbjorklund]

So the issue here was:

1. Both client and server needs to be 4.9.0 to use podman farm, it is not enough with just the new podman-remote

2. Using insecure registries currently does not work out-of-the-box, needs an explicit --tls-verify=false build flag

Some other takeaways:

1. The feature needs to use external servers, so that the build can happen natively instead of running in emulation

2. If you can use --platform and qemu-user, that is much faster than running virtual machines with qemu-system