podman farm and registries with insecure ports

[afbjorklund]

Issue Description

It seems like the new podman farm doesn't support local registries, such as docker distribution?

Steps to reproduce the issue

Steps to reproduce the issue

1. farm build --local=false --format=docker testbuild --tag localhost:5000/testbuild

Describe the results you received

Error: build: building: 1 error occurred:
* pushing image {"063b93b4f7a0c41748e4a457d56b73afc8226682d7fe57a2ee0fbb3fc7e1fea6" "docker-archive"} to registry: Invalid image name "localhost:5000/testbuild@@unknown-digest@@", unknown transport "localhost"

Describe the results you expected

It was expected to be able to parse host:port

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

Client: v4.9.0
Server: v4.8.3, "Fedora Linux 39 (Cloud Edition)", amd64

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[rhatdan]

@umohnani8 PTAL

[umohnani8]

Yes that is because you are running your registry on your local machine and it is not something that is accessible by the farm machine on localhost. To be able to push to your local registry from your farm node, you will have to make the registry accessible to other machines. I believe steps for that are to expose the registry port and use the local machine's ip address, but I haven't tested that out.

Also was your build successful? It looks like it is trying to do a premature push and I am not able to reproduce your error.

[afbjorklund]

The issue was with the parsing, not with the actual setup of the registry. The real details can be found here:

• Podman Build Farm support podman-desktop/podman-desktop#5565 (comment)

The build goes fine, and podman push works. But not through podman farm.

I still have my old deployment, but I think that registry now moved from Docker to CNCF (much like Compose):

https://docs.docker.com/registry/ (v2 was called "distribution") -> https://distribution.github.io/distribution/

It might need an example on how to set it up, since it seems to be a requirement

---

Error: cannot create manifest list without a name, value for --tag is required

Error: "testbuild" is not a full image reference name

As far as I can tell, the farm build defaults to multi-arch and manifests and thus requires a registry to push to?

It doesn't seem to be able to do a single host build, like building on a remote server and copying back to local.

[afbjorklund]

Something like this, to run the registry on localhost (i.e. 0.0.0.0):

podman run -d -p 5000:5000 --name registry docker.io/registry:2

And to mimic the default docker settings, where localhost is HTTP:

registries.conf

[[registry]]
location = "localhost:5000"
insecure = true

---

EDIT: Note, "localhost" is just an example - could be any local host.

[umohnani8]

I think I know why you are hitting that error, what version of podman are you running on your farm machine? The version there also needs to be v4.9.0, as we updated the push semantics to support the farm build work. Can you please update that and try again.

Yes, the farm requires a registry to push to for all build. Even if you build on a single machine, it will create a manifest list and push that as well. There is no restriction on how many architectures you can build for, you can build for one to a many as you would like but we will always push the built images to a registry, create a manifest list with the digests of the images pushed to the registry and then finally push that manifest list to the registry as well.

Currently there is no way to pull the images back to local. For that you will have to do a podman pull reg/image to pull the image of your architecture back.

[afbjorklund]

Ah, OK. Didn't know that it needed a newer server as well. (It was podman-4.8.3-1.fc39.x86_64)

I figured it would work with the older server version, since Fedora CoreOS isn't synched afaik

---

It seems to work much better with the parsing, after upgrading the remote version to match

[anders@lima-podman anders]$ sudo dnf install -y https://kojipkgs.fedoraproject.org/packages/podman/4.9.0/1.fc39/x86_64/podman-4.9.0-1.fc39.x86_64.rpm

But I am not sure that is reading the registries.conf, since it doesn't find the "insecure" flag:

Error: build: building: 1 error occurred:
	* pushing image {"e1d07964853a0fce53101f3e7fbbd99820a856ca7ed04f403e7d8b4d6d4f80ee" "docker-archive"} to registry: trying to reuse blob sha256:2e112031b4b923a873c8b3d685d48037e4d5ccd967b658743d93a6e56c3064b9 at destination: pinging container registry localhost:5000: Get "https://localhost:5000/v2/": http: server gave HTTP response to HTTPS client

[umohnani8]

Can you try with the --tls-verify=false flag?

We are sending authentication information from the local machine, but I think we are not sending the registries.conf information. I will add that to my todo.

[afbjorklund]

Now it worked better, got some error output at the end but the image got uploaded.

finished build for [{linux amd64 }] at "lima-podman": built e1d07964853a0fce53101f3e7fbbd99820a856ca7ed04f403e7d8b4d6d4f80ee
Getting image source signatures
Copying blob sha256:2e112031b4b923a873c8b3d685d48037e4d5ccd967b658743d93a6e56c3064b9
Copying blob sha256:3d3ad2865b7903887ed0ec784985ec3720e06f19f01916cb072da827494e97b7
Copying config sha256:e1d07964853a0fce53101f3e7fbbd99820a856ca7ed04f403e7d8b4d6d4f80ee
Writing manifest to image destination
Error: build: error clearing list "localhost:5000/testbuild"

EDIT: Probably a result of earlier testing, and trying to overwrite an image with a manifest?

Writing manifest to image destination
INFO[0005] pushed image 408dae5cebc3b7755d4e36a34b794dd9c651e0c7ef108da0dce8dd57b5e0d308 
DEBU[0005] DoRequest Method: PUT URI: http://d/v4.9.1/libpod/manifests/localhost:5000%2Ftestbuild 
DEBU[0005] DoRequest Method: POST URI: http://d/v4.9.1/libpod/manifests/localhost:5000%2Ftestbuild/registry/localhost:5000%2Ftestbuild 
Getting image list signatures
Copying 0 images generated from 1 images in list
Writing manifest list to image destination
Storing list signatures
Saved list to "localhost:5000/testbuild"
INFO[0005] build: ok                                    
DEBU[0005] Called build.PersistentPostRunE(./bin/podman-remote --log-level=debug --connection lima-podman farm build --format=docker --local=false testbuild --tag localhost:5000/testbuild --tls-verify=false) 
DEBU[0005] Shutting down engines                        

[afbjorklund]

The workaround is good enough for now, can create a separate issue for tracking the registries.conf feature.

[umohnani8]

@afbjorklund awesome! Thank you for testing this out!

Yes the error you saw is from your earlier testing that failed and we detected an image/list with an existing name and failed to clear it. I will look into making that error make more sense.

And another issue on the registries.conf part will be great, thank you! Please assign it to me.
I think we can close this issue now with the workaround.

[afbjorklund]

Will test with two Lima, one for each arch, and then I think we can close it. (+ open a backlog issue)

It doesn't work for Podman Machine, since you can only start one machine at a time (i.e. since 4.5).

[umohnani8]

Yeah the plan is to ensure that farm build works with multiple podman machines once we are able to have multiple of them running in parallel.