Login to corporate registry with self-signed cert fails

[dschulten]

Issue Description

I need to access a corporate registry that uses a self-signed certificate. I am using a rootful podman engine with user-mode networking in Windows 10 with a corporate proxy and proxy settings that have NO_PROXY settings containing .registry.example.com as proxy exception.

Note that I have installed podman on a machine where Docker Desktop (Hyper-V) is installed, too. But I make sure to stop that docker engine when running the podman engine.

I had to go rootful with --user-mode-networking because of:

• Problem running podman commands in windows but podman ssh is working #22373
• https://stackoverflow.com/questions/75762020/podman-pull-on-win-10-fails-with-temporary-failure-in-name-resolution

I am able to execute the test container:

podman run ubi8-micro date
Sun Sep 29 11:23:07 UTC 2024

Login into the corporate registry fails:

podman login gitlab.registry.example.com:5050
Username: example-admin
Password: 
Error: authenticating creds for "gitlab.registry.example.com:5050": pinging container registry gitlab.registry.example.com:5050: Get "https://gitlab.registry.example.com:5050/v2/": tls: 
    failed to verify certificate: x509: certificate signed by unknown authority

I have added the registry's PEM file to /etc/pki/ca-trust/source/anchors/gitlab-registry-example-com.pem using vi after establishing a root session using podman machine ssh.
Then I have executed update-ca-trust, which gives me no output, normally a sign that it worked.
When I inspect the PEM file, it tells me that issuer and owner have the same DN, namely gitlab.registry.example.com.

In addition to adding the self-signed certificate as a trusted certificate, I also tried to mark the registry as insecure by adding the following entry to /etc/containers/registries.conf:

[[registry]]
location = "gitlab.registry.example.com:5050"
insecure = true

That makes no difference.

Pulling and running images from the registry also fails:

Error: initializing source docker://gitlab.registry.example.com:5050/my/image/path/image:latest: pinging container registry gitlab.registry.example.com:5050: 
    received unexpected HTTP status: 503 Service Unavailable

However, the service is not unavailable - login and pulling images in docker desktop works just fine (when I shut down the podman machine and run docker desktop instead)

I have also tried to use the Podman Desktop UI to add registries with self-signed certificates - after a lengthy period, the UI shows the following error message:

Unable to find auth info for https://gitlab.registry.example.com:5050/v2/. 
    Error: RequestError: Bad response: 503

I also tried to enable --log-level=debug, but it gives me no clue why it cannot establish trust, only where it attempts to find credentials:

λ podman --log-level=debug login gitlab.registry.example.com:5050
time="2024-10-01T08:10:27+02:00" level=info msg="podman filtering at log level debug"
time="2024-10-01T08:10:27+02:00" level=debug msg="Called login.PersistentPreRunE(podman --log-level=debug login gitlab.registry.example.com:5050)"
time="2024-10-01T08:10:27+02:00" level=debug msg="SSH Ident Key \"C:\\\\Users\\\\itbc000257\\\\.local\\\\share\\\\containers\\\\podman\\\\machine\\\\machine\" SHA256:VsXcn0fGuftsd/ZpT9TOQ/vSyhf2cx/H/qwBOQrrXVg ssh-ed25519"
time="2024-10-01T08:10:27+02:00" level=debug msg="DoRequest Method: GET URI: http://d/v5.2.3/libpod/_ping"
time="2024-10-01T08:10:27+02:00" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
time="2024-10-01T08:10:27+02:00" level=debug msg="No credentials matching gitlab.registry.example.com:5050 found in C:\\Users\\itbc000257\\.config\\containers\\auth.json"
time="2024-10-01T08:10:27+02:00" level=debug msg="No credentials matching gitlab.registry.example.com:5050 found in C:\\Users\\itbc000257\\.config\\containers\\auth.json"
time="2024-10-01T08:10:27+02:00" level=debug msg="Found an empty credential entry \"gitlab.registry.example.com:5050\" in \"C:\\\\Users\\\\itbc000257\\\\.docker\\\\config.json\" (an unhandled credential helper marker?), moving on"
time="2024-10-01T08:10:27+02:00" level=debug msg="No credentials matching gitlab.registry.example.com:5050 found in C:\\Users\\itbc000257\\.dockercfg"
time="2024-10-01T08:10:27+02:00" level=debug msg="No credentials for gitlab.registry.example.com:5050 found"
Username: egov-admin
Password:
time="2024-10-01T08:10:41+02:00" level=debug msg="Looking for TLS certificates and private keys in \\etc\\docker\\certs.d\\gitlab.registry.example.com:5050"
time="2024-10-01T08:10:41+02:00" level=debug msg="GET https://gitlab.registry.example.com:5050/v2/"
time="2024-10-01T08:10:41+02:00" level=debug msg="Ping https://gitlab.registry.example.com:5050/v2/ err Get \"https://gitlab.registry.example.com:5050/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown authority (&url.Error{Op:\"Get\", URL:\"https://gitlab.registry.example.com:5050/v2/\", Err:(*tls.CertificateVerificationError)(0xc0001b2240)})"
time="2024-10-01T08:10:41+02:00" level=debug msg="GET https://gitlab.registry.example.com:5050/v1/_ping"
time="2024-10-01T08:10:41+02:00" level=debug msg="Ping https://gitlab.registry.example.com:5050/v1/_ping err Get \"https://gitlab.registry.example.com:5050/v1/_ping\": tls: failed to verify certificate: x509: certificate signed by unknown authority (&url.Error{Op:\"Get\", URL:\"https://gitlab.registry.example.com:5050/v1/_ping\", Err:(*tls.CertificateVerificationError)(0xc00049f2c0)})"
Error: authenticating creds for "gitlab.registry.example.com:5050": pinging container registry gitlab.registry.example.com:5050: Get "https://gitlab.registry.example.com:5050/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
time="2024-10-01T08:10:41+02:00" level=debug msg="Shutting down engines"

Steps to reproduce the issue

Steps to reproduce the issue

1. podman login the.registry:5050
2. Enter credentials

Describe the results you received

Login failure, certificate signed by unknown authority

Describe the results you expected

Login should work

podman info output

`podman info` shows that my registry is marked as insecure:
λ podman info 
host:
  arch: amd64
  buildahVersion: 1.37.3
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.12-2.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 99.66
    systemPercent: 0.21
    userPercent: 0.13
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "40"
  eventLogger: journald
  freeLocks: 2045
  hostname: SN602534
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.153.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 15888338944
  memTotal: 16628350976
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-2.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.17
      commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.x86_64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 4294967296
  swapTotal: 4294967296
  uptime: 0h 20m 31.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  gitlab.registry.example.com:5050:
    Blocked: false
    Insecure: true
    Location: gitlab.registry.example.com:5050
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: gitlab.registry.example.com:5050
    PullFromMirror: ""
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 269427478528
  graphRootUsed: 1064660992
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.2.3
  Built: 1727136000
  BuiltTime: Tue Sep 24 02:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.3

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

no environment details

Additional information

no additional information

[Sativarsainath-26]

@dschulten I set up a private registry locally with a self-signed certificate, and here is the registries.conf file I used:

[[registry]]
location = "localhost:5000"
insecure = true

To log in, I used:

root@zaasvmd:~/private-registry-certs# podman login localhost:5000
Username: testuser
Password:
Login Succeeded!

Does this match your setup, or do you have a different configuration?

[dschulten]

I am trying to follow the same basic configuration, but it fails. The difference between your setup und mine could maybe lie in the fact that I am behind a corporate proxy (configured via ENV Vars) or maybe the podman version. Also, my registry is on a different machine and it is a gitlab registry, probably a different product than your registry. Furthermore, I am running rootful and in user-mode-networking.

The error message says that podman does not trust the certificate. Is there some additional log I can enable to track how podman attempts to establish trust? I have tried to add the selfsigned cert as trusted cert, but obviously that doesn't have the expected effect, not even in combination with the insecure flag.

In which file did you define the insecure registry?

[Sativarsainath-26]

In which file did you define the insecure registry?

I defined in /etc/containers/registries.conf file path.

Is there some additional log I can enable to track how podman attempts to establish trust?

yes

podman login --log-level=debug localhost:5000

[dschulten]

I defined in /etc/containers/registries.conf file path.

I take it we both previously logged into the podman machine with ssh, and edited the same file.

The --log-level=debug switch does not give me helpful information as to why the certificate is not trusted, although the registry is insecure and I have added the selfsigned certificate to trusted CAs. It only shows where it looks for credentials.
I will add its debug output to the description above.

Which podman version did you use?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Horcrux7]

I have the similar problem with "certificate signed by unknown authority". The root certificate is publish in the windows system that browsers and apps accept it as valid. Only podman does not accept it.