Mirroring Repository and Install failed by x509

[rupang790]

Describe the bug

Trying to install OKD 4.5.0-0.okd-2020-10-15-235428 by mirroring repository to my local registry.
After starting bootstrap, check logs by journalctl -b -f -u release-image.service -u bootkube.service and it shows error "Pull failed. ...".

 6월 24 06:22:35 bootstrap.eluon.okd.com release-image-download.sh[1342]: Pull failed. Retrying mirror-registry.eluon.okd.com:5000/openshift/okd4@sha256:67cc7cb47d56237adcf0ecc2ee76446785add5fa236cd08746b55f578b9200a5...
 6월 24 06:22:35 bootstrap.eluon.okd.com release-image-download.sh[52391]: Error: unable to pull mirror-registry.eluon.okd.com:5000/openshift/okd4@sha256:67cc7cb47d56237adcf0ecc2ee76446785add5fa236cd08746b55f578b9200a5: Error initializing source docker://mirror-registry.eluon.okd.com:5000/openshift/okd4@sha256:67cc7cb47d56237adcf0ecc2ee76446785add5fa236cd08746b55f578b9200a5: error pinging docker registry mirror-registry.eluon.okd.com:5000: Get "https://mirror-registry.eluon.okd.com:5000/v2/": x509: certificate signed by unknown authority
 6월 24 06:22:35 bootstrap.eluon.okd.com release-image-download.sh[1342]: Pull failed. Retrying mirror-registry.eluon.okd.com:5000/openshift/okd4@sha256:67cc7cb47d56237adcf0ecc2ee76446785add5fa236cd08746b55f578b9200a5...

So I check the my local registry .crt value in ca-bundle.trust.crt on bootstrap, then I found the crt value is different from original. It seems some additional value was added on.

original my registry.crt file

-----BEGIN CERTIFICATE-----
MIIEQjCCAyqgAwIBAgIJANGZCCxK19gdMA0GCSqGSIb3DQEBCwUAMHsxCzAJBgNV
BAYTAktSMRQwEgYDVQQIDAtHeWVvbmdnaS1kbzERMA8GA1UEBwwIU2VvbmduYW0x
CzAJBgNVBAoMAktUMQ4wDAYDVQQLDAVFbHVvbjEmMCQGA1UEAwwdbWlycm9yLXJl
Z2lzdHJ5LmVsdW9uLm9rZC5jb20wIBcNMjEwNjI0MDEzODUxWhgPMzAyMDEwMjUw
MTM4NTFaMHsxCzAJBgNVBAYTAktSMRQwEgYDVQQIDAtHeWVvbmdnaS1kbzERMA8G
A1UEBwwIU2VvbmduYW0xCzAJBgNVBAoMAktUMQ4wDAYDVQQLDAVFbHVvbjEmMCQG
A1UEAwwdbWlycm9yLXJlZ2lzdHJ5LmVsdW9uLm9rZC5jb20wggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQDN0NS+1uy3gpAe1ad7D07VNWinBwuMW2PDQuZK
tiwZWrRYRgQekiZJ/s9ZDsz+c+jpX/XtM5NrJhEmFt70nKtHb3pdKF7fQbZFPM1F
txFWF4qE/NjVOhfJbFf3P4+08og6KqXG9tf1aCPw3k6sZDlpgib0vni+5ECSeQLW
mmFipoPVjVxIcwGSjwFkrtPNr1dSMycskBR1+jkaIg1gymaGeGnF4pcaXpwCrYMx
xEHdRlmx5d0qJdFEwVh8ZLTXTWj4RshfA7DR8jB/n8bicoThb8ZKyZ3d52Sf3Suz
xJqvgvvtuCGHZ7qI7U1NFj29bTsS+Sn+Ucq5AJji9wvnYWJXAgMBAAGjgcYwgcMw
DgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMIGbBgNVHREEgZMw
gZCCHW1pcnJvci1yZWdpc3RyeS5lbHVvbi5va2QuY29tghJyZWdpc3RyeS5yZWRo
YXQuaW+CB3F1YXkuaW+CBmdjci5pb4IJZG9ja2VyLmlvghRyZWdpc3RyeS0xLmRv
Y2tlci5pb4IRZG9ja2VyLmVsYXN0aWMuY2+CFnJlZ2lzdHJ5LmVsdW9uLm9rZC5j
b20wDQYJKoZIhvcNAQELBQADggEBABPwjVrs/zfEAr2hNGOYUQOz3rF2DWseb+N5
ssQA9Brn1Brx5p3r+2b3X3/g7vfvd6G4Sv565Ruv6UXjoRy9GEhK4gN9Dxy2KB2R
SGqCz1IuAtDTWl4OqYShCqwGd6S5fPefnooOv70WbawwY3qwtPd7yPD3VvUmjZQ0
qDg4/E2SpJ8ibKgFIOwZnWtvoGRlL/tWV4XS2+A8YETU3XtX5rWh61Jq4olpzySM
YMSJBq/se00CzZMTjYMb+BhmniPpHCSBdOw/rH8YYR8aBVe5NoxLC4Dd/F5zHoXi
+NKquK3ik6RlkZte0600OJJAC3yIhlnp9aJEJAY1ldvbXhmWYz0=
-----END CERTIFICATE-----

ca-bundle.trust.crt on bootstrap

# mirror-registry.eluon.okd.com
-----BEGIN TRUSTED CERTIFICATE-----
MIIEQjCCAyqgAwIBAgIJANGZCCxK19gdMA0GCSqGSIb3DQEBCwUAMHsxCzAJBgNV
BAYTAktSMRQwEgYDVQQIDAtHeWVvbmdnaS1kbzERMA8GA1UEBwwIU2VvbmduYW0x
CzAJBgNVBAoMAktUMQ4wDAYDVQQLDAVFbHVvbjEmMCQGA1UEAwwdbWlycm9yLXJl
Z2lzdHJ5LmVsdW9uLm9rZC5jb20wIBcNMjEwNjI0MDEzODUxWhgPMzAyMDEwMjUw
MTM4NTFaMHsxCzAJBgNVBAYTAktSMRQwEgYDVQQIDAtHeWVvbmdnaS1kbzERMA8G
A1UEBwwIU2VvbmduYW0xCzAJBgNVBAoMAktUMQ4wDAYDVQQLDAVFbHVvbjEmMCQG
A1UEAwwdbWlycm9yLXJlZ2lzdHJ5LmVsdW9uLm9rZC5jb20wggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQDN0NS+1uy3gpAe1ad7D07VNWinBwuMW2PDQuZK
tiwZWrRYRgQekiZJ/s9ZDsz+c+jpX/XtM5NrJhEmFt70nKtHb3pdKF7fQbZFPM1F
txFWF4qE/NjVOhfJbFf3P4+08og6KqXG9tf1aCPw3k6sZDlpgib0vni+5ECSeQLW
mmFipoPVjVxIcwGSjwFkrtPNr1dSMycskBR1+jkaIg1gymaGeGnF4pcaXpwCrYMx
xEHdRlmx5d0qJdFEwVh8ZLTXTWj4RshfA7DR8jB/n8bicoThb8ZKyZ3d52Sf3Suz
xJqvgvvtuCGHZ7qI7U1NFj29bTsS+Sn+Ucq5AJji9wvnYWJXAgMBAAGjgcYwgcMw
DgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMIGbBgNVHREEgZMw
gZCCHW1pcnJvci1yZWdpc3RyeS5lbHVvbi5va2QuY29tghJyZWdpc3RyeS5yZWRo
YXQuaW+CB3F1YXkuaW+CBmdjci5pb4IJZG9ja2VyLmlvghRyZWdpc3RyeS0xLmRv
Y2tlci5pb4IRZG9ja2VyLmVsYXN0aWMuY2+CFnJlZ2lzdHJ5LmVsdW9uLm9rZC5j
b20wDQYJKoZIhvcNAQELBQADggEBABPwjVrs/zfEAr2hNGOYUQOz3rF2DWseb+N5
ssQA9Brn1Brx5p3r+2b3X3/g7vfvd6G4Sv565Ruv6UXjoRy9GEhK4gN9Dxy2KB2R
SGqCz1IuAtDTWl4OqYShCqwGd6S5fPefnooOv70WbawwY3qwtPd7yPD3VvUmjZQ0
qDg4/E2SpJ8ibKgFIOwZnWtvoGRlL/tWV4XS2+A8YETU3XtX5rWh61Jq4olpzySM
YMSJBq/se00CzZMTjYMb+BhmniPpHCSBdOw/rH8YYR8aBVe5NoxLC4Dd/F5zHoXi
+NKquK3ik6RlkZte0600OJJAC3yIhlnp9aJEJAY1ldvbXhmWYz0wHwwdbWlycm9y
LXJlZ2lzdHJ5LmVsdW9uLm9rZC5jb20=
-----END TRUSTED CERTIFICATE-----

I used install-config.yaml as:

apiVersion: v1
baseDomain: okd.com
metadata:
  name: eluon

compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0

controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3

networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16

fips: false
platform:
  none: {}

imageContentSources:
- mirrors:
  - mirror-registry.eluon.okd.com:5000/openshift/okd4
  source: quay.io/openshift/okd
- mirrors:
  - mirror-registry.eluon.okd.com:5000/openshift/okd4
  source: quay.io/openshift/okd-content

additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIEQjCCAyqgAwIBAgIJANGZCCxK19gdMA0GCSqGSIb3DQEBCwUAMHsxCzAJBgNV
  BAYTAktSMRQwEgYDVQQIDAtHeWVvbmdnaS1kbzERMA8GA1UEBwwIU2VvbmduYW0x
  CzAJBgNVBAoMAktUMQ4wDAYDVQQLDAVFbHVvbjEmMCQGA1UEAwwdbWlycm9yLXJl
  Z2lzdHJ5LmVsdW9uLm9rZC5jb20wIBcNMjEwNjI0MDEzODUxWhgPMzAyMDEwMjUw
  MTM4NTFaMHsxCzAJBgNVBAYTAktSMRQwEgYDVQQIDAtHeWVvbmdnaS1kbzERMA8G
  A1UEBwwIU2VvbmduYW0xCzAJBgNVBAoMAktUMQ4wDAYDVQQLDAVFbHVvbjEmMCQG
  A1UEAwwdbWlycm9yLXJlZ2lzdHJ5LmVsdW9uLm9rZC5jb20wggEiMA0GCSqGSIb3
  DQEBAQUAA4IBDwAwggEKAoIBAQDN0NS+1uy3gpAe1ad7D07VNWinBwuMW2PDQuZK
  tiwZWrRYRgQekiZJ/s9ZDsz+c+jpX/XtM5NrJhEmFt70nKtHb3pdKF7fQbZFPM1F
  txFWF4qE/NjVOhfJbFf3P4+08og6KqXG9tf1aCPw3k6sZDlpgib0vni+5ECSeQLW
  mmFipoPVjVxIcwGSjwFkrtPNr1dSMycskBR1+jkaIg1gymaGeGnF4pcaXpwCrYMx
  xEHdRlmx5d0qJdFEwVh8ZLTXTWj4RshfA7DR8jB/n8bicoThb8ZKyZ3d52Sf3Suz
  xJqvgvvtuCGHZ7qI7U1NFj29bTsS+Sn+Ucq5AJji9wvnYWJXAgMBAAGjgcYwgcMw
  DgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMIGbBgNVHREEgZMw
  gZCCHW1pcnJvci1yZWdpc3RyeS5lbHVvbi5va2QuY29tghJyZWdpc3RyeS5yZWRo
  YXQuaW+CB3F1YXkuaW+CBmdjci5pb4IJZG9ja2VyLmlvghRyZWdpc3RyeS0xLmRv
  Y2tlci5pb4IRZG9ja2VyLmVsYXN0aWMuY2+CFnJlZ2lzdHJ5LmVsdW9uLm9rZC5j
  b20wDQYJKoZIhvcNAQELBQADggEBABPwjVrs/zfEAr2hNGOYUQOz3rF2DWseb+N5
  ssQA9Brn1Brx5p3r+2b3X3/g7vfvd6G4Sv565Ruv6UXjoRy9GEhK4gN9Dxy2KB2R
  SGqCz1IuAtDTWl4OqYShCqwGd6S5fPefnooOv70WbawwY3qwtPd7yPD3VvUmjZQ0
  qDg4/E2SpJ8ibKgFIOwZnWtvoGRlL/tWV4XS2+A8YETU3XtX5rWh61Jq4olpzySM
  YMSJBq/se00CzZMTjYMb+BhmniPpHCSBdOw/rH8YYR8aBVe5NoxLC4Dd/F5zHoXi
  +NKquK3ik6RlkZte0600OJJAC3yIhlnp9aJEJAY1ldvbXhmWYz0=
  -----END CERTIFICATE-----

pullSecret: '{"auths":{"mirror-registry.eluon.okd.com:5000":{"auth":"", "email":"[email protected]"}}}'

sshKey: |
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWEt0k9sUV4wuznCrDt++LF3d9TfU/UxV9VXOjXep9Hxk+cebg3z6GSrOrb3QZ+wGUSdB4R9/XvPN8JAXZTqs+SNwxYhnMOX6NlANlPEEs2ewbVd6fpQCyBIReVStQeT5o43ahTS1OsCJCPbsvjs2b6li4UB4q8e5fw24jjJMnglgq02Lw4T0deJ+Ybp8wD4f4qR76m+980YI1SVeHKv9a2CKxcDIFlFhHz8AkMPKYuxMe37FFWUjUesd4NiwwIVqORiZB0hYoN6emvhkF0HzbJiGfZftWQFqnGCqIJycDLrkc3cXqkOIDkyffoGQ6eWtJ6UNOjuWkkc2MZ67S3xQoHQaslUgVWHnsihZQYppuMBiVX8P70t8Ss01UvntAjGS8xhOmd+TF5Qg9eQwftQ3uMy2XfyI8gVJAJ7Mbp8+uYpcVvzAgHRzRcjbBSxSvNqPTm9PfoWXk+j/6+H8JqjcgYcr6G5Z/guYbjI7bBj01Zpz5JYukMyLTDyYlNxyJTuk= root@okd-bastion01

I think It occurs pull failed error with x509: certificate signed by unknown authority.
How could I solve this?
or Did I miss something?

Version

4.5.0-0.okd-2020-10-15-235428

How reproducible

Log bundle

[rupang790]

Is there any solution or reason that it occurred?

[LorbusChris]

Can you confirm that your registry is exposed via TLS, and that you've added the CA cert that signed the TLS cert in question (plus the entire trust chain for it, if any) to the additionalTrustBundle in your install config?

[rupang790]

@LorbusChris,
My registry environment is:

• CentOS7
• Create .crt and .key by openssl
• Using Docker-distribution for registry

And also In my install-config.yaml, I add the additionalTrustBundle with .crt values as I attached before.

I thought that the changed(Actually added values) values is related to this error. But from your question, it is not related, isn't it?

[LorbusChris]

I think you're missing a few steps in creating your certs here.
You need to create a self-signed CA cert/key, then create a TLS cert that is signed by the CA you created. Then configure the registry to use the TLS cert, and finally add the CA cert to OKD's trust bundle.

[rupang790]

First of all, thank you for your comment.
From that, I check certification on registry again and I found the reason.
It was because of old certification. After refresh and put new certification, it works well. Thank you.

[LorbusChris]

That's great :) Can you mark this discussion as answered? (I don't seem to be able to do so myself)