Testing of BareMetal Cluster: Imageregistry not working

[okdlukas]

I am running a BareMetal Cluster with 3 Masters, 3 Workers and 1 Service Node. I got to the point where i can actually use the Web-Console and i setup a persistant PV on the Service Node and wanted to start a WordPress Configuration as Project to test with. However i cannot pull the needed images and register them:
Back-off pulling image "image-registry.openshift-image-registry.svc:5000/python-test/python-sample:latest"

Also i cannot reach the cluster registry anymore:

oc get clusteroperators
NAME                                       VERSION                          AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.15.0-0.okd-2024-03-10-010116   True        False         False      177m    
baremetal                                  4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
cloud-controller-manager                   4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
cloud-credential                           4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
cluster-autoscaler                         4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
config-operator                            4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
console                                    4.15.0-0.okd-2024-03-10-010116   True        False         False      3d3h    
control-plane-machine-set                  4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
csi-snapshot-controller                    4.15.0-0.okd-2024-03-10-010116   True        False         False      7d21h   
dns                                        4.15.0-0.okd-2024-03-10-010116   True        False         False      7d21h   
etcd                                       4.15.0-0.okd-2024-03-10-010116   True        False         False      3d12h   
image-registry                             4.15.0-0.okd-2024-03-10-010116   False       True          True       3h2m    Available: The deployment does not have available replicas...
ingress                                    4.15.0-0.okd-2024-03-10-010116   True        False         False      7d21h   
insights                                   4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
kube-apiserver                             4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
kube-controller-manager                    4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
kube-scheduler                             4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
kube-storage-version-migrator              4.15.0-0.okd-2024-03-10-010116   True        False         False      7d      
machine-api                                4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
machine-approver                           4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
machine-config                             4.15.0-0.okd-2024-03-10-010116   True        False         False      7d      
marketplace                                4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
monitoring                                 4.15.0-0.okd-2024-03-10-010116   True        False         False      3d4h    
network                                    4.15.0-0.okd-2024-03-10-010116   True        False         False      7d21h   
node-tuning                                4.15.0-0.okd-2024-03-10-010116   True        False         False      7d1h    
openshift-apiserver                        4.15.0-0.okd-2024-03-10-010116   True        False         False      3d3h    
openshift-controller-manager               4.15.0-0.okd-2024-03-10-010116   True        False         False      7d21h   
openshift-samples                          4.15.0-0.okd-2024-03-10-010116   True        False         False      7d1h    
operator-lifecycle-manager                 4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
operator-lifecycle-manager-catalog         4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
operator-lifecycle-manager-packageserver   4.15.0-0.okd-2024-03-10-010116   True        False         False      3d12h   
service-ca                                 4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h   
storage                                    4.15.0-0.okd-2024-03-10-010116   True        False         False      7d22h 

oc get -o yaml clusteroperator image-registry
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  annotations:
    capability.openshift.io/name: ImageRegistry
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
  creationTimestamp: "2024-03-27T14:46:59Z"
  generation: 1
  name: image-registry
  ownerReferences:
  - apiVersion: config.openshift.io/v1
    controller: true
    kind: ClusterVersion
    name: version
    uid: c0b15ac1-94e7-4863-8d08-30f09c2df263
  resourceVersion: "3317162"
  uid: 11244278-867f-48ff-8397-c4445bbe5154
spec: {}
status:
  conditions:
  - lastTransitionTime: "2024-04-04T10:15:03Z"
    message: |-
      Available: The deployment does not have available replicas
      NodeCADaemonAvailable: The daemon set node-ca has available replicas
      ImagePrunerAvailable: Pruner CronJob has been created
    reason: NoReplicasAvailable
    status: "False"
    type: Available
  - lastTransitionTime: "2024-04-04T10:15:03Z"
    message: |-
      Progressing: The deployment has not completed
      NodeCADaemonProgressing: The daemon set node-ca is deployed
    reason: DeploymentNotCompleted
    status: "True"
    type: Progressing
  - lastTransitionTime: "2024-04-04T10:16:05Z"
    message: 'Degraded: The deployment does not have available replicas'
    reason: Unavailable
    status: "True"
    type: Degraded
  extension: null
  relatedObjects:
  - group: imageregistry.operator.openshift.io
    name: cluster
    resource: configs
  - group: imageregistry.operator.openshift.io
    name: cluster
    resource: imagepruners
  - group: rbac.authorization.k8s.io
    name: system:registry
    resource: clusterroles
  - group: rbac.authorization.k8s.io
    name: registry-registry-role
    resource: clusterrolebindings
  - group: rbac.authorization.k8s.io
    name: openshift-image-registry-pruner
    resource: clusterrolebindings
  - group: ""
    name: openshift-image-registry
    resource: namespaces
  versions:
  - name: operator
    version: 4.15.0-0.okd-2024-03-10-010116

I am not sure what the Problem is. But something in the imageregisty:

apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
  creationTimestamp: "2024-03-27T15:16:08Z"
  finalizers:
  - imageregistry.operator.openshift.io/finalizer
  generation: 3
  name: cluster
  resourceVersion: "3317159"
  uid: 807a1601-17af-4548-871f-796034c2a791
spec:
  httpSecret: cca5b4ccb64b0a739df9b306ab3ac5950fca4337cfb39e567453c39f9a8a7e81b58f8e6fcbfdd283de5d843a6b570e7e79fe58274ea374571542bdbebada97d2
  logLevel: Normal
  managementState: Managed
  observedConfig: null
  operatorLogLevel: Normal
  proxy: {}
  replicas: 1
  requests:
    read:
      maxWaitInQueue: 0s
    write:
      maxWaitInQueue: 0s
  rolloutStrategy: RollingUpdate
  storage:
    managementState: Managed
    pvc:
      claim: image-registry-storage
  unsupportedConfigOverrides: null
status:
  conditions:
  - lastTransitionTime: "2024-03-27T15:16:10Z"
    reason: AsExpected
    status: "False"
    type: ImageConfigControllerDegraded
  - lastTransitionTime: "2024-04-04T10:15:03Z"
    message: The deployment has not completed
    reason: DeploymentNotCompleted
    status: "True"
    type: Progressing
  - lastTransitionTime: "2024-04-04T10:15:03Z"
    message: The deployment does not have available replicas
    reason: NoReplicasAvailable
    status: "False"
    type: Available
  - lastTransitionTime: "2024-04-04T10:16:05Z"
    message: The deployment does not have available replicas
    reason: Unavailable
    status: "True"
    type: Degraded
  - lastTransitionTime: "2024-04-04T10:15:03Z"
    status: "False"
    type: Removed
  - lastTransitionTime: "2024-03-27T15:18:06Z"
    reason: AsExpected
    status: "False"
    type: ImageRegistryCertificatesControllerDegraded
reason: AsExpected
    status: "False"
    type: ImageRegistryCertificatesControllerDegraded
  - lastTransitionTime: "2024-03-27T15:18:22Z"
    reason: AsExpected
    status: "False"
    type: AzureStackCloudControllerDegraded
  - lastTransitionTime: "2024-03-27T15:30:44Z"
    message: The daemon set node-ca has available replicas
    reason: AsExpected
    status: "True"
    type: NodeCADaemonAvailable
  - lastTransitionTime: "2024-03-28T13:11:24Z"
    message: The daemon set node-ca is deployed
    reason: AsExpected
    status: "False"
    type: NodeCADaemonProgressing
  - lastTransitionTime: "2024-03-27T15:18:27Z"
    reason: AsExpected
    status: "False"
    type: NodeCADaemonControllerDegraded
  - lastTransitionTime: "2024-04-04T10:15:03Z"
    reason: PVC Exists
    status: "True"
    type: StorageExists
  generations:
  - group: apps
    hash: ""
    lastGeneration: 1
    name: node-ca
    namespace: openshift-image-registry
    resource: daemonsets
- group: apps
    hash: ""
    lastGeneration: 2
    name: image-registry
    namespace: openshift-image-registry
    resource: deployments
  observedGeneration: 3
  readyReplicas: 0
  storage:
    managementState: Managed
    pvc:
      claim: image-registry-storage
  storageManaged: true

[melledouwsma]

The image-registry operator reports Degraded: The deployment does not have available replicas. Those pods should be in namespace openshift-image-registry. My first step would be investigating why those pods are not starting or not healthy.

[melledouwsma]

Yes, this issue is probably related to storage. If you'd like a more detailed analysis, please run oc adm must-gather, compress the folder that the command will produce and upload it to a public file sharing service.

[okdlukas]

I don't even get the ROUTE of the Image registry with OC get route but i can see other PODS i created.

[okdlukas]

MustGather

[okdlukas]

Ok The Problem actually was related to the storage in that way, that the PVC had to change its name to "image-registry-storage". After that change the image registry did not throw that Error anymore. However. When i try to start a test application the registry still shows Problems. How can i solve the available replica Problem with the Image-registry.

lastTransitionTime: "2024-04-11T13:51:54Z"
    message: |-
      Available: The deployment does not have available replicas
      NodeCADaemonAvailable: The daemon set node-ca has available replicas
      ImagePrunerAvailable: Pruner CronJob has been created
    reason: NoReplicasAvailable
    status: "False"
    type: Available
  - lastTransitionTime: "2024-04-11T13:51:54Z"
    message: |-
      Progressing: The deployment has not completed
      NodeCADaemonProgressing: The daemon set node-ca is deployed
    reason: DeploymentNotCompleted
    status: "True"
    type: Progressing
  - lastTransitionTime: "2024-04-08T16:36:24Z"
    message: |-
      Degraded: The deployment does not have available replicas
      ImagePrunerDegraded: Job has reached the specified backoff limit
    reason: ImagePrunerJobFailed::Unavailable

[okdlukas]

I could solve that Error by deleting all Pods that were stuck. Now the cluster does not show any Image-Registry Alerts but it is still impossible to push any application to the registry. No matter what application

error: build error: Failed to push image: writing blob: initiating layer upload to /v2/wordpress-test/java-springboot-basic/blobs/uploads/ in image-registry.openshift-image-registry.svc:5000: received unexpected HTTP status: 500 Internal Server Error

[okdlukas]

ok because i use an NFS for the PV this seems to be a problem of Permissions set. In the OKD Documentation it says that the Permissions i set for the PV Folder is also the Permission for the NFS Share from OKD and i could likewise use a User ID instead of Setting the Folder Permission to create Apps but how can i set new Pods to a specific User ID?