OKD 4.15 Deployment does not have minimum availability

[okdlukas]

Hi there, i am running a bare Metal configuration with 3 Masters, one Service Node and a few compute Nodes. We had some trouble with the DNS and Firewall Setup but now the Nodes are getting there ignition files but the Installation is not complete.
I can't access the web console and the ingress complains about missing minimum availability. I have put the minimum setting of 3 replicas into the install.yaml as i should.

oc get clusteroperators
NAME                                       VERSION                          AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.15.0-0.okd-2024-02-23-163410   False       True          True       38m     APIServerDeploymentAvailable: no apiserver.openshift-oauth-apiserver pods available on any node....
baremetal                                  4.15.0-0.okd-2024-02-23-163410   True        False         False      35m     
cloud-controller-manager                   4.15.0-0.okd-2024-02-23-163410   True        False         False      41m     
cloud-credential                           4.15.0-0.okd-2024-02-23-163410   True        False         False      43m     
cluster-autoscaler                         4.15.0-0.okd-2024-02-23-163410   True        False         False      36m     
config-operator                            4.15.0-0.okd-2024-02-23-163410   True        False         False      38m     
console                                    4.15.0-0.okd-2024-02-23-163410   False       False         False      16s     RouteHealthAvailable: route not yet available, https://console-openshift-console.apps.lab.testing.de returns '503 Service Unavailable'
control-plane-machine-set                  4.15.0-0.okd-2024-02-23-163410   True        False         False      36m     
csi-snapshot-controller                    4.15.0-0.okd-2024-02-23-163410   False       True          False      49s     CSISnapshotControllerAvailable: Waiting for Deployment...
dns                                        4.15.0-0.okd-2024-02-23-163410   False       True          True       5s      DNS "default" is unavailable.
etcd                                       4.15.0-0.okd-2024-02-23-163410   True        True          False      17m     NodeInstallerProgressing: 1 nodes are at revision 4; 1 nodes are at revision 5; 1 nodes are at revision 7; 0 nodes have achieved new revision 9
image-registry                             4.15.0-0.okd-2024-02-23-163410   False       True          False      38s     Available: The registry is removed...
ingress                                    4.15.0-0.okd-2024-02-23-163410   False       True          False      4s      The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.)
insights                                   4.15.0-0.okd-2024-02-23-163410   True        False         False      30m     
kube-apiserver                             4.15.0-0.okd-2024-02-23-163410   True        True          False      16m     NodeInstallerProgressing: 1 nodes are at revision 0; 1 nodes are at revision 4; 1 nodes are at revision 6; 0 nodes have achieved new revision 8
kube-controller-manager                    4.15.0-0.okd-2024-02-23-163410   True        True          False      25m     NodeInstallerProgressing: 1 nodes are at revision 0; 2 nodes are at revision 8
kube-scheduler                             4.15.0-0.okd-2024-02-23-163410   True        True          False      16m     NodeInstallerProgressing: 1 nodes are at revision 7; 2 nodes are at revision 8
kube-storage-version-migrator              4.15.0-0.okd-2024-02-23-163410   False       True          False      20s     KubeStorageVersionMigratorAvailable: Waiting for Deployment
machine-api                                4.15.0-0.okd-2024-02-23-163410   True        False         False      35m     
machine-approver                           4.15.0-0.okd-2024-02-23-163410   True        False         False      36m     
machine-config                             4.15.0-0.okd-2024-02-23-163410   True        False         False      34m     
marketplace                                4.15.0-0.okd-2024-02-23-163410   True        False         False      36m     
monitoring                                                                  False       True          True       27s     the server was unable to return a response in the time allotted, but may still be processing the request (get featuregates.config.openshift.io cluster)
network                                    4.15.0-0.okd-2024-02-23-163410   True        False         False      11m     
node-tuning                                4.15.0-0.okd-2024-02-23-163410   False       True          False      46s     DaemonSet "tuned" has no available Pod(s)
openshift-apiserver                        4.15.0-0.okd-2024-02-23-163410   True        True          False      43s     APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: 1/3 pods have been updated to the latest generation
openshift-controller-manager               4.15.0-0.okd-2024-02-23-163410   False       True          False      50s     Available: no pods available on any node.
openshift-samples                          4.15.0-0.okd-2024-02-23-163410   True        False         False      12m     
operator-lifecycle-manager                 4.15.0-0.okd-2024-02-23-163410   True        False         False      35m     
operator-lifecycle-manager-catalog         4.15.0-0.okd-2024-02-23-163410   True        False         False      35m     
operator-lifecycle-manager-packageserver   4.15.0-0.okd-2024-02-23-163410   False       True          False      4s      ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: ComponentUnhealthy, message: apiServices not installed
service-ca                                 4.15.0-0.okd-2024-02-23-163410   True        True          False      38m     Progressing: ...
storage                                    4.15.0-0.okd-2024-02-23-163410   True        False         False      38m     

oc describe route oauth-openshift -n openshift-authentication
Name:                   oauth-openshift
Namespace:              openshift-authentication
Created:                34 minutes ago
Labels:                 app=oauth-openshift
Annotations:            <none>
Requested Host:         oauth-openshift.apps.lab.testing.de
                           exposed on router default (host router-default.lab.testing.de) 32 minutes ago
Path:                   <none>
TLS Termination:        passthrough
Insecure Policy:        Redirect
Endpoint Port:          6443

Service:        oauth-openshift
Weight:         100 (100%)
Endpoints:      10.128.0.96:6443, 10.129.0.37:6443, 10.130.0.17:6443

oc get all -n openshift-authentication
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                                   READY   STATUS    RESTARTS   AGE
pod/oauth-openshift-76fbf9f64c-8fqdj   1/1     Running   0          35m
pod/oauth-openshift-76fbf9f64c-jdplf   1/1     Running   0          27m
pod/oauth-openshift-76fbf9f64c-tc788   1/1     Running   0          27m

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/oauth-openshift   ClusterIP   172.30.49.124   <none>        443/TCP   61m

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/oauth-openshift   3/3     3            3           35m

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/oauth-openshift-697796b4f4   0         0         0       35m
replicaset.apps/oauth-openshift-76fbf9f64c   3         3         3       35m
replicaset.apps/oauth-openshift-797485df58   0         0         0       35m

NAME                                       HOST/PORT                                              PATH   SERVICES          PORT   TERMINATION            WILDCARD
route.route.openshift.io/oauth-openshift   oauth-openshift.apps.lab.testing.de        oauth-openshift   6443   passthrough/Redirect   None

Reprinting Cluster State:
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
ClusterID: 16be3fde-378d-4c3a-b16c-c62ac193b858
ClientVersion: 4.15.0-0.okd-2024-02-23-163410
ClusterVersion: Installing "4.15.0-0.okd-2024-02-23-163410" for 17 hours: Unable to apply 4.15.0-0.okd-2024-02-23-163410: the cluster operator monitoring is not available
ClusterOperators:
        clusteroperator/ingress is degraded because The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: DeploymentReplicasAllAvailable=False (DeploymentReplicasNotAvailable: 1/2 of replicas are available)
        clusteroperator/kube-controller-manager is degraded because GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": dial tcp: lookup thanos-querier.openshift-monitoring.svc on 172.30.0.10:53: no such host
        clusteroperator/monitoring is not available (UpdatingPrometheusOperator: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: context deadline exceeded) because UpdatingPrometheusOperator: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: context deadline exceeded

[melledouwsma]

Can you attach (or upload to the public file sharing service) the result of a oc adm must-gather? More information on gathering cluster data can be found here.

[okdlukas]

Done
I attached it to the Question.

Ok, after some reading i am pretty sure the problem is that the second worker is missing but it is neccessary for the ingress.
and with the ingress missing, the services cannot work. I will test that.

[melledouwsma]

Done I attached it to the Question.

That's only the summary. oc adm must-gather produced a folder on your local machine that is named something like must-gather.local.[timestamp] containing debug information on the state of your cluster. Please compress the folder and attach it to this issue or upload it a public file-sharing service, if you'd like a more detailed analysis.

Ok, after some reading i am pretty sure the problem is that the second worker is missing but it is necessary for the ingress. and with the ingress missing, the services cannot work. I will test that.

In general, yes, there a couple of operators (like authentication and console) that are dependent on the ingress operator. The ingress operator is not available because of DeploymentReplicasNotAvailable: 1/2 of replicas are available.

[okdlukas]

I setup a second Worker Node, because that was not done correctly and after that it worked. I can access the web console and the cluster. I am still checking if something else is acting up but for now it seems to be just fine.