Cluster trying to schedule all custom pods on master nodes before required cluster components are deployed

[danielchristianschroeter]

For some unknown reason, sometimes after or during an OKD upgrade (it happens again with version 4.8.0-0.okd-2021-10-24-061736) the cluster tried to deploy all custom applications/pods to the first master node first. If this happens, the cluster is not able to deploy cluster components any more to this master node and everything stuck until I restart manually some of the custom application pods. The custom pods will be than redeployed to a worker node and OKD is able to deploy the required cluster components successfully to the master node.
We have 3 master+worker nodes and 10 worker nodes. The worker nodes have enough memory and cpu resources to deploy all pods.

Why OKD prefer to deploy all custom applications/pods on the master nodes and after that the master node do not have enough cpu/memory resources anymore for critical stuff like apiserver, etcd, kube-apiserver,..

Is there a way to tell OKD that all cluster components will be deployed first on any node and than the custom applications/pods and not vice versa?

If this is not possible, maybe we should remove the worker role from the master nodes explained at https://access.redhat.com/solutions/4564851

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

[danielchristianschroeter]

Here is the must-gather: https://drive.google.com/file/d/1jTQqDAlU17n9fKfjVNblCkTCP8Z-yn9M/view?usp=sharing
The update with "oc adm upgrade --to-latest=true" started 1:00 o'clock (CEST).
I cleaned the situation at around 8:30 o'clock (CEST).

[vrutkovs]

Your masters have both node-role.kubernetes.io/master: "" and node-role.kubernetes.io/worker: "", so its being taken into account by the scheduler. It has picked a node and tried to add as many containers as possible according to requests/limits.

There are several ways to control this:

• Add anti-affinity to app pods so that these won't be placed on one node if possible
• Set scheduler profile or customize scheduler policy
• Rearrange scheduler decisions using descheduler

[danielchristianschroeter]

I think it is no problem to use the available node resources/limits. But wouldn't it make more sense to prioritize the necessary cluster components/pods? According to your explanation, such prioritization does not seem to be foreseen. This means, if some custom applications are deployed first for some reason on a master node and for example apiserver can not deployed anymore on the master node because node limits are reached, you need to manually delete such pods until apiserver or whatever master component can be deployed successfully on the master node.
Honestly, in my eyes, removing the worker role from the master nodes seems to be the only "simple" solution to deal with such behavior. (On bare metal installations, master nodes feature both master and worker roles by default.)

[vrutkovs]

if some custom applications are deployed first for some reason on a master node

Its not a master node. Its a worker node where master components are allowed to run (or vice versa), so for the scheduler it is a good as any other worker node

[danielchristianschroeter]

Just for documentation. We decided to add the following node affinity to our custom deployments/statefulsets:

spec:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-role.kubernetes.io/master
                    operator: DoesNotExist

Now all custom Deployments/Statefulsets will never be deployed on nodes with label key "node-role.kubernetes.io/master". OKD/OpenShift stuff is still deployable on master nodes. It is not so restrictive as "oc patch schedulers.config.openshift.io/cluster --type merge -p '{"spec":{"mastersSchedulable":false}}'".