Add CAPI cluster autoscaling warning

[anders-elastisys]

⚠️ IMPORTANT ⚠️: This is a public repository. Make sure to not disclose:

• personal data beyond what is necessary for interacting with this Pull Request;
• business confidential information, such as customer names.

Quality gates:

• I'm aware of the Contributor Guide and did my best to follow the guidelines.
• I'm aware of the Glossary and did my best to use those terms.

Application Developer applications could potentially block the cluster autoscaler from downscaling nodes. This PR adds a warning notice about this in the Cluster API documentation.

Related to https://github.com/elastisys/ck8s-issue-tracker/issues/313

[cristiklein]

Great addition and something I learned today. Thanks!

[Xartos]

TIL that you can set emptyDir medium to memory 🤯

Question: do you know if all this can be fixed by setting "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"? I'm just thinking if we should also add that here

[anders-elastisys]

Question: do you know if all this can be fixed by setting "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"? I'm just thinking if we should also add that here

I have not tested this myself but according to the docs that seems to fix it for problematic pods The link added in the warning notice would inform about using the label, but I guess it cant hurt to make it more visible by adding it directly in this document as well.

[Xartos]

Question: do you know if all this can be fixed by setting "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"? I'm just thinking if we should also add that here

I have not tested this myself but according to the docs that seems to fix it for problematic pods The link added in the warning notice would inform about using the label, but I guess it cant hurt to make it more visible by adding it directly in this document as well.

I guess it might be good to "force" them to go through the list to maybe fix it properly (Like fix the emptyDir or whatever) and not just incentivize using this "catch-all" solution. Because as you say, they should be able to find that themselves pretty quickly anyway if they look

[anders-elastisys]

@Xartos then I will merge this as it is, I also went ahead and created an issue for possibly adding gatekeeper constraints that can help prevent issues with this in the future: elastisys/compliantkubernetes-apps#2318