Refine and detail k8s cluster upgrade policy

[consideRatio]

Builds on work by @GeorgianaElena by detailing and tightening up an k8s upgrade policy. To tighten up the policy is possible now that we have managed to get all our clusters upgraded to 1.26+ (after next week, all are at 1.27+).

• closes Decide on a cluster upgrade policy #412

Documentation preview at https://2i2c-pilot-hubs--3577.org.readthedocs.build/howto/upgrade-cluster/, inlined below:

[GeorgianaElena]

Thank you @consideRatio

[consideRatio]

Thank you @GeorgianaElena!

@damianavila this policy declares that we should check in on k8s upgrades at the beginning of each quarter to ensure we schedule required actions to get upgrades done. Do you think this is acceptable and do you have ideas on how we ensure to check in on scheduling required upgrade actions at the beginning of the quarter?

[damianavila]

@damianavila this policy declares that we should check in on k8s upgrades at the beginning of each quarter to ensure we schedule required actions to get upgrades done. Do you think this is acceptable and do you have ideas on how we ensure to check in on scheduling required upgrade actions at the beginning of the quarter?

That's a good question about recurring tasks on the maintenance load (I think another example of that is the recurrent key/secrets updates), let me think a little bit more about it and talk with @haroldcampbell to discuss how to make sure we follow the policy successfully.