[Epic] 3.6: Upgrading CoW EKS clusters to 1.28

[ahhda]

Epic label: E:3.6 Upgrading CoW EKS clusters to 1.27
Planned start date: 29th April
Due date: 30th May

Summary

CoW EKS clusters are running older versions of Kubernetes i.e. 1.23. This was because we did a lift and shift migration from Gnosis and wanted to have minimum drift. As the old Kubernetes version reach end of support we must update our clusters to a newer version as soon as possible.

Acceptance Criteria

• Staging cluster upgraded to 1.27
• Production cluster upgraded to 1.27
• Addons (kube-proxy, vpc-cni, ebs-csi-driver) are added via EKS addon and are versioned
• All supported software including ingress controller, prometheus, logging are upgraded to support the latest versions

Tasks

• Launch a separate RPC nodes cluster for accelerated updates of node groups, https://github.com/cowprotocol/infrastructure/pull/1429
• Switch staging deployments to use dev cluster RPC nodes, https://github.com/cowprotocol/infrastructure/pull/1432
• Upgrade staging cluster and node groups to 1.24, cowprotocol/infrastructure#1409
• Switch production deployments to use dev cluster RPC nodes, https://github.com/cowprotocol/infrastructure/pull/1485
• Upgrade production cluster and node groups to 1.24, cowprotocol/infrastructure#1435
• Stabilise production cluster on 1.24 after the recent outage
• Upgraded staging cluster and node groups to 1.25, https://github.com/cowprotocol/infrastructure/pull/1410
• Upgrade production cluster and node groups to 1.25, https://github.com/cowprotocol/infrastructure/pull/1476
• Upgrade staging cluster and node groups to 1.26, https://github.com/cowprotocol/infrastructure/pull/1411
• Upgrade production cluster and node groups to 1.26, https://github.com/cowprotocol/infrastructure/pull/1514
• Upgrade staging cluster and node groups to 1.27, https://github.com/cowprotocol/infrastructure/pull/1502
• Upgrade production cluster and node groups to 1.27, https://github.com/cowprotocol/infrastructure/pull/1514

About switching RPC nodes

Since our internal cluster RPC nodes currently run on ephemeral storage, as soon as we upgrade the managed node group the instances need to resync the data which can take more than a day slowing down multiple version upgrades.

Changing our RPCs to use external URL's will allow us to migrate our Kubernetes cluster in an accelerated way.

The plan is to switch to these RPC's and remove our internal RPC nodes temporarily till we upgrade our clusters to the latest versions. Once the clusters are upgraded we'll switch back to internal RPC's. Erigon node on the internal cluster is taking a very long time to sync, hence we direct RPC requests for Gnosis to the OVH instance.

• staging.nodes.cow.fi points to a dev cluster running mainnet and sepolia nodes (see #1429)
• 141.95.126.150 points to a OVH instance which we were going to use for GNO validators

Viewing logs:

• For verifying the nodes mainnet and sepolia node logs, we can connect to the dev cluster and check RPC logs
• For verifying the OVH logs, we can SSH into the instance and use tmux to check logs

For just normally checking the endpoints

NODE_URL=staging.nodes.cow.fi/mainnet
REQUEST='{"id":1,"jsonrpc":"2.0","method":"eth_blockNumber"}'
curl --data $REQUEST -H "Content-Type: application/json" -X POST $NODE_URL

wscat -c ws://staging.nodes.cow.fi/sepolia-ws

[ahhda]

All upgrades have been completed
cc: @fleupold @beefbrisket911