Karpenter Support

[pragmaticivan]

Hi, will this project work with Karpenter + EKS managed pools in the same cluster?

[teevans]

Hey there! Yes! This should work just fine.

[pragmaticivan]

This is done by cordoning all nodes in the cluster (other than our new g1-small node), and then reducing the node pool sizes to 0.

Mind sharing how would that work?

I know it would be able to reduce the pool to 0 for the Karpenter controllers but I don't see how that would be able to delete the Karpenter Managed Nodes since they are not managed by EKS.

[pragmaticivan]

Here's what I'm roughly expecting:

Cluster ABC in EKS

1 Node Pool for Karpenter (1 node)
1 Node Pool for CriticalADdons (coreDNS) (1 node)
Multiple EC2 images managed by Karpenter (No EKS pools here)

From 8pm-5am I want:

1. ClusterTurnDown will create a new tiny pool for its controller.
2. ClusterTurnDown will cordon the 2 EKS pools available
3. ClusterTurnDown should pause Karpenter controller
4. ClusterTurnDown should TErminate all nodes managed by Karpenter
5. ClusterTurnDown should resize the 2 EKs pools to 0
6. Profit

[michaelmdresser]

I don't believe that Cluster Turndown has been tested with EKS+Karpenter. The general turndown implementation can be found here:

• The most important one, which uses the links below - the Turndown Manager:

  cluster-turndown/pkg/turndown/turndown.go

  Line 154 in b6ddfe3

  func (ktdm *KubernetesTurndownManager) ScaleDownCluster() error {

• The flattener, which is used when the cluster has autoscaling groups:

  cluster-turndown/pkg/cluster/flattener.go

  Lines 29 to 30 in b6ddfe3

  	// Flattener is the type used to set specific kubernetes annotations and configurations\
  	// to entice the autoscaler to downscale the cluster.

• Node draining: https://github.com/kubecost/cluster-turndown/blob/develop/pkg/cluster/draininator.go
• The ClusterProvider interface: https://github.com/kubecost/cluster-turndown/blob/develop/pkg/cluster/provider/clusterprovider.go
• EKS ClusterProvider implementation: https://github.com/kubecost/cluster-turndown/blob/develop/pkg/cluster/provider/eksclusterprovider.go

I suspect that Cluster Turndown would end up fighting Karpenter a bit because it currently does not interact with Karpenter, meaning it won't be able to pause the Karpenter controller as you suggest.

[vinicius-loureiro-lacerda]

One thing that I think it is important to mention when using Karpenter + Cluster-Turndown is that the Cluster-Turndown apparently needs to be scheduled on EKS Node Groups and not on Karpenter Node Pools, otherwise it will throw the errors below:

2024-04-09T17:58:47Z INF Determined to be running in a cluster. Using in-cluster K8s config.
2024-04-09T17:58:47Z DBG Recommendation service IsAvailable() GET finished endpoint=http://kubecost-release-cost-analyzer.kubecost:9090/model/savings/requestSizingV2 status=500
2024-04-09T17:58:47Z INF Kubescaler run started
2024-04-09T17:58:47Z INF Starting main loop
2024-04-09T17:58:47Z INF Found ProviderID starting with "aws" and eks nodegroup, using EKS Provider
2024-04-09T17:58:47Z WRN Failed to load valid access key from secret. Err=Failed to locate service account file: /var/keys/service-key.json
2024-04-09T17:58:47Z DBG No workloads have autoscaling enabled. Sleeping for a while.
2024-04-09T17:58:47Z INF Could not find NodeGroup '' in Cluster '<CLUSTER_NAME>' error="InvalidParameter: 1 validation error(s) found.\n- minimum field size of 1, DescribeNodegroupInput.NodegroupName.\n"
2024-04-09T17:58:47Z ERR Failed to initialize cluster provider. Components like Turndown, Continuous Cluster Sizing, and 1-click Cluster Sizing will not initialize. error="initializing cluster data: Failed to locate Clusters which have node groups containing the current instance: <INSTANCE_ID>"
2024-04-09T17:58:47Z DBG Cluster Info service IsAvailable() GET finished endpoint=http://kubecost-release-cost-analyzer.kubecost:9090/model/savings/abandonedWorkloads status=200

You guys can correct me if I'm wrong, but I couldn't manage to make it run on Karpenter-created nodes because Karpenter doesn't create EKS Node Groups.

I can also send the configuration we're using if needed, but at least for now the behavior here was:
-> Running on EKS Node Group nodes: ✅
-> Running on Karpenter nodes: ❌

[Smana]

@vinicius-loureiro-lacerda Hello, I'm not sure to understand. Does it mean that if the Cluster turndown runs on an EKS node group, it is able to shutdown even karpenter nodes? (e.g. all nodepools...).
If so why is this issue still opened :) ?