Is there support to deploy Karpenter on existing clusters?

[alexo1088]

Is an existing page relevant?

https://github.com/Azure/karpenter/tree/main/examples

What karpenter features are relevant?

Provisioner

How should the docs be improved?

This page seems to indicate that there is support for existing clusters, but it points to an example that doesn't seem to exist

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[Bryce-Soghigian]

@alexo1088 you can run the self hosted version on existing clusters, but just know the surface area we tested on is limited to the cluster configuration we have the makefile create commands.

If you are using different CNI configuration, or have some addons we haven't fully explored, you may see some adverse behavior.

So the short answer to the question can I run karpenter on existing clusters? is yes. The answer to the question Should I run karpenter on my existing cluster, depends on your existing clusters configuration.

For Node Autoprovisioning preview we currently do not support enabling it on existing clusters, but plan on enabling it in the near future.

Feel free to share more here on your cluster configuration, and I can advise you further.

[AdminTurnedDevOps]

Is there an ETA of when there will be some testing/configurations for running Karpenter on an existing cluster?

[tallaxes]

We have validated that enabling NAP on the existing clusters works - as long as they satisfy certain constraints (e.g. Azure Overlay + Cilium CNI, some more on this on the NAP page). So, az aks update ... --node-provisioning-mode Auto on a compatible cluster should work. We will be relaxing some of the constraints over time.

This leaves manual (self-hosted) deployment of Karpenter on an existing cluster - again, a compatible one. While already possible, this currently requires building from source. What's missing for a good experience is a published image and Helm chart. We are working on this, expect this to be available within a week; will update this issue when done.

[philwelz]

We have validated that enabling NAP on the existing clusters works - as long as they satisfy certain constraints (e.g. Azure Overlay + Cilium CNI, some more on this on the NAP page). So, az aks update ... --node-provisioning-mode Auto on a compatible cluster should work. We will be relaxing some of the constraints over time.

This leaves manual (self-hosted) deployment of Karpenter on an existing cluster - again, a compatible one. While already possible, this currently requires building from source. What's missing for a good experience is a published image and Helm chart. We are working on this, expect this to be available within a week; will update this issue when done.

@tallaxes i tested enabling NAP on an existing cluster and it worked like charm. I also wrote a small article about it.

[alexo1088]

Thanks for all the help on this, everyone!

@philwelz Your article is great, but after following it, and ensuring that NAP is configured on my cluster, I can't seem to get workloads to trigger scaling through NAP/Karpenter. None of the resources created seem to generate useful logs, so i'm wondering if someone can help me understand what's missing? I'm assuming an actual provisioner does not need to be created, despite that CRD seemingly being installed after deploying NAP. Here's the output of commands showing NAP has been successfully installed:

╰─$ kubectl api-resources | grep -e aksnodeclasses -e nodeclaims -e nodepools
aksnodeclasses                     aksnc,aksncs        karpenter.azure.com/v1alpha2           false        AKSNodeClass
nodeclaims                                             karpenter.sh/v1beta1                   false        NodeClaim
nodepools                                              karpenter.sh/v1beta1                   false        NodePool

╰─$ kubectl get nodepools
NAME           NODECLASS
default        default
system-surge   system-surge

╰─$ kubectl get aksnodeclasses
NAME           AGE
default        90m
system-surge   90m

The inflate workload is set to only run on the non-system node pool but it remains unschedulable due to no new nodes being created:

╰─$ kubectl get pods 
NAME                       READY   STATUS    RESTARTS   AGE
inflate-7446c566bf-dvjdc   0/1     Pending   0          24m
inflate-7446c566bf-q8xrx   0/1     Pending   0          24m
inflate-7446c566bf-xtgxd   0/1     Pending   0          24m

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  24m                  default-scheduler  0/2 nodes are available: 2 node(s) had untolerated taint {CriticalAddonsOnly: true}. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  3m57s (x4 over 19m)  default-scheduler  0/2 nodes are available: 2 node(s) had untolerated taint {CriticalAddonsOnly: true}. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..

No nodeclaims or events showing anything related to karpenter. Anyone have any ideas on what may be happening? Thank you in advance!

[Bryce-Soghigian]

@alexo1088 looking on the backend for the cluster scheduling those inflate pods, it looks like karpenter is crashing due to a failure to resolve the vnet dynamically. Is your cluster using custom vnet? Currently karpenter is not compatible with CustomVNET.

[alexo1088]

thanks @Bryce-Soghigian ! I'm actually not very familiar with custom vnets in the context of AKS at all. Quick research seems to indicate that that is connected to the kubenet plugin, but I have the network plugin set to azure. Is there somewhere I should be looking to confirm if Im using custom Vnets?

[Bryce-Soghigian]

This is the error I see for your cluster.
message": "The Resource 'Microsoft.Network/virtualNetworks/aks-vnet-redacted' under resource group 'MC_redacted_redacted-aks_eastus2' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix

Can you check under the MC Resource group if you have a vnet there? It could be a bug if you didn't configure custom vnet it may be retrieving the wrong vnetName as well.

In the code we retrieve the cluster vnet by the mc resource group: https://github.com/Azure/karpenter-provider-azure/blob/main/pkg/providers/instance/azure_client.go#L117C1-L118C1

[alexo1088]

Thanks @Bryce-Soghigian

So, the RG I deploy NAP into is not the MC RG, it's the RG that I used to create the cluster, which is also where the VNET is deployed into as well. There is no VNET associated with that MC RG. Is this expected behavior?

[Vinaum8]

i used this command and worked for me:

az aks update --name $CLUSTER_NAME --resource-group $CLUSTER_RG --network-plugin azure --network-plugin-mode overlay

With AZ CLI version:

{
  "azure-cli": "2.58.0",
  "azure-cli-core": "2.58.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {
    "aks-preview": "2.0.0b4"
  }
}

[alexo1088]

Thanks, @Vinaum8.

@Bryce-Soghigian
I re-ran all of the commands, and I'm now seeing from the output that it seems to be automatically associating Karpenter with the RG you mentioned above, despite me making sure that I'm defining the RG variable with a different name:

Here's what I ran:

CLUSTER_NAME=*******-aks-aks
CLUSTER_RG=******-aks

az aks update \
  --name $CLUSTER_NAME \
  --resource-group $CLUSTER_RG \
  --node-provisioning-mode Auto

Here are some excerpts from the output after the command:

        "resourceId": "/subscriptions/XXXXXXXXXXXX/resourcegroups/MC_XXXXX-aks_XXXXX-aks-aks_eastus2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/XXXXXXX"
      }

Basically, all resources are being created in the MC RG. This RG has no Vnet associated with it, and I don't seem to be able to control which RG the Karpenter resources get deployed into, since it didn't seem to honor the variable I defined when running the first command. Can someone help me understand if this is a bug that needs to be addressed?

[Bryce-Soghigian]

@alexo1088 let me create a separate issue to discuss custom vnet and other networking scenarios

[tallaxes]

Issue for custom VNET support: #199

[Bryce-Soghigian]

Created a new issue cutting more specific requirements for a first pass, see: https://github.com/Azure/karpenter-provider-azure/issues/231. 

Will have PR and design for this within the week

[CCOLLOT]

@alexo1088 Did you get it to work?

I have the same symptoms (the helm releases are here, along with the NodeClass, NodePool etc.) but no node is created when deploying workloads.

The VNET is in xxx-aks-networking-rg
The cluster is in xxx-aks-rg
The system vmss is in xxx-aks-node-rg

When looking my cluster properties, I can see:

resourceGroup: xxx-aks-rg
nodeResourceGroup: xxxx-aks-node-rg
agentPoolProfiles[0].vnetSubnetId: <id-path of the subnet in the networking rg>

I don't know what is going wrong 😞

[Bryce-Soghigian]

Custom vnets dont work as explained in that issue I shared above.. But it will i fixed it here #238. I just need to cut a new version and test things in nap then roll out a new release!

[CCOLLOT]

Oh ok my bad, I misread the PR in the first place and thought having VnetSubnetId populated would be enough.
Do you know how long it will take to get it deployed to the live Azure NAP feature (europe west in my case)?

[CCOLLOT]

@Bryce-Soghigian Is the fix already live? I'm still facing the same behavior.

[Bryce-Soghigian]

@CCOLLOT the fix is rolling out. Its reached a couple of regions. What region are you in?

I tested support with this command.

Note that to use custom vnet with nap you need to assign an rbac as well along with the feature work to get karpenter to understand the custom vnet.
Run the below commands to get that role assigned to the vnet RG and fix your existing cluster.

CLUSTER_IDENTITY_ID=$(shell az aks show --name $(AZURE_CLUSTER_NAME) --resource-group $(AZURE_RESOURCE_GROUP) | jq  -r ".identity.principalId"))
az role assignment create --assignee $(CLUSTER_IDENTITY_ID) --scope /subscriptions/$(AZURE_SUBSCRIPTION_ID)/resourceGroups/$(VNET_RG) --role "Network Contributor"

[CCOLLOT]

I'm running in Europe West.
The role is already assigned on the VNET RG but no node is created and my test pod remains pending.

[Bryce-Soghigian]

@CCOLLOT Looks like west europe is one of the last regions in the release. I can let you know when it reaches there