Unable to set Auto on nodeProvisioningProfile for existing cluster due to aadProfile

[nfsouzaj]

Version

Karpenter Version: v0.0.0

Kubernetes Version: v1.27.3

Expected Behavior

I am trying to shift an existing cluster, setting the node provisioning mode to Auto. by running the following command:
az aks update --resource-group myGroup --name myAKS --node-provisioning-mode Auto

Actual Behavior

After running the command:
az aks update --resource-group myGroup --name myAKS --node-provisioning-mode Auto
I get the following error:

Argument '--node-provisioning-mode' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
The behavior of this command has been altered by the following extension: aks-preview
(OperationNotAllowed) .properties.nodeProvisioningProfile.mode cannot be Auto if .properties.aadProfile is specified.
Code: OperationNotAllowed
Message: .properties.nodeProvisioningProfile.mode cannot be Auto if .properties.aadProfile is specified.

Steps to Reproduce the Problem

1. Created a cluster using terraform that has one nodepool with Manual Provision
2. Register the NodeAutoProvisioningPreview feature flag following this
3. Run the CLI cmd in the expected behavior to update the provision mode to Auto
4. Cli complains that the operation is not allowed

Resource Specs and Logs

AKS cluster 1.27.3
CNI: Azure with Overlay
Cilium DataPlane enabled

I am trying to create a cluster using terraform and afterwards change the autoprovision so that I can create nodepools with Karpenter however the error in the section Actual Behavior happens.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[Bryce-Soghigian]

Currently AKS does not support enabling karpenter through nap on existing clusters, so the command needs to be run on cluster create. Karpenter also will not be supporting service principal and only allow for MSI auth.

[Bryce-Soghigian]

Related to #59

[nfsouzaj]

@Bryce-Soghigian - thanks for your answer.

We are using managed identity, not SP. With regards to running the command while the cluster is being created, I am using terraform. How can I do that?
Regards,

[philwelz]

@nfsouzaj you could use the AzAPi Terraform Provider but as per docs the latest API does not support NAP. You have to wait until a new API version gets released. Until that NAP can be only enabled via azure-cli on new clusters.

[nfsouzaj]

@philwelz thanks for the answer. Is there an ETA?

[nfsouzaj]

@philwelz @Bryce-Soghigian appreciate a feedback on ETA at the earliest convenience.

[Bryce-Soghigian]

@nfsouzaj the AKS Team doesn't own the AzAPI terraform SDK :( so we will need to contact someone else for an answer there. Let me do some asking around internally and get back to you!

[Bryce-Soghigian]

@nfsouzaj talked to the azapi terraform provider team, and seems the docs that Phil shared are out of date. This is what was said.

---

The docs are not update to date, but the azapi provider does support the latest 2023-10-02-preview, here's an example of how to use the node provisioning profile:

resource "azapi_resource" "rg" {
  type     = "Microsoft.Resources/resourceGroups@2020-06-01"
  name     = "henglu0109aks"
  location = "westeurope"

}
resource "azapi_resource" "cluster" {
  type      = "Microsoft.ContainerService/managedClusters@2023-10-02-preview"
  parent_id = azapi_resource.rg.id
  name      = "henglu0109aks"
  location  = "westeurope"
  identity {
    type = "SystemAssigned"
    identity_ids = []
  }
  
  body = jsonencode({
    properties = {
      agentPoolProfiles = [
        {
          count  = 1
          mode   = "System"
          name   = "default"
          vmSize = "Standard_DS2_v2"
        },
      ]
      nodeProvisioningProfile = {
        mode = "Auto"
      }
      dnsPrefix = "henglu0109aks"
    }
  })
}

[Bryce-Soghigian]

Going to close this issue as enabling karpenter on existing clusters is an unsupported scenario for now.

[tallaxes]

Reopening, as enabling NAP on existing clusters is supported, and we determined the issue here is with rejecting addProfile altogether, whether only certain deprecated configurations should be rejected.

[nfsouzaj]

@tallaxes hi! Do you have a feedback on this? Also, would you know when will the TF provider have karpenter in it?

[tallaxes]

@nfsouzaj The fix for addProfile is included in AKS Release 2024-02-07, which should already be rolled out everywhere.

You can already enable NAP via Terraform as shown above. I don't expect any more explicit support for Karpenter/NAP in TF (at least until/unless we decide to expose some AKS API configuration surface), but you should already be able to deploy NodePools and AKSNodeClasses resources via TF's Kubernetes or kubectl provider.

[nfsouzaj]

@tallaxes sorry, I am not following how to enable the node auto provisioning on terraform. I also don't see in the portal any scale mode other than manual and autoscale. Its been really frustrating trying to use this feature.

[tallaxes]

@nfsouzaj Sorry to hear that. It is a preview feature, so ways of enabling it are still limited. (In particular, there is no Portal UX yet.)

I am not that familiar with Terraform AzAPI provider, so won't be able to help much, but the key part in the example above is setting nodeProvisioningProfile = { mode = "Auto" } , which should do the job. What I am not sure about is how it interacts with preview feature enablement ...

So your best bet / easiest way right now is likely to use Azure CLI, following the directions under Node Autoprovisioning (preview). Using CLI to enable NAP on an existing cluster - including one created with Terraform, as you were originally trying - should work as well, now that we fixed the aadProfile bug. If that path still does not work for you - please share the results, and we should be able to help.

[aslafy-z]

The Terraform implementation was not accepted. It appears there are ongoing debates about whether preview features should be incorporated into their provider: hashicorp/terraform-provider-azurerm#25084 (comment)

[nfsouzaj]

Thanks for the feedback @aslafy-z . Disappointing news...

[nfsouzaj]

@aslafy-z hi, is there news on when this is coming to tf?

[aslafy-z]

@nfsouzaj No news... I think the best thing to do is to ask your Microsoft TAM to push the feature. Hopefully it will come one day.