- Introduction
- Marketplace Prerequisites
- Service Principals and Identity Providers
- CLI flags
- Cluster Definition (aka API Model)
- Azure Stack Hub Instances Registered with Azure's China cloud
- Disconnected Azure Stack Hub Instances
- Supported AKS Engine Versions
- Azure Monitor for containers
- Known Issues and Limitations
- Frequently Asked Questions
Specific AKS Engine versions can be used to provision self-managed Kubernetes clusters on Azure Stack Hub. AKS Engine's generate, deploy, upgrade, and scale commands can be executed as if you were targeting Azure's public cloud. You are only required to slightly update your cluster definition to provide some extra information about your Azure Stack Hub instance.
The goal of this guide is to explain how to provision Kubernetes clusters to Azure Stack Hub using AKS Engine and to capture the differences between Azure and Azure Stack Hub. Bear in mind as well that not every AKS Engine feature or configuration option is currently supported on Azure Stack Hub. In most cases, these are not available because dependent Azure components are not part of Azure Stack Hub.
Because Azure Stack Hub instances do not have infinite storage available, Azure Stack Hub administrators are in charge of managing it by selecting which marketplace items are downloaded from Azure's marketplace. The Azure Stack Hub administrator can follow this guide for a general explanation about how to download marketplace items from Azure.
Before you try to deploy the first Kubernetes cluster, make sure these marketplace items were made available to the target subscription by the Azure Stack Hub administrator.
Custom Script for Linux 2.0virtual machine extension- Required
AKS Base Imagevirtual machine
Kubernetes uses a service principal identity to talk to Azure Stack Hub APIs to dynamically manage resources such as storage or load balancers. Therefore, you will need to create a service principal before you can provision a Kubernetes cluster using AKS Engine.
This guide explains how to create and manage service principals on Azure Stack Hub for both Azure Active Directory (AAD) and Active Directory Federation Services (ADFS) identity providers. This other guide is a good resource to understand the permissions that the service principal requires to deploy under your subscription.
Once you have created the required service principal, make sure to assign it the contributor role at the target subscription scope.
To indicate to AKS Engine that your target platform is Azure Stack Hub, all commands require CLI flag azure-env to be set to "AzureStackCloud".
If your Azure Stack Hub instance uses ADFS to authenticate identities, then flag identity-system is also required.
aks-engine deploy \
--azure-env AzureStackCloud \
--api-model kubernetes.json \
--location local \
--resource-group kube-rg \
--identity-system adfs \ # Optional if azure_ad
--client-id $SPN_CLIENT_ID \
--client-secret $SPN_CLIENT_SECRET \
--subscription-id $TENANT_SUBSCRIPTION_ID \
--output-directory kube-rgThis section details how to tailor your cluster definitions in order to make them compatible with Azure Stack Hub. You can start off from this template if your identity provider is AAD or from this other template if you are using ADFS instead.
Unless otherwise specified down below, standard cluster definition properties should also work with Azure Stack Hub. Please create an issue if you find that we missed a property that should be called out.
| Name | Required | Description |
|---|---|---|
| location | yes | The region name of the target Azure Stack Hub. |
kubernetesConfig describes Kubernetes specific configuration.
| Name | Required | Description |
|---|---|---|
| addons | no | A few addons are not supported on Azure Stack Hub. See the complete list down below. |
| kubernetesImageBase | no | For AKS Engine versions lower than v0.48.0, this is a required field. It specifies the default image base URL to be used for all Kubernetes-related containers such as hyperkube, cloud-controller-manager, pause, addon-manager, etc. This property should be set to "mcr.microsoft.com/k8s/azurestack/core/". |
| networkPlugin | yes | Specifies the network plugin implementation for the cluster. Valid values are "kubenet" (default) for k8s software networking implementation and "azure", which provides an Azure native networking experience. |
| networkPolicy | no | Specifies the network policy enforcement tool for the cluster (currently Linux-only). Valid values are: "azure" (experimental) for Azure CNI-compliant network policy (note: Azure CNI-compliant network policy requires explicit "networkPlugin": "azure" configuration as well). |
| useInstanceMetadata | no | Use the Azure cloud provider instance metadata service for appropriate resource discovery operations. This property should be always set to "false". |
customCloudProfile contains information specific to the target Azure Stack Hub instance.
| Name | Required | Description |
|---|---|---|
| environment | no | The custom cloud type. This property should be always set to "AzureStackCloud". |
| identitySystem | yes | Specifies the identity provider used by the Azure Stack Hub instance. Valid values are "azure_ad" (default) and "adfs". |
| portalUrl | yes | The tenant portal URL. |
| dependenciesLocation | no | Specifies where to locate the dependencies required to during the provision/upgrade process. Valid values are "public" (default), "china", "german" and "usgovernment". |
masterProfile describes the settings for master configuration.
| Name | Required | Description |
|---|---|---|
| vmsize | yes | Specifies a valid Azure Stack Hub VM size. |
| distro | yes | Specifies the masters' Linux distribution. The supported value is "aks-ubuntu-16.04". This is a custom image based on UbuntuServer 16.04 that comes with pre-installed software necessary for Kubernetes deployments. |
agentPoolProfiles are used to create agents with different capabilities.
| Name | Required | Description |
|---|---|---|
| vmsize | yes | Describes a valid Azure Stack Hub VM size. |
| osType | no | Specifies the agent pool's Operating System. Supported values are "Windows" and "Linux". Defaults to "Linux". |
| distro | yes | Specifies the masters' Linux distribution. The supported value is "aks-ubuntu-16.04". This is a custom image based on UbuntuServer 16.04 that comes with pre-installed software necessary for Kubernetes deployments. |
| availabilityProfile | yes | Only "AvailabilitySet" is currently supported. |
| acceleratedNetworkingEnabled | yes | Use Azure Accelerated Networking feature for Linux agents. This property should be always set to "false". |
If your Azure Stack Hub instance is located in China, then the dependenciesLocation property of your cluster definition should be set to "china". This switch ensures that the provisioning process fetches software dependencies from reachable hosts within China's mainland.
By default, the AKS Engine provisioning process relies on an internet connection to download the software dependencies required to create or upgrade a cluster (Kubernetes images, etcd binaries, network plugins and so on).
If your Azure Stack Hub instance is air-gapped or if network connectivity in your geographical location is not reliable, then the default approach will not work, take a long time or timeout due to transient networking issues.
To overcome these issues, you should set the distro property of your cluster definition to "aks-ubuntu-16.04". This will instruct AKS Engine to deploy VM nodes using a base OS image called AKS Base Image. This custom image, generally based on Ubuntu Server, already contains the required software dependencies in its file system. Hence, internet connectivity won't be required during the provisioning process.
The AKS Base Image marketplace item has to be available in your Azure Stack Hub's Marketplace before it could be used by AKS Engine. Your Azure Stack Hub administrator can follow this guide for a general explanation about how to download marketplace items from Azure.
Each AKS Engine release is validated and tied to a specific version of the AKS Base Image. Therefore, you need to take note of the base image version required by the AKS Engine release that you plan to use, and then download exactly that base image version. New builds of the AKS Base Image are frequently released to ensure that your disconnected cluster can be upgraded to the latest supported version of each component.
| AKS Engine | AKS Base Image | Kubernetes versions | Notes |
|---|---|---|---|
| v0.43.1 | AKS Base Ubuntu 16.04-LTS Image Distro, October 2019 (2019.10.24) | 1.15.5, 1.15.4, 1.14.8, 1.14.7 | |
| v0.48.0 | AKS Base Ubuntu 16.04-LTS Image Distro, March 2020 (2020.03.19) | 1.15.10, 1.14.7 | |
| v0.51.0 | AKS Base Ubuntu 16.04-LTS Image Distro, May 2020 (2020.05.13), AKS Base Windows Image (17763.1217.200513) | 1.15.12, 1.16.8, 1.16.9 | API Model Samples (Linux, Windows) |
| v0.55.0 | AKS Base Ubuntu 16.04-LTS Image Distro, August 2020 (2020.08.24), AKS Base Windows Image (17763.1397.200820) | 1.15.12, 1.16.14, 1.17.11 | API Model Samples (Linux, Windows) |
Azure Monitor for containers can be deployed to AKS Engine clusters hosted in Azure Stack Hub Cloud Environments. Refer to Azure Monitor for containers for more details on how to onboard and monitor clusters, nodes, pods, containers inventory, performance metrics and logs.
This section lists all known issues you may find when you use the GA version.
AKS Engine includes a number of optional addons that can be deployed as part of the cluster provisioning process.
The list below includes the addons currently unsupported on Azure Stack Hub:
- AAD Pod Identity
- ACI Connector
- Blobfuse Flex Volume
- Cluster Autoscaler
- KeyVault Flex Volume
- NVIDIA Device Plugin
- Rescheduler
- SMB Flex Volume
The Basic load balancer SKU available on Azure Stack Hub limits the number of frontend IPs to 5. That implies that each cluster's agents pool is limited to 5 public IPs.
If you need to expose more than 5 services, then the recommendation is to route traffic to those services using an Ingress controller.
The output of the get-versions command only pertains to Azure and not Azure Stack Hub clouds. The different upgrade paths can be found here.
There is no official support for private-preview Kubernetes cluster with Windows nodes created with AKS Engine v0.43.1 to upgrade with AKS Engine v0.55.0. Users are encouraged to deploy new Kubernetes cluster with Windows nodes with the latest AKS Engine version.
Extensions in AKS Engine provide an easy way to include your own customization at provisioning time.
Because Azure and Azure Stack Hub currently rely on a different version of the Compute Resource Provider API, you may find that some of sample extensions fail to deploy correctly.
This can be resolved by making a small modification to the extension template.json file. Replacing all usages of template parameter apiVersionDeployments by the hard-code value 2017-12-01 (or whatever API version Azure Stack Hub runs at the time you try to deploy) should be all you need.
Once you are done updating the extension template, host the extension directory in your own Github repository or storage account. Finally, at deployment time, make sure that your cluster definition points to the new rootURL.
This how-to guide has a good high-level explanation of how AKS Engine interacts with the Azure Resource Manager (ARM) and lists a few potential issues that can cause AKS Engine commands to fail. Checkout the scripts in this repository to simplify the logs collection task.