[BUG] iSCSI driver does not work with Azure Linux AKS nodes

[jgmartinez]

Describe the bug
AKS and Azure Linux nodes present an error when using the Kubernetes iSCSI driver, that prevents mounting volumes using the iSCSI protocol.

This protocol is used, among other services, by storage solutions like Azure Elastic SAN.

To Reproduce
Steps to reproduce the behavior:

1. Deploy an AKS cluster in version 1.28.5 with an Azure Linux NodePool and configure it to talk to Azure Storage SAN
2. Deploy the iSCSI driver as indicated in Azure's documentation
3. Create a StorageClass, PV, PVC and Pod as in the documentation
4. Observe how the pod never gets deployed due to the error MountVolume.SetUp failed for volume "iscsiplugin-pv" : rpc error: code = Internal desc = exit status 127
5. Check the logs of the Storage Driver pod in the same node as the pod was assigned.

Expected behavior
The pod mounts the volume, gets deployed and the storage works as expected.

Screenshots

Environment (please complete the following information):

• Azure Linux - AKSCBLMariner-V2gen2-202402.12.0
• Kubernetes version - 1.28.5

Additional context

The error is most likely caused because the Azure Linux image is missing the iscsiadm binary, which is used by the driver to mount the volumes in the nodes.

This error can't be observed in Ubuntu Linux nodes, and the storage driver works as expected.

[andyzhangx]

are you able to workaround this issue by installing iscsiadm package on Mariner OS?

CloudCopilot
Mariner OS is a Linux distribution based on CentOS, so you can install the iscsi-initiator-utils package, which includes the iscsiadm command, using the yum package manager. Here are the steps:

Open a terminal window on your Mariner OS system.

Update the package list by running the following command:

sudo yum update
Install the iscsi-initiator-utils package by running the following command:

sudo yum install iscsi-initiator-utils
This command installs the iscsi-initiator-utils package, which includes the iscsiadm command.

[CasperGN]

Hi @andyzhangx,

We’ll look at this Monday however as @jgmartinez wrote we didn’t observe this when using the regular Ubuntu image for our nodes. This is what we see when ssh’ing into the iscsi pod on Azure Linux:

# cat /sbin/iscsiadm
#!/bin/sh
if [ -x /host/sbin/iscsiadm ]; then
  chroot /host /sbin/iscsiadm "$@"
elif [ -x /host/usr/local/sbin/iscsiadm ]; then
  chroot /host /usr/local/sbin/iscsiadm "$@"
elif [ -x /host/bin/iscsiadm ]; then
  chroot /host /bin/iscsiadm "$@"
elif [ -x /host/usr/local/bin/iscsiadm ]; then
  chroot /host /usr/local/bin/iscsiadm "$@"
else
  chroot /host iscsiadm "$@"
fi
# ./sbin/iscsiadm
chroot: failed to run command 'iscsiadm': No such file or directory
#

Cheers,
Casper

[jgmartinez]

Hi @andyzhangx,

We managed to make it work in Azure Linux nodes following these steps:

1. Spawn a shell in one of the working nodes by running k debug node/aks-nodepool-XXXXXXXX-vmss000000 -it --image=mcr.microsoft.com/cbl-mariner/busybox:2.0 and then chroot /host (XXXXX being your node id)
2. Run sudo yum update && sudo yum install iscsi-initiator-utils
3. Deploying the example PV, PVC and pod, but adding nodeName: aks-nodepool-XXXXXXXX-vmss000000 to the pod spec, so it gets scheduled in the node we installed the required packages.

After following these steps, we can confirm that the Azure Linux node mounts the Elastic SAN volume through iSCSI.

We can assume that adding the iscsi-initiator-utils package to the Azure Linux base image will solve this issue.

[andyzhangx]

would be fixed by Azure/AgentBaker#4159

[CasperGN]

Thanks, Andy!