Couldn't find any host available to create Volume

[smejkal]

After spending many hours trying to solve the problem myself, I need help please...

I'm already using multiple LUNs (9 targets and 9 LUNs) on the DS918+ with my Kubernetes cluster, but by iscsi pv, not with the Synology CSI.

Versions:

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2
Synology CSI: latest

client-info.yml

clients:
  - host: 192.168.2.224
    port: 5000
    https: false
    username: user
    password: password

storageclass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: synology-iscsi-storage
  annotations:
    storage-class.kubernetes.io/is-default-class: "false"
provisioner: csi.san.synology.com
parameters:
  dsm: '192.168.2.224'
  location: '/volume1'
  fsType: 'btrfs'
  formatOptions: '--nodiscard'
  type: thin
reclaimPolicy: Retain
allowVolumeExpansion: true`

claim:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  namespace: test
spec:
  storageClassName: synology-iscsi-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi

job:

apiVersion: batch/v1
kind: Job
metadata:
  name: write
  namespace: test
spec:
  template:
    metadata:
      name: write
    spec:
      containers:
        - name: write
          image: registry.access.redhat.com/ubi8/ubi-minimal:latest
          command: ["dd","if=/dev/zero","of=/mnt/pv/test.img","bs=1G","count=1","oflag=dsync"]
          volumeMounts:
            - mountPath: "/mnt/pv"
              name: test-volume
      volumes:
        - name: test-volume
          persistentVolumeClaim:
            claimName: test-claim
      restartPolicy: Never

Controller log:

controller.go:1279] provision "test/test-claim" class "synology-iscsi-storage": started
connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"test", Name:"test-claim", UID:"3024a15d-4557-4c40-ba86-e0ebdf7d1ac9", APIVersion:"v1", ResourceVersion:"30936475", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "test/test-claim"
connection.go:184] GRPC request: {"capacity_range":{"required_bytes":4294967296},"name":"pvc-3024a15d-4557-4c40-ba86-e0ebdf7d1ac9","parameters":{"dsm":"192.168.2.224","formatOptions":"--nodiscard","fsType":"btrfs","location":"/volume1"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"btrfs"}},"access_mode":{"mode":1}}]}
controller.go:956] error syncing claim "3024a15d-4557-4c40-ba86-e0ebdf7d1ac9": failed to provision volume with StorageClass "synology-iscsi-storage": rpc error: code = Internal desc = Couldn't find any host available to create Volume

What does 'couldn't find any host... " mean? Not find any DSM? Not find any node?

I read all the log files 1000 times... No more ideas to resolve the issue. Not to see the forest anymore... Please help ;-)

[smejkal]

Unbelievable, I tried the same with my old ds216j and it works.

The procedure was exactly the same, just created a new user with admin rights and entered it in the client_info.yml. The only difference is the file system. With the ds2176 I now used ext4 and no brtfs. But I had already tested this with ext4 on the DS918 without success.

What else could be the reason? Any higher security settings? The firewall is disabled on both.

And different are the existing manual created luns on the 918+. Could that be the problem?

[smejkal]

The system is screwing me over.

After I changed the storage class back to the ds918+, it worked there too.

The only difference now is that the client-info.yml contains 2 entries. The template also had 2 entries. Does this mean that it only works if there are 2 entries? Please not really...

[aSauerwein]

it's working for me with only 1 client configured.
I had the same error message and my fault was that the user must have 2FA disabled and must be in administrators group

[plaurin84]

I am encountering the exact same issue.
I am not using any parameters in the storage class to keep the defaults.

My user has admin rights and 2FA is disabled also.

[plaurin84]

I am doing more tests... I believe I might have found the issue. But still trying to replicate the issue properly before sharing some more details.

I suspect there is a problem with too complex passwords in the secret. Maybe some special characters are not sent properly for authentication with DSM. Can't confirm for now.

EDIT: The password is not the issue. But also, it started working for both iscsi and samba, and I don't understand why... However, I restarted the synology-csi-contoller pod at some point and that might be it. Still investigating more.

[smejkal]

That surprises me very much. I have uninstalled and reinstalled the drivers several times. And so the pods were recreated too. Only after changing to another DS and back again did it work. But I can't explain why either.

I'm very excited about your findings.

[plaurin84]

Well... I am disappointed. It always work and I cannot reproduce the issue anymore. This is really strange.
I even deployed from a fresh Kubernetes cluster and still works out of the box without issue now.
¯\(ツ)/¯

[plaurin84]

The synology NAS I have been working on is not mine. I can't do much more at this point.

To help reproduce the issue, someone who owns a NAS and encountered the issue should probably try and factory-reset the NAS to see if it's possible to encounter the issue again. (Of course, if you don't have any data on it or you are okay with losing everything).

I hope to get one sometimes next year, I could try to figure this out again if not resolved.

[software-testing-professional]

Hi guys.
I also struggled with these Couldn't find any host available to create Volume issues on my DS923+.
After lots of try and error, I added the CSI user to the 'Administrators' group in my Synology NAS.
Since then the provisioning of volumes does work.

So the Couldn't find any host available to create Volume message seems to be related to insufficient user rights on the Synology NAS. The error message is misleading and did cost me some hours...

The message should be changed to cover this case.
Either by using a different message.
Or by extending the existing message, for example
Couldn't find any host available to create Volume or user does not have sufficient permissions to create Volume.

I did not test different filesystems in this scenario, btw.

[bplein]

I struggled with this today. I initially followed the instructions and copied the client-info-template.yml and edited to my needs.

Then I noticed that the example client-info.yml in the README.md is like this:

clients:
- host: 192.168.1.1
  port: 5000
  https: false
  username: <username>
  password: <password>
- host: 192.168.1.2
  port: 5001
  https: true
  username: <username>
  password: <password>

While the client-info-template.yml template file has the individual hosts outdented as this:

---
clients:
  - host: 192.168.1.1
    port: 5000
    https: false
    username: username
    password: password
  - host: 192.168.1.2
    port: 5000
    https: false
    username: username
    password: password

I don't know if they are equivalent YAML but I got it going with using the first example from the README. When I tried to embed the information into the storageclass, it never did seem to work for me.

[myst3k]

I was just having the same problem, editing the client-info.yml to remove the two spaces on all the lines fixed the issue.

[nuttingd]

[EDIT: This solved it for me!]

As far as I can tell, it was MFA, and specifically "Adaptive MFA" being enabled for the administrators group. I certainly want this feature to be enabled, but it sounds like this may be a concession if you plan on using the Synology CSI.

Once I made the change, I kicked the controller pod and it immediately provisioned the PVs:

kubectl delete pod -n synology-csi synology-csi-controller-0

Original comment with context:

I have been having the same intermittent problems with the error Couldn't find any host available to create Volume.

I had some success debugging the issue by running the tests against my config/client-info.yml. I was able to see that the client was getting a 403 Forbidden response from the Synology API.

Run make test from the repo after creating your client-info.yml file:

❯ make test
go clean -testcache
go test -v ./test/...
=== RUN   TestSanity
Failed to add DSM: mynas.network.lan, error: Failed to login to DSM: [mynas.network.lan]. err: DSM Api error. Error code:403
    sanity_test.go:57: No available DSM.
--- FAIL: TestSanity (0.56s)
FAIL
FAIL    github.com/SynologyOpenSource/synology-csi/test/sanity  0.856s
FAIL
make: *** [test] Error 1

... snip ...

The CSI service account is admin, and I had success provisioning LUNs previously. I've read that 2FA can interfere with the auth flow, so my gut tells me the CSI service account is triggering 2FA. Maybe all of the fiddling and configuration is triggering the "Adaptive MFA" feature (Control Panel > Security > Account) for the administrator group users. It seems like unchecking this may be the only option, given that the CSI service account must be an admin.

It's too bad the Synology security model doesn't have better support for scoped access to SAN Manager / LUN provisioning.

If I can confirm my theory, I'll update this comment.

[jhughes2112]

I have been working on this same problem myself. I thought the way to configure iscsi with my DSM was to enable Global CHAP and set the username/password there to be the same as the one I put in the client-info.yml, but clearly that's not what everyone else is doing. I appreciate that the csi walkthrough explains how to install the driver into k8s, but it would be even better if the walkthrough tells you exactly how to configure the user and volume in the DSM itself.

Here's what I discovered:

• Do not set up Global CHAP.
• Do not create a LUN. The driver does this automatically as needed per PVC.
• Create a new user and password that matches what is in the client-info.yml and add it to the Administrators group. You can deny it access to any other shares. I did not Allow or Deny access to any specific Applications.

And once I did that, it started working.

[dkrizic]

I am struggling make bring this all to work by myself. If you run

kubectl -n synology-csi logs synology-csi-controller-0

you only see the logs from the generic csi-provisioner. The Synology specific logs can be found here:

kubectl -n synology-csi logs synology-csi-controller-0 -c csi-plugin

[plaurin84]

I am struggling make bring this all to work by myself. If you run

kubectl -n synology-csi logs synology-csi-controller-0

you only see the logs from the generic csi-provisioner. The Synology specific logs can be found here:

kubectl -n synology-csi logs synology-csi-controller-0 -c csi-plugin

Good point for debugging. There are sometimes many containers in a single pod, so yeah, definitely a good idea.

[avanier]

Adding another data point here. I only had the default Synology cert installed and tried using https: true. Apparently the connection was silently failing TLS validation. 🤷