Can't mount CSI volume in unprivileged container

[p1u3o]

I'm having an issue where when attempting to use a volume_mount and volume stanza as follows below, the volume can not be mounted.

If I add privileged = true to the container config, the mount works, but I don't see this as wise in production.

rpc error: code = Unknown desc = failed to start task, could not start container: cannot start container, status code: 500: {"cause":"operation not supported","message":"lsetxattr /opt/nomad/data/client/csi/node/juicefs0/per-alloc/34725522-7bb4-8d4a-6f06-0d7646902b75/mxp-swift/rw-file-system-multi-node-multi-writer: operation not supported","response":500}

volume "cache-volume" {
      type            = "csi"
      source          = "mxp-swift"
      read_only = false
      attachment_mode = "file-system"
      access_mode     = "multi-node-multi-writer"
    }

  volume_mount {
    volume      = "cache-volume"
    destination = "/data/job"
  }

[jdoss]

What distro are you running this on and is SELinux set to enforcing?

[p1u3o]

$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.1 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"

$ getenforce
Permissive

$ podman version
Client:       Podman Engine
Version:      4.5.0-dev
API Version:  4.5.0-dev
Go Version:   go1.18.9
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

I am not sure if relevant, the the CSI Driver I am using is JuiceFS with extended attributes turned off.

[jdoss]

What is the nomad-driver-podman version? v0.4.2 addresses some of the issues you are seeing which might help but the last time I tried to use JuiceFS via the CSI driver, I always had to run the job as privileged to get the mounts to work correctly.

[p1u3o]

@jdoss I am using 0.4.2 specifically because it worked with the JuiceFS driver

I suppose a work around is to mount the volume in a sidecar in the alloc directory, but I worry about unforseen issues with doing something like that.

I also tried to manually edit the Podman seccomp policy to allow the unallowed function call, but it failed. Perhaps it does the function call only when not running privileged?

I will try running the JuiceFS mount with extended attributes enabled and see if that helps.

Edit: Nope, no difference.

[lgfa29]

Hi @p1u3o 👋

Apologies for the delay here, somehow this issue fell through the cracks of my GitHub notifications.

Which task driver are you using to run the CSI plugin? Would you be able to share that job with us?