Fine grained permissions per user

[casassg]

I wanted to ask wether there's any way to implement a finer grained set of permissions based on os_login for the Slurm VMs. In a large cluster setting, it would be interesting to ensure each user that logins can act as a different service account ensuring only those allowed can access certain data from GCS for example. Something like Workload identity in GKE.

Would be interesting to have some example setup to do this! Or just in general how to setup the cluster such that it's secure w lots of users.

Also would be nice to have some documentation to lock down root on installation (I assume just writing to sudoers as part of the install scripts could work but still would be nice)

[rohitramu]

@casassg thank you for the post!

Currently, we don't have a good solution integrated for fine-grained permissions per user. Regarding root access - are you trying to prevent certain users from being able to run sudo? I believe users with the following roles on the GCP project will be able to sudo:

• roles/owner
• roles/admin
• roles/compute.osAdminLogin
• roles/compute.instanceAdmin

More info on roles: https://cloud.google.com/compute/docs/access/iam#compute.osAdminLogin

[rohitramu]

To be a bit more specific, the above applies to the nodes that users SSH into directly (e.g. login or controller nodes). A job will not have sudo access even if the user who submitted it is assigned the roles/compute.osAdminLogin role.

Regarding finer grained permissions - see examples/hpc-enterprise-slurm.yaml for an example of how to create and assign a service account to the login and compute nodes. I would recommend reducing the scope of the storage permissions for compute nodes (e.g. replace storage.objectCreator with storage.objectViewer) and then have users run the following commands on the login node before submitting job:

gcloud auth application-default login
gcloud auth application-default set-quota-project ${PROJECT_ID}

Make sure to replace ${PROJECT_ID} with the ID of the project where the Slurm VMs are deployed.

These commands will ensure that their credentials (and therefore permissions) are used when their jobs make API calls (e.g. reading from or writing to shared storage) rather than the assigned service account's permissions specified in the blueprint. The cluster must have a shared /home directory in order for this to work (as in the example linked above).

We have not previously tested this exact workflow, but I hope it is helpful for your use case!

[casassg]

yeah, we have been able to avoid giving sudo (using only osLogin vs osAdminLogin), but would been nice to do something like workload-identity in GKE where each user can automatically assume the identity of a service account. Wasn't able to find it in GCE docs (we found https://cloud.google.com/iam/docs/managed-workload-identity but had a hard time making sense of it).

Also ty for the extensive answer here :D

[nick-stroud]

Warning: I have not tested the following approach but am just brainstorming.

I think the gcloud auth application-default login approach that @rohitramu suggests above would result in jobs using user credentials. If instead you wanted to use individual service account credentials (as you suggest), then each user could set up a service account key and store it in their home folder. Then as part of the job, set GOOGLE_APPLICATION_CREDENTIALS to point to the key file. This would give the job the credentials of that specific service account. ref

I realize that this is extra setup on end users. We don't currently have something like workload-identity automated for Slurm.