v1.41.0 Adoption of Slurm 24.05 and Improvements to GKE Support

[harshthakkar01]

What's Changed

Key New Features 🎉

• Update Slurm 23.11.8 -> 24.05.3 by @mr0re1 in Update Slurm 23.11.8 -> 24.05.3 #3054

New Modules 🧱

• resource-policy module implemented by @sharabiani in resource-policy module implemented #3066
• gke-topology-scheduler module implemented by @sharabiani in gke-topology-scheduler module implemented #3080
• add GKE support for parallelstore through gke-storage module by @chengcongdu in add GKE support for parallelstore through gke-storage module #3120

Module Improvements 🔨

• Added compatibility check for GPUDirect and GKE version by @sharabiani in Added compatibility check for GPUDirect and GKE version #3079
• Support template file for kueue configuration in kubectl-apply module by @sharabiani in Support template file for kueue configuration in kubectl-apply module #3111
• Implement xpk-gke-a3-megagpu blueprint by @sharabiani in Implement xpk-gke-a3-megagpu blueprint #3108
• Use sackd for the login nodes by @mr0re1 in Use sackd for the login nodes #3126
• gke-node-pool default name conflict fixed by @sharabiani in gke-node-pool default name conflict fixed #3127
• improve dws_flex ux by @abbas1902 in improve dws_flex ux #3122
• Include deployment name in Spack and Ramble bucket names (like startup-script) by @rohitramu in Include deployment name in Spack and Ramble bucket names (like startup-script) #3136

Improvements 🛠

• Create and use non-default service accounts in GKE by @annuay-google in Create and use non-default service accounts in GKE #3123
• Added documentation on cloud-ops-agent installation and stackdriver removal by @jrossthomson in Added documentation on cloud-ops-agent installation and stackdriver removal #3029
• Ensure local SSD filesystem is assembled into a RAID even upon power off/on cycles by @tpdownes in Ensure local SSD filesystem is assembled into a RAID even upon power off/on cycles #3129

Deprecations 💤

• Freeze slurm-gcp v5 hybrid blueprints with the latest cluster toolkit version support by @harshthakkar01 in Freeze slurm-gcp v5 hybrid blueprints with the latest cluster toolkit version support #3117
• Update Slurm-gcp v5 deprecation details by @harshthakkar01 in Update Slurm-gcp v5 deprecation details #3118
• Update badge for slurm-gcp v5 and slurm-gcp v6 by @harshthakkar01 in Update badge for slurm-gcp v5 and slurm-gcp v6 #3116

Version Updates ⏫

• Update A3-High NeMo to 24.07 and NCCL solution to latest recommended values by @akiki-liang0 in Update A3-High NeMo to 24.07 and NCCL solution to latest recommended values #3130
• Update Slurm-GCP to 6.8.2 by @tpdownes in Update Slurm-GCP to 6.8.2 #3132

Bug fixes 🐞

• Fixed the exact number constraint problem for additional vpcs in gpu_direct checks by @sharabiani in Fixed the exact number constraint problem for additional vpcs in gpu_direct checks #3078
• Provide explicit project information by @wiktorn in Provide explicit project information #3060
• Chrome Remote Desktop: increase resilience of apt operations by @tpdownes in Chrome Remote Desktop: increase resilience of apt operations #3093
• Add mount parallelstore service to mount parallelstore for every reboot by @harshthakkar01 in Add mount parallelstore service to mount parallelstore for every reboot #3125

New Contributors

• @akiki-liang0 made their first contribution in Update A3-High NeMo to 24.07 and NCCL solution to latest recommended values #3130
• @ighosh98 made their first contribution in GKE A3 high integration test #3124

Full Changelog: v1.40.1...v1.41.0

---

This discussion was created from the release v1.41.0 Adoption of Slurm 24.05 and Improvements to GKE Support.