Update initialization scripts to install latest RAPIDS 23.12 OR 24.02

[skirui-source]

CC @jacobtomlinson

Following the instructions to deploy RAPIDS in Dataproc:

1. I was able to successfully launch a dataproc cluster with rapids v23.12 stable, cuda 11.8 (both parsed via the metadata flags --rapids-version and --cuda-version)

2. The install_gpu_driver.sh script downloads ubuntu 18.04, and has outdated versions of cuda and gpu drivers ; installs rapids v22.12 and cuda 11.2 by default; doesn't include cuda 12 as an option

3. The latest rapids 24.02 is only compatible with ubuntu 20.04 and 22.04 (with cuda 11.8 or 12.) ; Thus the install scripts need to be updated accordingly with the newer drivers as well

4. To test RAPIDS libraries in the notebook environment, we need to edit rapids.sh script to activate the conda environment (dask-rapids) and register it as a kernel in Jupyter Lab/ notebook

5. For now the users will have to manually conda activate and register the dask-rapids kernel from the terminal in Jupyter.

6. Alternatively, users can use the dataproc:conda.env.config.uri, which is absolute path to a Conda environment YAML config file located in Cloud Storage. This file will be used to create and activate a new Conda environment on the cluster. But this option is redundant because you first have to export the conda env into a .yaml file

[cjac]

Hello team, I'm unable to install cudatoolkit > 11.8 since it has not been packaged with mamba yet

I've made some changes in #1199 to address some of these requests.

[cjac]

we need to edit rapids.sh script to activate the conda environment (dask-rapids) and register it as a kernel in Jupyter Lab/ notebook

Can you tell me how you would like the script activated, specifically?

• What do you mean by "activate the conda environment"?
• What is required to register "it" as a kernel in Jupyter Lab/ notebook? What exactly is "it"?