Explore options for providing users with on-demand resources

[kabilar]

Description

We utilize AWS spot instances to minimize costs while providing the JupyterHub as a service to all users. Users may end up being kicked off when the server is no longer available, resulting in a poor user experience. Perhaps we can explore ways to provide select users (e.g. during training workshops) with on-demand instances.

Next steps

1. We should define and document under what circumstances that on-demand instances are made available to users.
2. Determine a strategy for implementing access to on-demand vs spot instances.

cc @bendichter @asmacdo

[ovalerio]

[asmacdo edit] Moved to new issue:

Click to expand

H ello @kabilar,

Comments on the issue: "JupyterHub intermittently freezing and asking to be reboot"

Thanks for following up on the JupyterHub issues.

As you might know, last week, during the DANDI ReHack at Janelia we were using both the dandi-hub and the hub-staging for running JupyterLab Notebooks. In my case I was using custom conda environments.

The issue could not easily be pinpointed to inactivity, since the disconnection sometimes occurred while the Notebooks Jupyter Kernels were busy doing computation. There was an error message that suggest to restart the notebook. That was the recommendation and if we follow that recommendation then the currently installed packages on the conda environment were lost.

Sometimes if I opt to dismiss the message the session could eventually recover, but with a degraded performance.

To answer your questions:

Did they see any error messages?

Yes, there was a pop-up message in JupyterHub with the option to restart the session and/or dismiss.

Were they actively using the server when kicked off? We have a culler for idle machines, I think its set to 60 minutes.

Sometimes it happen when actively in use, sometimes I was also making use of the instance using the JupyterHub Extension in VSCode. I post a separate feature request on that subject last week, but I think there I forgot to mention that I was already using the JupyterHub Extension to connect to my allocated instance, but using notebooks that I was creating locally.

What time were they kicked, I can check to see if theres anything in the logs

After I noticed that every time the session restarted I was losing my python environment I opt to create my environment using a persistent location (/home/jovyan) and that worked.

Example:

  conda init
  source /opt/conda/bin/activate
  mkdir /home/jovyan/envs
  conda create --prefix /home/jovyan/envs/pynapple python=3.8
  conda activate /home/jovyan/envs/pynapple
  conda install -c conda-forge ipykernel
  python -m ipykernel install --user --name pynapple
  python -m pip install pynapple