[New Hub] M2LiNES Pangeo hub

[choldgraf]

Hub Description

M2LINES is an international climate modeling collaboration. They'd like a Pangeo-style hub for their community.

• Leads issue: https://github.com/2i2c-org/leads/issues/64

Community Representative(s)

Our contact right now is:

• @rabernat

TODO: is there a better person at NYU to be the community rep?

Important dates

• Required start date: TODO: is there a specific date or just ASAP?
• Target start date:
• Any important dates for usage:

Hub Authentication Type

GitHub Authentication (e.g., @MyGitHubHandle)

Hub logo information

• URL to Hub Image: https://twitter.com/M2LInES/photo
• URL for Image Link: https://m2lines.github.io/

Hub user image

• Repository for user image: https://github.com/pangeo-data/pangeo-docker-images (I believe they wish to use the Pangeo images)
• User image registry: TODO: figure out which registry Pangeo uses
• User image tag and name: TODO: figure out which image should be used (or maybe juts latest)

Extra features you'd like to enable

• Specific cloud provider or datacenter: Same as other Pangeo hubs
• Dedicated Kubernetes cluster
• Scalable Dask Cluster

Other relevant information

This hub should be similar to the Pangeo and LEAP hubs.

TODO: confirm that github authentication is preferred

Hub URL

m2lines.2i2c.cloud TODO: confirm this is OK w/ community rep

Hub Type

daskhub

Tasks to deploy the hub

• Engineer who will deploy the hub is assigned
• Deploy information filled in above
• Initial Hub deployment PR: Add m2lines hub #1227
• Administrators able to log on
• Community Representative satisfied with hub environment
• Hub now in steady-state

[choldgraf]

cc @rabernat - there are a few "TODO" questions up there that you might be able to help answering (or feel free to loop in any other person that would be able to help clarify).

[yuvipanda]

Does this get its own cloud project with its own billing for cloud resource use, or is this being rolled into the cloud resource / billing for pangeo-hubs?

I'll pick this one up.

[rabernat]

TODO: is there a better person at NYU to be the community rep?

@johannag126 may be the best choice here because she is in contact with all the project participants on a regular basis. Johanna, would you be comfortable with the Community Representative Responsibilities? If you have questions about this, feel free to ask them here in this thread.

• TODO: is there a specific date or just ASAP?

Asap without any specific urgency.

• TODO: figure out which registry Pangeo uses

We are still using dockerhub. If you want to help move pangeo-docker-images to quay, PR welcome 😉 .

• TODO: figure out which image should be used (or maybe juts latest)

M2LInES needs here are mostly identical to LEAP. Ideally we would be able to select from all the pangeo-docker-image tags via a customized spawner, as discussed a bit in #1050 (comment). We also need optional GPUs.

TODO: confirm that github authentication is preferred

Yes. The roles are simpler than with LEAP. Let's just allow anyone part of the m2lines org (https://github.com/orgs/m2lines) to access the hub.

m2lines.2i2c.cloud TODO: confirm this is OK w/ community rep

👍

[johannag126]

TODO: is there a better person at NYU to be the community rep?

@johannag126 may be the best choice here because she is in contact with all the project participants on a regular basis. Johanna, would you be comfortable with the Community Representative Responsibilities? If you have questions about this, feel free to ask them here in this thread.

I am happy to be the relay between the team and hub engineer but would this role require technical knowledge ? The description reads "This role is usually filled by someone that is a member of the hub’s community of practice."

[colliand]

Yes @yuvipanda this hub should be set up on a dedicated GCP cluster with the billing account managed by 2i2c/CS&S. Cloud costs will be passed on to NYU/M2LInES by our colleagues at CS&S.

[yuvipanda]

@colliand great. I'll take this on and try to get it done by next week.

[choldgraf]

Hey all - what is the status on this hub? We have now set up the invoicing for it, so it should be running ASAP.

[yuvipanda]

@choldgraf I'll get this done this week.

[yuvipanda]

@johannag126 @rabernat this is now up at https://m2lines.2i2c.cloud! Please check it out.

Features:

• GitHub Auth, anyone part of https://github.com/m2lines org can log in
• Dask-gateway is enabled
• Latest pangeo docker image is used
• gh-scoped-creds (https://github.com/yuvipanda/gh-scoped-creds/) is
  enabled for secure pushing to GitHub
• Scratch GCS storage bucket is available, accessed via the
  SCRATCH_BUCKET environment variable

[yuvipanda]

I can't test if folks part of the https://github.com/m2lines org can log in - can you test logging in, @johannag126?

[johannag126]

@yuvipanda thank you! I was able to log in with no issue

[yuvipanda]

@johannag126 great! I've now merged this and I think the hub is ready to go!

@rabernat @johannag126 can you speak more about the GPU requirement? we can open another issue to discuss that.

[rabernat]

See #1237

[damianavila]

Since we have a follow-up for GPU support, I think we can close this one now.

[rabernat]

Sorry to revive this old issue, but there is one final item we need to resolve before launching the LEAP and M2LInES hubs: the ability to customize the user image independently from the hardware profile. This is being discussed in:

• matrix of profile_list options jupyterhub/kubespawner#307
• Support dropdown list choices for profile_list profiles via profile_options jupyterhub/kubespawner#607

Specifically, in jupyterhub/kubespawner#607, Yuvi has implemented the ability to have a dropdown menu of possible images for each profile. I want to state clearly that that is an acceptable solution to this issue and I would prefer to move forward with that, rather than iterating further on the design (as I had suggested earlier in jupyterhub/kubespawner#607 (comment)).

In particular, we need to make sure that GPU users can choose from TWO different possible images:

• "Tensorflow Notebook": https://hub.docker.com/r/pangeo/ml-notebook
• "PyTorch Notebook": https://hub.docker.com/r/pangeo/pytorch-notebook

Given the velocity of change of these images, I do not think it is ever wise to use latest tags. We should always be pinning a specific version (this goes for pangeo-notebook as well). Ideally we would populate the list of available images with all of the possible recent versions, so that users can go back and forth between versions. We want users to become aware of which image and version they are using by exposing this clearly though our UI.

[yuvipanda]

@rabernat to not block m2lines usage on that PR merging, I've just added an extra profile with pytorch here: #1267. I've also pinned the images. We can consolidate once that PR lands.