Deploy a BinderHub on AWS and add it to the federation

[choldgraf]

In jupyterhub/team-compass#642 we discussed that we would like to deploy a BinderHub on AWS so that we can start to bring in credits resources for the federation via that route. This is an issue to track that effort.

Technical steps

I believe that @manics and @sgibson91 may have extra context about what is technically possible here and what we need to do. Can you two flesh out the to-dos here?

1. Set up the BinderHub on AWS
2. Connect with funds
3. Connect to the federation
4. Ensure others on the team have access to maintain
5. Document this hub for others to learn from

Funding steps

We need to find credits for AWS that will let us run the infrastructure. There were a few leads mentioned before:

1. @scottyhq may have credits for a few months of service. The credits are now gone https://discourse.pangeo.io/t/aws-pangeo-jupyterhubs-to-shut-down-friday-march-17/3228
2. @cgentemann mentioned that the NASA TOPS program may allow us to apply for credits in the coming months.
3. Explore CurveNote AWS credits for mybinder.org #2629
4. @rabernat may have funds that could support Binder via a grant for Pangeo, but this may require us deploying the AWS cluster in a specific way
   • xref: New federation member on AWS backed by Pangeo #2449

We'll then need to connect those credits to the deployed infrastructure so that it can start running with them.

[sgibson91]

#2467 was my attempt. I last visited when setting up the auto-scaler (which is another thing that doesn't come out of the box with EKS clusters). It worked, but it spun up more nodes than the maximum desired nodes I'd configured via terraform and I was confused as to whether that was intentional.

I have access to Scott's AWS account which is where I was testing these things.
Update: Pangeo have actually pulled down their AWS deployments, announced in this Discourse discourse.pangeo.io/t/aws-pangeo-jupyterhubs-to-shut-down-friday-march-17/3228 Stating the reason "Credits have depleted" - so potentially Scott's account is no longer the best place to be testing/host the new AWS deployment

[minrk]

If it's a blocker, would it help to deploy initially with a fixed size and capacity to get things started, then figure out the autoscaler later? OVH 1 always had a fixed size.

[sgibson91]

@minrk Maybe blocker isn't quite the right word. The autoscaler works, just while autoscaling it exceeds the maximum number of nodes I set. And that concerns me because I'm not sure if that's okay, or if it's a misconfiguration. I also don't have the capacity to work it out right now.

[sgibson91]

I have updated the top comment to reflect this. But the credits Scott had are now gone https://discourse.pangeo.io/t/aws-pangeo-jupyterhubs-to-shut-down-friday-march-17/3228

[manics]

I think the technical steps are:

Initial test deployment

1. Admin access to an AWS account that at least two people have access to
2. Deploy a plain EKS cluster using Terraform (auto-scaling optional, but for an MVP a fixed sized cluster is fine)
3. Decide on the BinderHub setup: do we go with
   • an external container registry such as quay.io
   • or ECR with IRSA. I've got this working with some BinderHub changes, code not yet merged Support dynamic registry tokens, add wrapper for working with registry microservice binderhub#1637 (comment), but I'm happy to finish off that work
4. Deploy BinderHub

Production deployment

1. Assuming we'll use the same AWS account....
2. Create a deployment OIDC IAM role in the AWS account which avoids having to store secret credentials to access AWS https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-cloud-providers
3. Setup a private encrypted S3 bucket for storing the Terraform state
4. Deploy EKS (with IRSA and ECR configuration if we're going that way) with Terraform in a GitHub workflow using the OIDC role
5. Setup the automated BinderHub deployment, again we can use https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-cloud-providers to avoid hardcoding secrets. For EKS you authenticate the GitHub workflow with OIDC, then request a kubeconfig using the AWS CLI https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html

[choldgraf]

Might have a source of credits

I have a colleague with a connection at AWS that might be able to help us with credits. Not sure if it'll come through or not but just FYI. One thing I'm not certain of is whether this would need to go to 2i2c or if it could be used for the AWS deployment here, but I will update as I learn more.

I suspect this would play out over the next few months though, so might be some time.

[manics]

Closing, follow #2629 instead