Add LEAP hub #1074

yuvipanda · 2022-03-09T10:18:48Z

sgibson91

One typo to fix and one non-blocking comment. Otherwise, LGTM!

sgibson91 · 2022-03-09T10:28:48Z

config/clusters/leap/cluster.yaml

@@ -0,0 +1,29 @@
+name: pangeo-hubs


Suggested change

name: pangeo-hubs

name: leap

I think this should be leap, right?

Ah yes! Fixed.

sgibson91 · 2022-03-09T10:31:57Z

config/clusters/leap/enc-prod.secret.values.yaml

+                GitHubOAuthenticator:
+                    client_id: ENC[AES256_GCM,data:auopWrLSGIBDtQ4PrZZMYe5XFWM=,iv:+tzBIkE6R3PfJm7oYyJOq84yyD6tB3GXeQ++sYPU7S8=,tag:vQrhczBhRRaFoqqwRWeGHg==,type:str]
+                    client_secret: ENC[AES256_GCM,data:xLL5GJTKSnucssmIQjVhCUwwXyZaYl54/+QzXPFx0dJpX63kaeJufw==,iv:2cyHZvDaoNQtlKiPKf2ACoNuvlww6WE7vcGG6jVXISI=,tag:IRcogW5ZDZWA6Pv8DyVcPg==,type:str]
+                    oauth_callback_url: ENC[AES256_GCM,data:d+/oCcmELV7Tvfe86P4YH8DCnLHI0yid0WoUWH0IKT022b9Ba/rnptYp,iv:SVHQK5yK26JOHV6uWycsLYUk42g6Kl8RahOf5oMbqxc=,tag:HjT+HYo30qqP5yCKcgVZCQ==,type:str]


I'm in two minds about having oauth_callback_url here. I know it makes the number of files you need to pass around nicer, but this value doesn't need to be encrypted really. And I did lose some time thinking that the uwhackweek hubs were misconfigured because of this pattern before.

No specific action to take in this PR, but I'd like to discuss as a team some guidelines around this for readability's sake.

@sgibson91 yeah I was thinking about that too. While keeping the number of files down helps, I think keeping it separate works alright for me too. But in that case I want to split it into three files - common, staging and prod, rather than just staging and prod. How does that sound?

Sounds reasonable. The deployer will work regardless, so long as they're listed in the cluster.yaml file! 🎉

@sgibson91 i've split this now into two non-secret files staging.yaml + prod.yaml.

yuvipanda · 2022-03-15T07:41:59Z

@sgibson91 can you take another look with the changes I've made? It's all deployed now.

sgibson91 · 2022-03-15T09:43:43Z

terraform/gcp/projects/leap.tfvars

+filestore_capacity_gb = 1024
+
+user_buckets = [
+  "pangeo-scratch"


Didn't we establish elsewhere that this will create a bucket called Pangeo scratch, but JupyterHub is configured to look for a bucket called <cluster_name>-<hub_name> and so this bucket will never get connected to the hub?

See slack message: https://2i2c.slack.com/archives/C028WU9PFBN/p1637755725001500?thread_ts=1637754043.001100&cid=C028WU9PFBN

And what JupyterHub is expecting:

infrastructure/hub-templates/basehub/values.yaml

Line 358 in 0e6006b

bucket_name = f'{project_id}-{release}-scratch-bucket'

@sgibson91 ah good catch. see d83461a.

sgibson91

LGTM! I have one question about the scratch buckets, but the rest is great.

yuvipanda · 2022-03-15T13:07:36Z

@sgibson91 aha, looks like scratch bucket access doesn't actually work. Boo. I'll investigate.

sgibson91 · 2022-03-15T13:33:12Z

@yuvipanda this issue may be relevant #1046

terraform/gcp/projects/leap.tfvars

Ref 2i2c-org#1050

Assign one user per node, using https://learnk8s.io/kubernetes-instance-calculator to calculate how big to make guarantees so pods stick one to a node. This provides a reasonable tradeoff for research use cases I think, although it should still be discussed.

Quite useful!

We aren't using config-connector anymore, so these need to be set explicitly.

- GCS storage protocol is gs, not gcs - Just hardcode the bucket name here, as env var substitution relies on ordering of env vars and that is just a bit messy.

null is not an accepted value, but 'default' is

yuvipanda · 2022-03-24T22:16:07Z

Thanks for the review, @damianavila and @sgibson91! This also doesn't have fully functional scratch buckets, something that'll be fixed in #1130.

yuvipanda · 2022-03-24T22:20:55Z

I also ran a terraform apply, and verified it is as clean as it can be. There's still changes that are see-sawing (with no actual effects) that'll be fixed in #1130.

google_artifact_registry_repository.registry: Refreshing state... [id=projects/leap-pangeo/locations/us-central1/repositories/leap-registry]
google_service_account.cluster_sa: Refreshing state... [id=projects/leap-pangeo/serviceAccounts/[email protected]]
google_service_account.cd_sa: Refreshing state... [id=projects/leap-pangeo/serviceAccounts/[email protected]]
google_project_iam_custom_role.identify_project_role: Refreshing state... [id=projects/leap-pangeo/roles/leap_user_sa_role]
google_storage_bucket.user_buckets["pangeo-scratch"]: Refreshing state... [id=leap-pangeo-scratch]
google_project_iam_member.cd_sa_roles["roles/container.admin"]: Refreshing state... [id=leap-pangeo/roles/container.admin/serviceAccount:[email protected]]
google_project_iam_member.cd_sa_roles["roles/artifactregistry.writer"]: Refreshing state... [id=leap-pangeo/roles/artifactregistry.writer/serviceAccount:[email protected]]
google_project_iam_member.cluster_sa_roles["roles/monitoring.metricWriter"]: Refreshing state... [id=leap-pangeo/roles/monitoring.metricWriter/serviceAccount:[email protected]]
google_project_iam_member.cluster_sa_roles["roles/logging.logWriter"]: Refreshing state... [id=leap-pangeo/roles/logging.logWriter/serviceAccount:[email protected]]
google_service_account_key.cd_sa: Refreshing state... [id=projects/leap-pangeo/serviceAccounts/[email protected]/keys/8192f1f77328895d53a3c3c2cb53cefa78cb4cc5]
google_project_iam_member.cluster_sa_roles["roles/monitoring.viewer"]: Refreshing state... [id=leap-pangeo/roles/monitoring.viewer/serviceAccount:[email protected]]
google_project_iam_member.cluster_sa_roles["roles/stackdriver.resourceMetadata.writer"]: Refreshing state... [id=leap-pangeo/roles/stackdriver.resourceMetadata.writer/serviceAccount:[email protected]]
google_project_iam_member.cluster_sa_roles["roles/artifactregistry.reader"]: Refreshing state... [id=leap-pangeo/roles/artifactregistry.reader/serviceAccount:[email protected]]
google_storage_bucket_iam_member.member["pangeo-scratch"]: Refreshing state... [id=b/leap-pangeo-scratch/roles/storage.admin/serviceAccount:[email protected]]
google_filestore_instance.homedirs[0]: Refreshing state... [id=projects/leap-pangeo/locations/us-central1-b/instances/leap-homedirs]
google_container_cluster.cluster: Refreshing state... [id=projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster]
google_project_iam_member.identify_project_binding: Refreshing state... [id=leap-pangeo/projects/leap-pangeo/roles/leap_user_sa_role/serviceAccount:[email protected]]
google_container_node_pool.dask_worker["huge"]: Refreshing state... [id=projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/dask-huge]
google_container_node_pool.notebook["large"]: Refreshing state... [id=projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/nb-large]
google_container_node_pool.dask_worker["medium"]: Refreshing state... [id=projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/dask-medium]
google_container_node_pool.dask_worker["large"]: Refreshing state... [id=projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/dask-large]
google_container_node_pool.notebook["huge"]: Refreshing state... [id=projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/nb-huge]
google_container_node_pool.dask_worker["small"]: Refreshing state... [id=projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/dask-small]
google_container_node_pool.notebook["medium"]: Refreshing state... [id=projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/nb-medium]
google_container_node_pool.core: Refreshing state... [id=projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/core-pool]
google_container_node_pool.notebook["small"]: Refreshing state... [id=projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/nb-small]
Note: Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the

last "terraform apply":
google_container_cluster.cluster has changed
~ resource "google_container_cluster" "cluster" {

id                          = "projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster"

name                        = "leap-cluster"

# (26 unchanged attributes hidden)
  ~ node_pool {
        name                        = "nb-small"
      ~ node_count                  = 0 -> 1
        # (6 unchanged attributes hidden)




        # (4 unchanged blocks hidden)
    }




    # (22 unchanged blocks hidden)
}

google_container_node_pool.notebook["small"] has changed
~ resource "google_container_node_pool" "notebook" {

id                          = "projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/nb-small"

name                        = "nb-small"

~ node_count                  = 0 -> 1

# (8 unchanged attributes hidden)
    # (4 unchanged blocks hidden)
}

Unless you have made equivalent changes to your configuration, or ignored the

relevant attributes using ignore_changes, the following plan may include

actions to undo or respond to these changes.
─────────────────────────────────────────────────────────────────────────────
Terraform used the selected providers to generate the following execution

plan. Resource actions are indicated with the following symbols:

~ update in-place
Terraform will perform the following actions:
google_container_node_pool.dask_worker["huge"] will be updated in-place
~ resource "google_container_node_pool" "dask_worker" {

id                          = "projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/dask-huge"

name                        = "dask-huge"

# (9 unchanged attributes hidden)
  ~ node_config {
        tags              = []
        # (12 unchanged attributes hidden)


      ~ workload_metadata_config {
          ~ mode = "GKE_METADATA" -> "MODE_UNSPECIFIED"
        }
        # (1 unchanged block hidden)
    }

    # (3 unchanged blocks hidden)
}

google_container_node_pool.dask_worker["large"] will be updated in-place
~ resource "google_container_node_pool" "dask_worker" {

id                          = "projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/dask-large"

name                        = "dask-large"

# (9 unchanged attributes hidden)
  ~ node_config {
        tags              = []
        # (12 unchanged attributes hidden)


      ~ workload_metadata_config {
          ~ mode = "GKE_METADATA" -> "MODE_UNSPECIFIED"
        }
        # (1 unchanged block hidden)
    }

    # (3 unchanged blocks hidden)
}

google_container_node_pool.dask_worker["medium"] will be updated in-place
~ resource "google_container_node_pool" "dask_worker" {

id                          = "projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/dask-medium"

name                        = "dask-medium"

# (9 unchanged attributes hidden)
  ~ node_config {
        tags              = []
        # (12 unchanged attributes hidden)


      ~ workload_metadata_config {
          ~ mode = "GKE_METADATA" -> "MODE_UNSPECIFIED"
        }
        # (1 unchanged block hidden)
    }

    # (3 unchanged blocks hidden)
}

google_container_node_pool.dask_worker["small"] will be updated in-place
~ resource "google_container_node_pool" "dask_worker" {

id                          = "projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/dask-small"

name                        = "dask-small"

# (9 unchanged attributes hidden)
  ~ node_config {
        tags              = []
        # (12 unchanged attributes hidden)


      ~ workload_metadata_config {
          ~ mode = "GKE_METADATA" -> "MODE_UNSPECIFIED"
        }
        # (1 unchanged block hidden)
    }

    # (3 unchanged blocks hidden)
}

google_container_node_pool.notebook["huge"] will be updated in-place
~ resource "google_container_node_pool" "notebook" {

id                          = "projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/nb-huge"

name                        = "nb-huge"

# (9 unchanged attributes hidden)
  ~ node_config {
        tags              = []
        # (12 unchanged attributes hidden)


      ~ workload_metadata_config {
          ~ mode = "GKE_METADATA" -> "MODE_UNSPECIFIED"
        }
        # (1 unchanged block hidden)
    }

    # (3 unchanged blocks hidden)
}

google_container_node_pool.notebook["large"] will be updated in-place
~ resource "google_container_node_pool" "notebook" {

id                          = "projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/nb-large"

name                        = "nb-large"

# (9 unchanged attributes hidden)
  ~ node_config {
        tags              = []
        # (12 unchanged attributes hidden)


      ~ workload_metadata_config {
          ~ mode = "GKE_METADATA" -> "MODE_UNSPECIFIED"
        }
        # (1 unchanged block hidden)
    }

    # (3 unchanged blocks hidden)
}

google_container_node_pool.notebook["medium"] will be updated in-place
~ resource "google_container_node_pool" "notebook" {

id                          = "projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/nb-medium"

name                        = "nb-medium"

# (9 unchanged attributes hidden)
  ~ node_config {
        tags              = []
        # (12 unchanged attributes hidden)


      ~ workload_metadata_config {
          ~ mode = "GKE_METADATA" -> "MODE_UNSPECIFIED"
        }
        # (1 unchanged block hidden)
    }

    # (3 unchanged blocks hidden)
}

google_container_node_pool.notebook["small"] will be updated in-place
~ resource "google_container_node_pool" "notebook" {

id                          = "projects/leap-pangeo/locations/us-central1-b/clusters/leap-cluster/nodePools/nb-small"

name                        = "nb-small"

# (9 unchanged attributes hidden)
  ~ node_config {
        tags              = []
        # (12 unchanged attributes hidden)


      ~ workload_metadata_config {
          ~ mode = "GKE_METADATA" -> "MODE_UNSPECIFIED"
        }
        # (1 unchanged block hidden)
    }

    # (3 unchanged blocks hidden)
}

Plan: 0 to add, 8 to change, 0 to destroy.
Warning: Argument is deprecated
with google_filestore_instance.homedirs,

on storage.tf line 4, in resource "google_filestore_instance" "homedirs":

4:   zone    = var.zone
Deprecated in favor of location.
(and one more similar warning elsewhere)
─────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't

guarantee to take exactly these actions if you run "terraform apply" now.

yuvipanda force-pushed the new-leap branch from 04e7f44 to 93aba4e Compare March 9, 2022 10:22

sgibson91 approved these changes Mar 9, 2022

View reviewed changes

yuvipanda mentioned this pull request Mar 10, 2022

[New Hub] LEAP Pangeo #1050

Closed

10 tasks

sgibson91 reviewed Mar 15, 2022

View reviewed changes

sgibson91 approved these changes Mar 15, 2022

View reviewed changes

sgibson91 mentioned this pull request Mar 16, 2022

Transition GCP config-connector to terraform - cleanup cloudResources #1046

Closed

4 tasks

yuvipanda mentioned this pull request Mar 18, 2022

Setup workload identity via terraform #1130

Merged

5 tasks

damianavila reviewed Mar 21, 2022

View reviewed changes

terraform/gcp/projects/leap.tfvars Outdated Show resolved Hide resolved

yuvipanda added 12 commits March 24, 2022 14:38

Add LEAP hub

3749d30

Ref 2i2c-org#1050

Fix title of cluster

d126bcc

Use leap-stc github org for restricting access

7394192

Use latest pangeo docker image

4453724

Allow named servers in leap hub

4f159cd

Quite useful!

Add leap hub terraform params

9f1aa2e

Put non-secret per-hub values into their own file

81b6e6d

Set SCRATCH_BUCKET explicitly for leap hub

89159f5

We aren't using config-connector anymore, so these need to be set explicitly.

Nest some config deeper

c218541

Fix scratch bucket env vars properly

5b11312

- GCS storage protocol is gs, not gcs - Just hardcode the bucket name here, as env var substitution relies on ordering of env vars and that is just a bit messy.

Fix comment about config_connector

0ee90ff

yuvipanda force-pushed the new-leap branch from 95e1b17 to 0ee90ff Compare March 24, 2022 21:42

yuvipanda added 2 commits March 24, 2022 15:04

Remove tabs from encrypted JFON file

62ed822

Re-add config that sets filestore network conditionally

21e50e9

null is not an accepted value, but 'default' is

yuvipanda merged commit 70b07c5 into 2i2c-org:master Mar 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add LEAP hub #1074

Add LEAP hub #1074

yuvipanda commented Mar 9, 2022

sgibson91 left a comment

sgibson91 Mar 9, 2022

yuvipanda Mar 9, 2022

sgibson91 Mar 9, 2022

yuvipanda Mar 9, 2022

sgibson91 Mar 9, 2022

yuvipanda Mar 15, 2022

yuvipanda commented Mar 15, 2022

sgibson91 Mar 15, 2022

sgibson91 Mar 15, 2022

yuvipanda Mar 15, 2022

sgibson91 Mar 15, 2022

sgibson91 left a comment

yuvipanda commented Mar 15, 2022

sgibson91 commented Mar 15, 2022

yuvipanda commented Mar 24, 2022

yuvipanda commented Mar 24, 2022

google_container_cluster.cluster has changed

google_container_node_pool.notebook["small"] has changed

google_container_node_pool.dask_worker["huge"] will be updated in-place

google_container_node_pool.dask_worker["large"] will be updated in-place

google_container_node_pool.dask_worker["medium"] will be updated in-place

google_container_node_pool.dask_worker["small"] will be updated in-place

google_container_node_pool.notebook["huge"] will be updated in-place

google_container_node_pool.notebook["large"] will be updated in-place

google_container_node_pool.notebook["medium"] will be updated in-place

google_container_node_pool.notebook["small"] will be updated in-place

Add LEAP hub #1074

Add LEAP hub #1074

Conversation

yuvipanda commented Mar 9, 2022

sgibson91 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuvipanda commented Mar 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgibson91 left a comment

Choose a reason for hiding this comment

yuvipanda commented Mar 15, 2022

sgibson91 commented Mar 15, 2022

yuvipanda commented Mar 24, 2022

yuvipanda commented Mar 24, 2022

google_container_cluster.cluster has changed

google_container_node_pool.notebook["small"] has changed

google_container_node_pool.dask_worker["huge"] will be updated in-place

google_container_node_pool.dask_worker["large"] will be updated in-place

google_container_node_pool.dask_worker["medium"] will be updated in-place

google_container_node_pool.dask_worker["small"] will be updated in-place

google_container_node_pool.notebook["huge"] will be updated in-place

google_container_node_pool.notebook["large"] will be updated in-place

google_container_node_pool.notebook["medium"] will be updated in-place

google_container_node_pool.notebook["small"] will be updated in-place