Setup workload identity via terraform

[yuvipanda]

The GKE config connector was helpful in letting us deploy
Google Cloud Service Accounts with permissions for cloud storage
directly just from helm. However, it has been difficult to debug,
and in #669
we decided to move away from it and towards creating these
cloud resources via Terraform.

This commit adds:

• Terraform code that will create a Google Service Account,
  bind it to a given Kubernetes Service Account, for a list of
  hub namespaces passed in. This means that some hub initial deployments
  now can not be done just with CD, but need manual work with
  terraform. I think this would be any hub that wants to use
  requestor pays or scratch buckets. This would need to be
  documented.
• Move meom-ige to use this new scheme. metadata concealment
  (https://cloud.google.com/kubernetes-engine/docs/how-to/protecting-cluster-metadata#concealment)
  which is what we were using earlier as alternative to config-connector
  • workload identity, is no longer supported by the terraform
    google provider. In b7b42ce,
    we changed the default from 'SECURE' to 'UNSPECIFIED', but
    it looks like 'UNSPECIFIED' really means 'use workload identity'
    haha. When Switch to using pd-balanced for all user & dask nodes #1124 was
    deployed to meom-ige yesterday, it seems to have enabled workload
    identity, causing cloud access to stop working, leading to
    https://2i2c.freshdesk.com/a/tickets/107. Further investigation on
    what happened here is needed, but I've currently fixed it by
    just deploying this change for meom-ige.
• Documentation is provided for each 'feature', and how 2i2c engineers can
  enable these on a per-hub basis. They are designed to be 'feature flags'
  in terraform wherever possible. This still involves copying some terraform
  outputs to helm values files unfortunately, as we use helm to create namespaces
  (and terraform doesn't have a way to say 'create this namespace if required',
  unlike eksctl)
• Add a 'features/' subdirectory to documentation, that lists specifically just
  hub features, with a corresponding entry in the topic page. Will slowly move
  other documentation about features to this structure.
• Rename current cloud-access.md document to clarify it's about auth
  for 2i2c engineers, not about accessing cloud features from the hub
• Remove scratch bucket from the 3 hubs that have it on the 2i2c cluster -
  dask-staging, ohw and catalyst-cooperative. None of these actually have
  used their bucket. requestor_pays is enabled for all of them.

TODO:

• Separate requestor pays & scratch bucket access into feature flags
• Document how to create new hubs that have either of these features
• Figure out how to deal with creating namespace for the secret to live in.
  Perhaps we can move the Kubernetes secret creation away from terraform
  and into helm to simplify this?
• Apply this to existing pangeo-hubs
• Apply this to hubs that use this right now in 2i2c cluster

Ref #669
Ref #1046

[yuvipanda]

This was caught by @sgibson91 in #1074, but then https://2i2c.freshdesk.com/a/tickets/107 happened today as well so I had to fix that. Automating this seemed the best way to go forward.

[yuvipanda]

Looking at the cloud buckets in https://console.cloud.google.com/storage/browser?forceOnBucketsSortingFiltering=false&project=two-eye-two-see&prefix=&forceOnObjectsSortingFiltering=false, none of them actually have any content in them! So nobody is actually using scratch buckes in the default pilot-hubs cluster.

[sgibson91]

Thank you for your work following this up @yuvipanda! 🙌

[GeorgianaElena]

I really love the level of documentation in this PR and I'm very grateful for understanding what's happening here with little knowledge of the terraform syntax. 🌼 Thank you, Yuvi!

[sgibson91]

I've left a few comments/typo fixes, but this looks good to me! 🚀

[sgibson91]

Should this default to true? I'm not sure if it's better to explicitly enable features or rely on default config

[yuvipanda]

I left a comment about that - but the short version is, we always have a service account attached to pods, regardless of wether they have any permissions assigned or not. I think if we don't, the default service account gets attached.

[yuvipanda]

pangeo-hubs I've preserved current access, as deploying this there would require #1153 to be done too.