Automate or Script deployment of functions

[ChristopherMancuso]

It would be good to have some scripts that can generate the needed files/scripts for deploying functions for the various hyperparameters

• Setting up the data needed by each function
• Writing the main.py and requirements.txt files
• deploy using gcloud functions deploy command

[falquaddoomi]

Storage pricing:

• After revieiwng a few storage options, Google Cloud Storage (aka GCS) is cheaper than using GitHub LFS
  • we pay ~$0.02/GB/mo for the standard storage class, and storage is free below 5GB. Our current usage of ~2.6GB falls below this, so we're not paying for storage yet. (source: https://cloud.google.com/storage/pricing/#cloud-storage-pricing)
  • network egress from North America (where the GCS bucket is stored) to worldwide locations cost $0.12/GB below 1TB, then becomes cheaper afterward. To deploy all our functions (again, ~2.6GB), it'd cost ~$0.31 per deployment.

GenePlexus storage notes:

• the data is currently stored in a private GCS bucket at gs://geneplexus-func-data, with subdirectories for each function
  • the data is gzipped and when extracted creates a folder called "data" containing the contents of the gzip

GH Action Caching:

• GitHub provides a free cache for actions of up to 10GB that's retained for 7 days. This means that we can pay $0.31 once within a seven-day window for as many deployments as we like (as long as the data isn't changing between deployments, of course). (source: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy)
• when the 10GB limit is reached, GitHub starts evicting cache entries. I assume they would evict them in a sensible order, e.g. by least-frequently-used or first-in-first-out, but they don't specify in their docs.

[falquaddoomi]

Regarding progress, so far I've tested the data retrieval from the bucket (which requires authenticating to GCP using a service account key), extracting the archive, caching, and verifying that the filesystem with the extracted data looks correct. I've been using act to do the action testing, which is surprisingly actually working for once, including emulating a local GH action cache.

I haven't tested actually deploying the functions because I didn't want to mess up the current deployment. @ChristopherMancuso, if you're ok with me trying and potentially having to revert to a previous deployment if things go awry, I can test that, too. FYI, I've been testing with the convert-ids function, since its data requirement is tiny, ~20mb.

[ChristopherMancuso]

This all sounds great. I feel like you can mess with the current GCP deployment as much as you want since you set the repo up to use functions-framework, I don't think Vince is actually using the GCP functions, just the local version, right?