FE-350 update manifest.py to remove managed access project ids from final partition set

[bahill]

Why

FE-350

This PR:

Updates the manifest.py script which is used to set up the partition sets for each of the four steps of the HCA ingest workflow.

All project ids - IE all datasets - should be validated, loaded into TDR and have a snapshot created. Only public access projects should then go through the final step of having their snapshot made public. This update removes the need for the human running these steps to manually exclude managed access project ids from that final step, and thus helps avoid accidentally making managed(controlled) access data public.

The Monster Ops Playbook has been updated. @snovod @sahakiann @John-Scira, you'll want to update your readme as well.

To Test:

• Create a Manifest like:

EBI,003d5674-9bf6-4e51-ab1b-8fed80c308b9,Yes
EBI,07e5ebc0-1386-4a33-8ce4-3007705adad8,No
EBI,14da8a72-8c00-41c2-a4fc-21a5563514e0,No

• Run docker compose run -w /hca-ingest app bash in the terminal to load the Docker dev env. If you run into problems accessing the image try

gcloud auth application-default login
gcloud auth configure-docker us-east4-docker.pkg.dev

• Authenticate with gcloud in the container

gcloud auth login
gcloud config set project mystical-slate-284720
gcloud auth application-default login

• Install dependencies

cd orchestration/
poetry install

• Forward ports for Dagster

cd ../ops/helmfiles/dagster
./forward_ports.sh dev &

• Run manifest.py
  cd ../../../orchestration/
  python3 hca_manage/manifest.py load -e prod -c dcp<test_your_initials>_manifest.csv -r dcp<test_your_initials>

• In a local terminal (not in the Docker image)

Forward ports for Dagster
cd ../ops/helmfiles/dagster
./forward_ports.sh prod

• Open a browser to http://localhost:8080/workspace/all_jobs@monster-hca-ingest/pipelines/validate_ingress:default/partitions

Verify that you see your release in the partition set lists and verify that the make_public partition set does not include 003d5674-9bf6-4e51-ab1b-8fed80c308b9

Checklist

• Documentation has been updated as needed.