[DDO-2905] Migrate local development config out of firecloud-develop

[choover-broad]

The goal of this PR is to do the minimal amount of work to migrate the existing Orch local development configuration out of firecloud-develop.

Under the existing process, developers clone firecloud-develop and run sh run-context/local/scripts/firecloud-setup.sh, which populates the configs subdirectory with a bunch of configuration files and secrets, as well as a script called docker-rsync-local-orch.sh, which can be run to spin up a local Orch server inside a container.

This PR replaces the firecloud-develop step with a new helper script, local-dev/bin/render, that generates the same set of files. The render tool generates 3 types of configuration files, giving it approximate parity with firecloud-develop's configure.rb:

• secrets pulled from Vault (declared in local-dev/secrets.yaml)
• flat configuration files (checked in under local-dev/templates without a .ctmpl extension)
• consul templates (checked in under local-dev/templates with a .ctmpl extension)

Testing

I verified that I could:

• run the docker-rsync-local-orch.sh script
• connect to Orch swagger at https://local.broadinstitute.org:10443/
• authorize to Swagger via the implicit OAuth flow
• successfully execute the GET /api/configurations and GET /api/workspaces API calls

Modified files

A diff on the files generated by the old process and the new process showed the following delta:

firecloud-orchestration.conf

Files config/firecloud-orchestration.conf and config.from-fc-dev/firecloud-orchestration.conf differ

The firecloud-develop template contains a large hard-coded JSON structure of whitelists. After chatting with Identiteam, it's fine to omit the nih config struct altogether.

local.broadinstitute.org cert & key

Only in config: server.crt
Only in config: server.key

These files are the local.broadinstitute.org cert & key pair located at secret/dsde/firecloud/local/common. They were removed from firecloud-develop configs for by https://github.com/broadinstitute/firecloud-develop/pull/3326, which broke TLS for Orch. This PR adds them back.

site.conf

Files config/site.conf and config.from-fc-dev/site.conf differ

Removed the Apache proxy's OAuth claim allowlist as done with Rawls.

TCell

Only in config.from-fc-dev: tcell_agent.config

Intentionally removed; there is no need to run TCell as part of local development.

trial billing account

Only in config.from-fc-dev: trial-billing-account.json
Only in config.from-fc-dev: trial-billing-account.pem

Intentionally removed; the trial billing SA key is no longer used by Orch.

Related

• https://github.com/broadinstitute/firecloud-develop/pull/3386 will be merged after this PR to prevent confusion (i.e. people updating old files that are no longer used)

---

Have you read CONTRIBUTING.md lately? If not, do that first.

I, the developer opening this PR, do solemnly pinky swear that:

• I've followed the instructions if I've made any changes to the API, especially if they're breaking changes
• I've updated the RC_XXX release ticket with any manual steps required to release this change
• I've updated the FISMA documentation if I've made any security-related changes, including auth, encryption, or auditing

In all cases:

• Get two thumbsworth of review and PO signoff if necessary
• Verify all tests go green
• Squash and merge; you can delete your branch after this unless it's for a hotfix. In that case, don't delete it!
• Test this change deployed correctly and works on dev environment after deployment

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (87c6e2a) 69.54% compared to head (14a6805) 69.54%.

❗ Current head 14a6805 differs from pull request most recent head 0604ae8. Consider uploading reports for the commit 0604ae8 to get more accurate results

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #1162   +/-   ##
========================================
  Coverage    69.54%   69.54%           
========================================
  Files          101      101           
  Lines         3471     3471           
  Branches       362      362           
========================================
  Hits          2414     2414           
  Misses        1057     1057           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[davidangb]

lgtm! Sorry for the delay and thanks for this important PR.

As far as the secret/dsde/firecloud/local/firecloud-orchestration/whitelists change, I'd love to hear from Identiteam about how to manage that, since they own these features.

[choover-broad]

I chatted with Identiteam on Slack and it sounds like they're fine with removing the NIH configs entirely for local development.

[em-may]

Seems reasonable and follows the pattern. Stamp.

[choover-broad]

jenkins retest

[choover-broad]

jenkins retest