Sandbox Environment (#4383)

* Initial sandbox prototype

Services are built, postgrest is online, but having difficulties with
the app, proxies, logshipper and scanner. Likely due to the fact that
sandbox can't create multiple spaces for the egress. Considering putting
them in the same sandbox space, and keeping everything on apps.internal

* Terraform runs, fails to stage

 2024-09-09T15:43:11.60-0400 [STG/0] ERR Warning: the last buildpack is not compatible with multi-buildpack apps and cannot make use of any dependencies supplied by the buildpacks specified before it
   2024-09-09T15:43:11.60-0400 [STG/0] ERR Warning: this buildpack can only be run as a supply buildpack, it can not be run alone
   2024-09-09T15:43:12.48-0400 [STG/0] ERR Failed to compile droplet: Failed to compile droplet: exit status 1

* App and AV deploy.. but..

App:
   2024-09-10T14:00:13.78-0400 [APP/PROC/WEB/0] OUT Invoking pre-start scripts.
   2024-09-10T14:00:13.99-0400 [APP/PROC/WEB/0] OUT STARTUP LOCAL_ENV No environment variable is set!
   2024-09-10T14:00:13.99-0400 [APP/PROC/WEB/0] OUT STARTUP STARTUP_CHECK setup_env FAIL
Looks like the app is atleast trying to run the .profile

AV:
Out of memory. Need to increase the memory, potentially ask cgov about expanding
sandbox quota limits for memory

* Modified setup to try and get a running instance

Due to the error presented (Which we expect)
2024-09-11T09:24:17.29-0400 [APP/PROC/WEB/0] OUT Invoking pre-start scripts.
   2024-09-11T09:24:17.47-0400 [APP/PROC/WEB/0] OUT STARTUP LOCAL_ENV Environment set as: SANDBOX
   2024-09-11T09:24:17.72-0400 [APP/PROC/WEB/0] OUT STARTUP STARTUP_CHECK setup_env PASS
   2024-09-11T09:24:17.72-0400 [APP/PROC/WEB/0] OUT STARTUP API_TEARDOWN BEGIN
   2024-09-11T09:24:17.72-0400 [APP/PROC/WEB/0] OUT STARTUP DROP_DEPRECATED_API_SCHEMA_AND_VIEWS BEGIN

   2024-09-11T09:24:18.19-0400 [APP/PROC/WEB/0] ERR show_deprecation_warning()
   2024-09-11T09:24:19.43-0400 [APP/PROC/WEB/0] ERR {"message": "RSA key format is not supported"}
   2024-09-11T09:24:19.43-0400 [APP/PROC/WEB/0] ERR {"message": "Key jar preparation failed for https://idp.int.identitysandbox.gov"}
   2024-09-11T09:24:19.43-0400 [APP/PROC/WEB/0] ERR {"message": "[******] The system broke while trying to log you in."}
   2024-09-11T09:24:19.43-0400 [APP/PROC/WEB/0] ERR {"message": "Unable to configure OpenID Connect provider. Users cannot log in."}

This will require some dev intervention

* Update gitignore

* Disable newrelic start

* Add note about proxy

* Update Readme

Force app replacement

* Use an old procfile to try and get system to boot

* Add --preload to debug stack

* Test without collectstatic

* Modify profile again

* Disable the entire .profile

* Try to just do something that will pass

* Just use an echo for now

* Preboot in sandbox

Ultimately will not work due to out of memory exceptions being thrown

* Single Worker thread

* executable for start command in terraform

* Modify based on SO post

* Fails gunicorn startup

* Reconfigure the app now that we can do strategy rolling

* update chmod for destroy

* Reset everything to try and get a working deployment

* Add fac-key-service

* disable api

* Proxy configuration seems good

At this time, there are still a few major issues, but.. it does appear that
the proxy is working as intended. Even though the deploy failed for staticfiles
and on the boot sequence, the attempts to reach out to identitysandbox seemed to work,
and were not 443'd.

* Disable collectstatic

* Reset proxy

* Add direct route back

* Remove --preload, maybe fix whitenoise missing import?

* Revert Procfile

* Add the correct routing and environment variables

* Make sure settings.py included sandbox space

* Remove depends_on

* Readd staticfiles in .profile

* disable once more

* Modify app

* add collectstatic back

* disable once again

* Get things back to stable

* Update System

* Modify bin/ops files to run

* Create a complete initial README.md

* Update gitignore

* Delete terraform lock file

* Have a dedicated https-proxy for sandbox

This moves the client registration of the network policies to be out of the proxy module
without impacting the live proxy

* Add acl for client registration

* add acl support for routes

* Specify only the branch

* Infra config improvements

* Ensure we have compiled assets

* Bump the timeout to 15m

* Disable smtp-proxy (for now)

* Re-enable API boot sequence

* Disable collectstatic in .profile

* CRLF -> LF

* Fix command

* add MV

* Executable

* Copy paste error

* Update Readme

* terraform fmt

* Add sandbox.tfvars helper

* Add helper script

* Update helper scripts

* Indent fix

* Update Readme

* Update readme

* Update readme

* Update Readme

* Execution bit, README edits

Some changes useful, some just flags for further editing.

* cleanup helper scripts

* Add backup db

* Enable partial config with s3 to store tfstate

* Update Readme

* Update Readme

* Fix cat

* Uncomment Destroy

* Fix proxy

* Revert

* Update Readme

* Update Readme

* Update Readme

* Update Readme

* Update Readme

* Update Readme

* Update Readme

* Test fixing migrations

* revert

* Update Readme

* Undo smtp proxy change

* CRLF -> LF

* CRLF -> LF

* CRLF -> LF

* Cleanup .profile

* CRLF -> LF

* Regenerate Examples with Meta Module

* Upgrade Provider

* Module Upgrades

* terraform fmt

* Update provider

---------

Co-authored-by: Matt Jadud <[email protected]>

.gitignore

@@ -167,6 +167,14 @@ terraform/**/*.tfstate*
 terraform/**/*.tfvars
 terraform/shared/modules/egress-proxy/proxy.zip
 terraform/shared/modules/egress-proxy/test/client.zip
+terraform/shared/modules/app/app.zip
+terraform/shared/modules/https-proxy/proxy.zip
+terraform/shared/modules/https-proxy/.terraform.lock.hcl
+terraform/shared/modules/stream-proxy/proxy.zip
+terraform/shared/modules/https-proxy/.terraform.lock.hcl
+terraform/shared/modules/sandbox-proxy/proxy.zip
+terraform/shared/modules/sandbox-proxy/.terraform.lock.hcl
+
 
 # XLSX ignores
 .~*#

backend/.profile

@@ -1,9 +1,9 @@
 #!/bin/bash
 
 # Source everything; everything is now a function.
-# Remember: bash has no idea if a function exists, 
+# Remember: bash has no idea if a function exists,
 # so a typo in a function name will fail silently. Similarly,
-# bash has horrible scoping, so use of `local` in functions is 
+# bash has horrible scoping, so use of `local` in functions is
 # critical for cleanliness in the startup script.
 source tools/util_startup.sh
 # This will choose the correct environment
@@ -14,6 +14,7 @@ source tools/migrate_app_tables.sh
 source tools/api_standup.sh
 source tools/run_collectstatic.sh
 source tools/seed_cog_baseline.sh
+source tools/materialized_views.sh
 
 #####
 # SETUP THE CGOV ENVIRONMENT
@@ -43,15 +44,18 @@ if [[ "$CF_INSTANCE_INDEX" == 0 ]]; then
     #####
     # COLLECT STATIC
     # Do Django things with static files.
-    run_collectstatic
-    gonogo "run_collectstatic"
+    # run_collectstatic
+    # gonogo "run_collectstatic"
 
     #####
     # SEED COG/OVER TABLES
     # Setup tables for cog/over assignments
     seed_cog_baseline
     gonogo "seed_cog_baseline"
+
+    # materialized_views
+    # gonogo "materialized_views"
 fi
 
 # Make psql usable by scripts, for debugging, etc.
-alias psql='/home/vcap/deps/0/apt/usr/lib/postgresql/*/bin/psql'
+alias psql='/home/vcap/deps/0/apt/usr/lib/postgresql/*/bin/psql'

backend/config/settings.py

@@ -227,8 +227,7 @@
 
 # Environment specific configurations
 DEBUG = False
-
-if ENVIRONMENT not in ["DEVELOPMENT", "PREVIEW", "STAGING", "PRODUCTION"]:
+if ENVIRONMENT not in ["SANDBOX", "DEVELOPMENT", "PREVIEW", "STAGING", "PRODUCTION"]:
     DATABASES = {
         "default": env.dj_db_url(
             "DATABASE_URL", default="postgres://postgres:[email protected]/backend"

backend/tools/materialized_views.sh

@@ -0,0 +1,9 @@
+source tools/util_startup.sh
+
+function materialized_views {
+    startup_log "RUN_MATERIALIZEDVIEWS" "BEGIN"
+    python manage.py materialized_views --create &&
+    local result=$?
+    startup_log "RUN_MATERIALIZEDVIEWS" "END"
+    return $result
+}

bin/ops/create_service_account.sh

@@ -70,4 +70,4 @@ cat << EOF
 
 cf_user = $username
 cf_password = $password
-EOF
+EOF

bin/ops/destroy_service_account.sh

@@ -50,4 +50,4 @@ cf target -o $org -s $space
 cf delete-service-key $service ${service}-key -f
 
 # destroy service
-cf delete-service $service -f
+cf delete-service $service -f

bin/ops/get_service_account.sh

@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+
+org="gsa-tts-oros-fac"
+
+usage="
+$0: Create a Service User Account for a given space
+
+Usage:
+  $0 -h
+  $0 -s <SPACE NAME> -u <USER NAME> [-r <ROLE NAME>] [-o <ORG NAME>]
+
+Options:
+-h: show help and exit
+-s <SPACE NAME>: configure the space to act on. Required
+-u <USER NAME>: set the service user name. Required
+-r <ROLE NAME>: set the service user's role to either space-deployer or space-auditor. Default: space-deployer
+-o <ORG NAME>: configure the organization to act on. Default: $org
+"
+
+set -e
+set -o pipefail
+
+space=""
+service=""
+role="space-deployer"
+
+while getopts ":hs:u:r:o:" opt; do
+  case "$opt" in
+    s)
+      space=${OPTARG}
+      ;;
+    u)
+      service=${OPTARG}
+      ;;
+    r)
+      role=${OPTARG}
+      ;;
+    o)
+      org=${OPTARG}
+      ;;
+    h)
+      echo "$usage"
+      exit 0
+      ;;
+  esac
+done
+
+if [[ $space = "" || $service = "" ]]; then
+  echo "$usage"
+  exit 1
+fi
+
+>&2 echo "Targeting org $org and space $space"
+cf target -o $org -s $space > /dev/null 2>&1
+
+# get service key
+cf service-key $service ${service}-key > /dev/null 2>&1
+
+# output service key to stdout in secrets.auto.tfvars format
+creds=`cf service-key $service ${service}-key | tail -n 7`
+username=`echo $creds | jq '.credentials.username'`
+password=`echo $creds | jq '.credentials.password'`
+
+cat <<EOM
+# generated with $0 -s $space -u $service -r $role -o $org
+# revoke with $(dirname $0)/destroy_service_account.sh -s $space -u $service -o $org
+
+cf_user = $username
+cf_password = $password
+EOM

terraform/.gitignore

@@ -1,6 +1,7 @@
-*/backend.tfvars
-*/.terraform
-*/terraform.tfstate
-*/terraform.tfstate.backup
-*/deployer-creds.auto.tfvars
-shared/config/backend.tfvars
+*/backend.tfvars
+*/.terraform
+*/terraform.tfstate
+*/terraform.tfstate.backup
+*/deployer-creds.auto.tfvars
+shared/config/backend.tfvars
+*/*.tfplan

terraform/dev/dev.tf-example

@@ -1,3 +1,12 @@
+# The content of this file is managed by Terraform. If you modify it, it may
+# be reverted the next time Terraform runs. If you want to make changes, do it
+# in ../meta/bootstrap-env/templates.
+
+# Add resources to this module describing what you want in the corresponding 
+# space in cloud.gov. You should probably just reference a shared module as in the
+# example below to keep consistency across spaces, but you can also vary the 
+# content for each environment as needed.
+
 module "dev" {
   source        = "../shared/modules/env"
   cf_space_name = "dev"

terraform/preview/preview.tf-example

@@ -2,6 +2,11 @@
 # be reverted the next time Terraform runs. If you want to make changes, do it
 # in ../meta/bootstrap-env/templates.
 
+# Add resources to this module describing what you want in the corresponding 
+# space in cloud.gov. You should probably just reference a shared module as in the
+# example below to keep consistency across spaces, but you can also vary the 
+# content for each environment as needed.
+
 module "preview" {
   source        = "../shared/modules/env"
   cf_space_name = "preview"

terraform/production/production.tf-example

@@ -1,3 +1,12 @@
+# The content of this file is managed by Terraform. If you modify it, it may
+# be reverted the next time Terraform runs. If you want to make changes, do it
+# in ../meta/bootstrap-env/templates.
+
+# Add resources to this module describing what you want in the corresponding 
+# space in cloud.gov. You should probably just reference a shared module as in the
+# example below to keep consistency across spaces, but you can also vary the 
+# content for each environment as needed.
+
 module "production" {
   source        = "../shared/modules/env"
   cf_space_name = "production"