Fix Staging Deployments

[rajadain]

The staging site: https://staging.modelmywatershed.org has been down since at least Monday April 15, 2024.

Bring it back up, and describe what broke it.

[rajadain]

The launch-stacks command:

python3 mmw_stack.py launch-stacks --aws-profile mmw-stg --mmw-profile staging --mmw-config-path ~/dev/model-my-watershed/scratch/default.yml --stack-color green --activate-dns

Fails with the following message:

[DEBUG] VPC-34fed9de992df2f38f0276d2bd45 UPDATE_COMPLETE
[INFO] VPC-34fed9de992df2f38f0276d2bd45 finished
[INFO] S3VPCEndpoint-23d722307cd4c23619 finished
[INFO] Stack Tiler-4b3486903019678bd5785d68cd created
[DEBUG] Tiler-4b3486903019678bd5785d68cd CREATE_IN_PROGRESS
[DEBUG] Tiler-4b3486903019678bd5785d68cd CREATE_IN_PROGRESS
[DEBUG] Tiler-4b3486903019678bd5785d68cd CREATE_IN_PROGRESS
[DEBUG] Tiler-4b3486903019678bd5785d68cd CREATE_IN_PROGRESS
[DEBUG] Tiler-4b3486903019678bd5785d68cd CREATE_COMPLETE
[INFO] Tiler-4b3486903019678bd5785d68cd finished
[INFO] PrivateHostedZone-1e8810baf4b563 finished
[DEBUG] TileDeliveryNetwork-a9c314b5583b UPDATE_COMPLETE
[INFO] TileDeliveryNetwork-a9c314b5583b finished
[INFO] Stack Application-50395618a72d6a685947 created
[DEBUG] Application-50395618a72d6a685947 CREATE_IN_PROGRESS
[DEBUG] Application-50395618a72d6a685947 CREATE_IN_PROGRESS
[DEBUG] Application-50395618a72d6a685947 ROLLBACK_COMPLETE
[ERROR] Application-50395618a72d6a685947 failed
[INFO] Stack Worker-0f217d4f07557999b7479bd9c created
[DEBUG] Worker-0f217d4f07557999b7479bd9c CREATE_IN_PROGRESS
[DEBUG] Worker-0f217d4f07557999b7479bd9c CREATE_IN_PROGRESS
[DEBUG] Worker-0f217d4f07557999b7479bd9c CREATE_IN_PROGRESS
[DEBUG] Worker-0f217d4f07557999b7479bd9c CREATE_IN_PROGRESS
[DEBUG] Worker-0f217d4f07557999b7479bd9c CREATE_IN_PROGRESS
[DEBUG] Worker-0f217d4f07557999b7479bd9c CREATE_IN_PROGRESS
[DEBUG] Worker-0f217d4f07557999b7479bd9c CREATE_IN_PROGRESS
[DEBUG] Worker-0f217d4f07557999b7479bd9c CREATE_COMPLETE
[INFO] Worker-0f217d4f07557999b7479bd9c finished
Traceback (most recent call last):
  File "/home/ttuhinanshu/dev/model-my-watershed/deployment/mmw_stack.py", line 105, in <module>
    main()
  File "/home/ttuhinanshu/dev/model-my-watershed/deployment/mmw_stack.py", line 102, in main
    args.func(mmw_config=mmw_config, **vars(args))
  File "/home/ttuhinanshu/dev/model-my-watershed/deployment/mmw_stack.py", line 16, in launch_stacks
    build_stacks(mmw_config, aws_profile, **kwargs)
  File "/home/ttuhinanshu/dev/model-my-watershed/deployment/cfn/stacks.py", line 112, in build_stacks
    public_hosted_zone_graph.go()
  File "/home/ttuhinanshu/dev/model-my-watershed/deployment/.venv/lib/python3.10/site-packages/majorkirby/majorkirby.py", line 277, in go
    self.heartbeat(0)
  File "/home/ttuhinanshu/dev/model-my-watershed/deployment/.venv/lib/python3.10/site-packages/majorkirby/majorkirby.py", line 399, in heartbeat
    self._launch_cfn()
  File "/home/ttuhinanshu/dev/model-my-watershed/deployment/.venv/lib/python3.10/site-packages/majorkirby/majorkirby.py", line 470, in _launch_cfn
    self.action()
  File "/home/ttuhinanshu/dev/model-my-watershed/deployment/cfn/public_hosted_zone.py", line 43, in action
    app_lb_endpoint = self.get_input('AppServerLoadBalancerEndpoint')
  File "/home/ttuhinanshu/dev/model-my-watershed/deployment/.venv/lib/python3.10/site-packages/majorkirby/majorkirby.py", line 187, in get_input
    raise MKUnresolvableInputError
majorkirby.majorkirby.MKUnresolvableInputError

Looking at the AWS Console, it seems that the App Server Load Balancer isn't being created successfully, thus it's endpoint cannot be fetched:

model-my-watershed/deployment/cfn/application.py

Lines 294 to 295 in 5e44175

	self.add_output(Output('AppServerLoadBalancerEndpoint',
	Value=GetAtt(app_server_lb, 'DNSName')))

Trying to see if I can get the launch-stacks command to run more verbosely.

[rajadain]

Looking at the CloudFormation Stacks logs, we see the following error:

This is a likely reference to the staging.portal.bigcz.org domain that was removed as part of #3610 (comment)

Now going to see if executing some of #3613, namely the deployment bit, can fix this.

[rajadain]

Seeing promising results in https://github.com/WikiWatershed/model-my-watershed/tree/tt/3617/remove-bigcz-provisioning locally. Now running this on Jenkins to see if it works: http://civicci01.phl.corp.element84.com/view/mmw/job/model-my-watershed-develop/1341/

[rajadain]

Staging site is back up: https://staging.modelmywatershed.org/. Going to open a PR with the above changes.