Deploy Opensearch #2341

coilysiren · 2024-10-02T19:18:56Z

Summary

Fixes #1656

Time to review: 10 ~ 30 mins

Changes proposed

Deploys an opensearch domain
Configures the opensearch domain to be high availability in prod, and as cheap as possible (while still being similarly configured) elsewhere. Prod is currently turned off, though.
Adds opensearch secrets to the API service

Context for reviewers

Opensearch takes about 20 minutes to deploy, similarly to RDS Postgres

Generating the password was a pain in the butt

This style of this infra code is similar to the style of platform code, so I'm going to post it in our platform channel for other people to see

This PR tangentially includes the updated secrets handling from navapbc/template-infra#602. I added that because I'm adding a bunch of secrets, and I don't want them to conflict with #2351.

Additional information

Opensearch deployment:

(staging has since been torn down)

The terraform diff for the API service layer:

                    {
                        Action = "ssm:GetParameters"
                        Effect = "Allow"
                      ~ Resource = "arn:aws:ssm:*:*:parameter/api/dev/api-auth-token" -> [
                          + "arn:aws:ssm:us-east-1:315341936575:parameter/search/dev/username"
                          + "arn:aws:ssm:us-east-1:315341936575:parameter/search/dev/password"
                          + "arn:aws:ssm:us-east-1:315341936575:parameter/search/dev/endpoint"
                          + "arn:aws:ssm:us-east-1:315341936575:parameter/api/dev/api-auth-token"
                        ]
                        Sid = "SecretsAccess"
                    }
                ]

                    secrets = [
                        {
                        }
                      + {
                          + name = "SEARCH_ENDPOINT"
                          + valueFrom = "arn:aws:ssm:us-east-1:315341936575:parameter/search/dev/endpoint"
                        }
                      + {
                          + name = "SEARCH_PASSWORD"
                          + valueFrom = "arn:aws:ssm:us-east-1:315341936575:parameter/search/dev/password"
                        }
                      + {
                          + name = "SEARCH_USERNAME"
                          + valueFrom = "arn:aws:ssm:us-east-1:315341936575:parameter/search/dev/username"
                        }
                    ]

### Time to review: __2 mins__ ## Changes proposed - Updated checkov - Adds various TODOs and ignores to our code, in service of making checkov pass ## Context for reviewers Checkov is a terraform security linter. I'm doing this PR now for two reasons: 1. renovate bot has been pinging me non-stop with checkov update requests 2. I got some strange checkov bugs in #2341, so I decided it was time to update

infra/modules/opensearch/main.tf

coilysiren · 2024-10-10T08:21:42Z

Related checkov Github issue: bridgecrewio/checkov#6760

mdragon

LGTM

chouinar

Just two small env var related comments

chouinar · 2024-10-10T14:35:43Z

infra/api/service/main.tf

@@ -138,8 +138,26 @@ module "service" {
    }
  } : null

-  extra_environment_variables = merge(local.service_config.extra_environment_variables, { "ENVIRONMENT" : var.environment_name })


It looks like you undid the change you made here

Ahah! Thanks

chouinar · 2024-10-10T14:39:39Z

infra/api/service/main.tf

+  secrets = concat(
+    [for secret_name in keys(local.service_config.secrets) : {
+      name      = secret_name
+      valueFrom = module.secrets[secret_name].secret_arn
+    }],
+    local.environment_config.search_config != null ? [{
+      name      = "SEARCH_USERNAME"
+      valueFrom = data.aws_ssm_parameter.search_username_arn[0].arn
+    }] : [],
+    local.environment_config.search_config != null ? [{
+      name      = "SEARCH_PASSWORD"
+      valueFrom = data.aws_ssm_parameter.search_password_arn[0].arn
+    }] : [],
+    local.environment_config.search_config != null ? [{
+      name      = "SEARCH_ENDPOINT"
+      valueFrom = data.aws_ssm_parameter.search_endpoint_arn[0].arn
+    }] : []
+  )


What port do we connect to OpenSearch on?

It didn't tell me! Whatever the default is

@chouinar env vars thing fixed o7

fast-follow to #2341

## Summary Fixes #2485 ### Time to review: __2 mins__ ## Changes proposed Fixes secrets stuff for the frontend and analytics modules ## Context for reviewers Build failure => https://github.com/HHS/simpler-grants-gov/actions/runs/11354826962/job/31583001030 Root cause => #2341 ## Additional information Testing: <details> <summary>frontend terraform plan</summary> ``` terraform plan -var "environment_name=dev" data.terraform_remote_state.current_image_tag[0]: Reading... module.app_config.data.external.account_ids_by_name: Reading... module.app_config.data.external.account_ids_by_name: Read complete after 0s [id=-] module.service.data.aws_iam_policy_document.ecs_tasks_assume_role_policy: Reading... module.service.data.aws_caller_identity.current: Reading... module.service.data.aws_ecr_repository.app[0]: Reading... module.monitoring.aws_cloudwatch_log_metric_filter.service_error_filter: Refreshing state... [id=service-error-filter] data.aws_acm_certificate.cert[0]: Reading... module.service.aws_cloudwatch_log_group.WafWebAclLoggroup[0]: Refreshing state... [id=aws-waf-logs-wafv2-web-acl-frontend-dev] module.service.aws_cloudwatch_log_group.service_logs: Refreshing state... [id=service/frontend-dev] module.service.aws_ecs_cluster.cluster: Refreshing state... [id=arn:aws:ecs:us-east-1:315341936575:cluster/frontend-dev] module.service.data.aws_iam_policy_document.ecs_tasks_assume_role_policy: Read complete after 0s [id=597844978] module.service.data.aws_region.current: Reading... module.service.data.aws_region.current: Read complete after 0s [id=us-east-1] module.secrets["API_URL"].data.aws_ssm_parameter.secret[0]: Reading... module.service.data.aws_caller_identity.current: Read complete after 0s [id=315341936575] module.secrets["SENDY_API_KEY"].data.aws_ssm_parameter.secret[0]: Reading... module.service.aws_wafv2_web_acl.waf[0]: Refreshing state... [id=c7b7b8d6-3f15-497d-8ec8-66c6239cdff2] module.secrets["API_URL"].data.aws_ssm_parameter.secret[0]: Read complete after 0s [id=/frontend/dev/api-url] module.secrets["SENDY_API_URL"].data.aws_ssm_parameter.secret[0]: Reading... module.secrets["SENDY_API_KEY"].data.aws_ssm_parameter.secret[0]: Read complete after 0s [id=/frontend/dev/sendy-api-key] module.secrets["SENDY_LIST_ID"].data.aws_ssm_parameter.secret[0]: Reading... module.secrets["API_AUTH_TOKEN"].data.aws_ssm_parameter.secret[0]: Reading... module.service.aws_s3_bucket.general_purpose: Refreshing state... [id=frontend-dev-general-purpose20240422224603864800000001] module.service.aws_s3_bucket.access_logs: Refreshing state... [id=frontend-dev-access-logs20230818175923948800000003] module.secrets["SENDY_API_URL"].data.aws_ssm_parameter.secret[0]: Read complete after 0s [id=/frontend/dev/sendy-api-url] module.secrets["SENDY_LIST_ID"].data.aws_ssm_parameter.secret[0]: Read complete after 0s [id=/frontend/dev/sendy-list-id] module.monitoring.aws_sns_topic.this: Refreshing state... [id=arn:aws:sns:us-east-1:315341936575:frontend-dev-monitoring] data.aws_vpc.network: Reading... module.service.aws_iam_role.app_service: Refreshing state... [id=frontend-dev-app] data.terraform_remote_state.current_image_tag[0]: Read complete after 1s module.service.aws_iam_role.task_executor: Refreshing state... [id=frontend-dev-task-executor] module.secrets["API_AUTH_TOKEN"].data.aws_ssm_parameter.secret[0]: Read complete after 1s [id=/frontend/dev/api-auth-token] module.service.data.aws_iam_policy_document.WafWebAclLoggingDoc[0]: Reading... module.service.data.aws_iam_policy_document.WafWebAclLoggingDoc[0]: Read complete after 0s [id=31210429] module.service.aws_cloudwatch_log_resource_policy.WafWebAclLoggingPolicy[0]: Refreshing state... [id=service-frontend-dev-webacl-policy] data.aws_acm_certificate.cert[0]: Read complete after 1s [id=arn:aws:acm:us-east-1:315341936575:certificate/3123d09e-efb2-43f8-bccc-30fd2e39944d] module.monitoring.aws_sns_topic_subscription.email_integration["[email protected]"]: Refreshing state... [id=arn:aws:sns:us-east-1:315341936575:frontend-dev-monitoring:d2493ff1-58fc-4c1c-86a3-80be4454588b] module.monitoring.aws_cloudwatch_metric_alarm.service_errors: Refreshing state... [id=frontend-dev-errors] module.service.data.aws_ecr_repository.app[0]: Read complete after 1s [id=simpler-grants-gov-frontend] module.service.data.aws_iam_policy_document.task_executor: Reading... module.service.data.aws_iam_policy_document.task_executor: Read complete after 0s [id=351829213] module.service.aws_iam_role_policy.task_executor: Refreshing state... [id=frontend-dev-task-executor:frontend-dev-task-executor-role-policy] data.aws_vpc.network: Read complete after 1s [id=vpc-08f522c5cc442d126] data.aws_subnets.public: Reading... data.aws_subnets.private: Reading... module.service.aws_security_group.alb: Refreshing state... [id=sg-09f21e3710e63e128] module.service.aws_lb_target_group.app_tg[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:us-east-1:315341936575:targetgroup/app-20240129183934308400000001/6291f09ee717b0d7] data.aws_subnets.public: Read complete after 1s [id=us-east-1] data.aws_subnets.private: Read complete after 1s [id=us-east-1] module.service.aws_security_group_rule.http_ingress: Refreshing state... [id=sgrule-982783969] module.service.aws_security_group_rule.https_ingress[0]: Refreshing state... [id=sgrule-1653939980] module.service.aws_security_group.app: Refreshing state... [id=sg-0ab68d90ef2574a9b] module.service.aws_s3_bucket_server_side_encryption_configuration.general_purpose_encryption: Refreshing state... [id=frontend-dev-general-purpose20240422224603864800000001] module.service.aws_s3_bucket_public_access_block.general_purpose: Refreshing state... [id=frontend-dev-general-purpose20240422224603864800000001] module.service.data.aws_iam_policy_document.general_purpose_put_access: Reading... module.service.aws_s3_bucket_lifecycle_configuration.general_purpose: Refreshing state... [id=frontend-dev-general-purpose20240422224603864800000001] module.service.data.aws_iam_policy_document.general_purpose_put_access: Read complete after 0s [id=1545896503] module.service.aws_s3_bucket_public_access_block.access_logs: Refreshing state... [id=frontend-dev-access-logs20230818175923948800000003] module.service.aws_s3_bucket_server_side_encryption_configuration.encryption: Refreshing state... [id=frontend-dev-access-logs20230818175923948800000003] module.service.data.aws_iam_policy_document.access_logs_put_access: Reading... module.service.data.aws_iam_policy_document.access_logs_put_access: Read complete after 0s [id=1899969278] module.service.aws_s3_bucket_lifecycle_configuration.access_logs: Refreshing state... [id=frontend-dev-access-logs20230818175923948800000003] module.service.aws_s3_bucket_policy.general_purpose: Refreshing state... [id=frontend-dev-general-purpose20240422224603864800000001] module.service.aws_ecs_task_definition.app: Refreshing state... [id=frontend-dev] module.service.aws_s3_bucket_policy.access_logs: Refreshing state... [id=frontend-dev-access-logs20230818175923948800000003] module.service.aws_ecs_service.app: Refreshing state... [id=arn:aws:ecs:us-east-1:315341936575:service/frontend-dev/frontend-dev] module.service.aws_appautoscaling_target.ecs_target[0]: Refreshing state... [id=service/frontend-dev/frontend-dev] module.service.aws_lb.alb[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:us-east-1:315341936575:loadbalancer/app/frontend-dev/3c9002fb0aa7756d] module.service.aws_wafv2_web_acl_logging_configuration.WafWebAclLogging[0]: Refreshing state... [id=arn:aws:wafv2:us-east-1:315341936575:regional/webacl/frontend-dev-wafv2-web-acl/c7b7b8d6-3f15-497d-8ec8-66c6239cdff2] module.service.aws_wafv2_web_acl_association.WafWebAclAssociation[0]: Refreshing state... [id=arn:aws:wafv2:us-east-1:315341936575:regional/webacl/frontend-dev-wafv2-web-acl/c7b7b8d6-3f15-497d-8ec8-66c6239cdff2,arn:aws:elasticloadbalancing:us-east-1:315341936575:loadbalancer/app/frontend-dev/3c9002fb0aa7756d] module.service.aws_lb_listener.alb_listener_https[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:us-east-1:315341936575:listener/app/frontend-dev/3c9002fb0aa7756d/28ac1bd2b35d1f12] module.service.aws_lb_listener.alb_listener_http[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:us-east-1:315341936575:listener/app/frontend-dev/3c9002fb0aa7756d/2d05cbc27c817b74] module.monitoring.aws_cloudwatch_metric_alarm.high_app_response_time: Refreshing state... [id=frontend-dev-high-app-response-time] module.monitoring.aws_cloudwatch_metric_alarm.high_load_balancer_http_5xx_count: Refreshing state... [id=frontend-dev-high-load-balancer-5xx-count] module.monitoring.aws_cloudwatch_metric_alarm.high_app_http_5xx_count: Refreshing state... [id=frontend-dev-high-app-5xx-count] module.service.aws_appautoscaling_policy.ecs_scale_policy_cpu[0]: Refreshing state... [id=frontend-dev-ecs-scale-policy-cpu] module.service.aws_lb_listener_rule.app_http_forward[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:us-east-1:315341936575:listener-rule/app/frontend-dev/3c9002fb0aa7756d/2d05cbc27c817b74/abec3e44c8a18ab0] module.service.aws_lb_listener_rule.redirect_http_to_https[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:us-east-1:315341936575:listener-rule/app/frontend-dev/3c9002fb0aa7756d/2d05cbc27c817b74/4edd4f8a6ced7c90] module.service.aws_lb_listener_rule.app_https_forward[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:us-east-1:315341936575:listener-rule/app/frontend-dev/3c9002fb0aa7756d/28ac1bd2b35d1f12/789e732545d7caa4] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place -/+ destroy and then create replacement Terraform will perform the following actions: # module.service.aws_ecs_service.app will be updated in-place ~ resource "aws_ecs_service" "app" { id = "arn:aws:ecs:us-east-1:315341936575:service/frontend-dev/frontend-dev" name = "frontend-dev" tags = {} ~ task_definition = "arn:aws:ecs:us-east-1:315341936575:task-definition/frontend-dev:176" -> (known after apply) # (15 unchanged attributes hidden) # (4 unchanged blocks hidden) } # module.service.aws_ecs_task_definition.app must be replaced -/+ resource "aws_ecs_task_definition" "app" { ~ arn = "arn:aws:ecs:us-east-1:315341936575:task-definition/frontend-dev:176" -> (known after apply) ~ arn_without_revision = "arn:aws:ecs:us-east-1:315341936575:task-definition/frontend-dev" -> (known after apply) ~ container_definitions = jsonencode( ~ [ ~ { ~ linuxParameters = { ~ capabilities = { - add = [] # (1 unchanged attribute hidden) } # (1 unchanged attribute hidden) } - mountPoints = [] name = "frontend-dev" ~ portMappings = [ ~ { - hostPort = 8000 - protocol = "tcp" # (1 unchanged attribute hidden) }, ] ~ secrets = [ ~ { name = "API_AUTH_TOKEN" ~ valueFrom = "/frontend/dev/api-auth-token" -> "arn:aws:ssm:us-east-1:315341936575:parameter/frontend/dev/api-auth-token" }, ~ { name = "API_URL" ~ valueFrom = "/frontend/dev/api-url" -> "arn:aws:ssm:us-east-1:315341936575:parameter/frontend/dev/api-url" }, ~ { name = "SENDY_API_KEY" ~ valueFrom = "/frontend/dev/sendy-api-key" -> "arn:aws:ssm:us-east-1:315341936575:parameter/frontend/dev/sendy-api-key" }, ~ { name = "SENDY_API_URL" ~ valueFrom = "/frontend/dev/sendy-api-url" -> "arn:aws:ssm:us-east-1:315341936575:parameter/frontend/dev/sendy-api-url" }, ~ { name = "SENDY_LIST_ID" ~ valueFrom = "/frontend/dev/sendy-list-id" -> "arn:aws:ssm:us-east-1:315341936575:parameter/frontend/dev/sendy-list-id" }, ] - systemControls = [] - volumesFrom = [] # (8 unchanged attributes hidden) }, ] # forces replacement ) ~ id = "frontend-dev" -> (known after apply) ~ revision = 176 -> (known after apply) - tags = {} -> null # (12 unchanged attributes hidden) } # module.service.aws_iam_role_policy.task_executor will be updated in-place ~ resource "aws_iam_role_policy" "task_executor" { id = "frontend-dev-task-executor:frontend-dev-task-executor-role-policy" name = "frontend-dev-task-executor-role-policy" ~ policy = jsonencode( ~ { ~ Statement = [ # (8 unchanged elements hidden) { Action = [ "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability", ] Effect = "Allow" Resource = "arn:aws:ecr:us-east-1:315341936575:repository/simpler-grants-gov-frontend" Sid = "ECRPullAccess" }, ~ { ~ Resource = [ ~ "arn:aws:ssm:*:*:parameter/frontend/dev/sendy-list-id" -> "arn:aws:ssm:us-east-1:315341936575:parameter/frontend/dev/sendy-list-id", ~ "arn:aws:ssm:*:*:parameter/frontend/dev/sendy-api-url" -> "arn:aws:ssm:us-east-1:315341936575:parameter/frontend/dev/sendy-api-url", ~ "arn:aws:ssm:*:*:parameter/frontend/dev/sendy-api-key" -> "arn:aws:ssm:us-east-1:315341936575:parameter/frontend/dev/sendy-api-key", ~ "arn:aws:ssm:*:*:parameter/frontend/dev/api-url" -> "arn:aws:ssm:us-east-1:315341936575:parameter/frontend/dev/api-url", ~ "arn:aws:ssm:*:*:parameter/frontend/dev/api-auth-token" -> "arn:aws:ssm:us-east-1:315341936575:parameter/frontend/dev/api-auth-token", ] # (3 unchanged attributes hidden) }, ] # (1 unchanged attribute hidden) } ) # (2 unchanged attributes hidden) } Plan: 1 to add, 2 to change, 1 to destroy. ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now. ``` </details> <details> <summary>analytics terraform plan</summary> ``` terraform plan -var "environment_name=dev" data.terraform_remote_state.current_image_tag[0]: Reading... module.app_config.data.external.account_ids_by_name: Reading... module.app_config.data.external.account_ids_by_name: Read complete after 0s [id=-] module.service.data.aws_caller_identity.current: Reading... data.aws_vpc.network: Reading... module.service.data.aws_region.current: Reading... module.service.data.aws_ecr_repository.app[0]: Reading... aws_cloudwatch_log_group.sprint_reports: Refreshing state... [id=/aws/vendedlogs/states/analytics-dev-sprint-reports20240402190032717900000001] module.service.data.aws_iam_policy_document.ecs_tasks_assume_role_policy: Reading... aws_scheduler_schedule_group.sprint_reports: Refreshing state... [id=analytics-dev-sprint-reports] module.service.aws_cloudwatch_log_group.service_logs: Refreshing state... [id=service/analytics-dev] module.service.data.aws_region.current: Read complete after 0s [id=us-east-1] module.service.aws_s3_bucket.access_logs: Refreshing state... [id=analytics-dev-access-logs20240507210641615800000002] module.service.data.aws_iam_policy_document.ecs_tasks_assume_role_policy: Read complete after 0s [id=597844978] module.secrets["GH_TOKEN"].data.aws_ssm_parameter.secret[0]: Reading... module.secrets["ANALYTICS_REPORTING_CHANNEL_ID"].data.aws_ssm_parameter.secret[0]: Reading... module.service.data.aws_caller_identity.current: Read complete after 1s [id=315341936575] module.secrets["ANALYTICS_SLACK_BOT_TOKEN"].data.aws_ssm_parameter.secret[0]: Reading... module.secrets["ANALYTICS_REPORTING_CHANNEL_ID"].data.aws_ssm_parameter.secret[0]: Read complete after 1s [id=/analytics/dev/reporting-channel-id] data.aws_iam_policy.app_db_access_policy[0]: Reading... data.aws_rds_cluster.db_cluster[0]: Reading... module.secrets["GH_TOKEN"].data.aws_ssm_parameter.secret[0]: Read complete after 1s [id=/analytics/github-token] data.aws_iam_policy.migrator_db_access_policy[0]: Reading... module.secrets["ANALYTICS_SLACK_BOT_TOKEN"].data.aws_ssm_parameter.secret[0]: Read complete after 0s [id=/analytics/slack-bot-token] module.service.aws_ecs_cluster.cluster: Refreshing state... [id=arn:aws:ecs:us-east-1:315341936575:cluster/analytics-dev] module.service.aws_iam_role.task_executor: Refreshing state... [id=analytics-dev-task-executor] module.service.aws_s3_bucket.general_purpose: Refreshing state... [id=analytics-dev-general-purpose20240507210641715100000003] data.terraform_remote_state.current_image_tag[0]: Read complete after 1s module.service.aws_iam_role.app_service: Refreshing state... [id=analytics-dev-app] module.service.data.aws_ecr_repository.app[0]: Read complete after 1s [id=simpler-grants-gov-analytics] data.aws_rds_cluster.db_cluster[0]: Read complete after 0s [id=analytics-dev] module.service.aws_iam_role.migrator_task[0]: Refreshing state... [id=analytics-dev-migrator] data.aws_vpc.network: Read complete after 2s [id=vpc-08f522c5cc442d126] data.aws_subnets.public: Reading... data.aws_subnets.private: Reading... module.service.aws_security_group.alb: Refreshing state... [id=sg-039c7cd496d543e13] module.service.data.aws_iam_policy_document.task_executor: Reading... module.service.data.aws_iam_policy_document.task_executor: Read complete after 0s [id=678723376] module.service.aws_iam_role_policy.task_executor: Refreshing state... [id=analytics-dev-task-executor:analytics-dev-task-executor-role-policy] data.aws_subnets.public: Read complete after 0s [id=us-east-1] data.aws_subnets.private: Read complete after 0s [id=us-east-1] module.service.aws_security_group_rule.http_ingress: Refreshing state... [id=sgrule-1959610007] module.service.aws_security_group.app: Refreshing state... [id=sg-0ed2e3fe9a482683c] module.service.aws_s3_bucket_public_access_block.access_logs: Refreshing state... [id=analytics-dev-access-logs20240507210641615800000002] module.service.aws_s3_bucket_server_side_encryption_configuration.encryption: Refreshing state... [id=analytics-dev-access-logs20240507210641615800000002] module.service.data.aws_iam_policy_document.access_logs_put_access: Reading... module.service.aws_s3_bucket_lifecycle_configuration.access_logs: Refreshing state... [id=analytics-dev-access-logs20240507210641615800000002] module.service.data.aws_iam_policy_document.access_logs_put_access: Read complete after 0s [id=21413137] module.service.aws_s3_bucket_policy.access_logs: Refreshing state... [id=analytics-dev-access-logs20240507210641615800000002] module.service.aws_vpc_security_group_ingress_rule.db_ingress_from_service[0]: Refreshing state... [id=sgr-005d1c8947c789bd0] module.service.aws_s3_bucket_public_access_block.general_purpose: Refreshing state... [id=analytics-dev-general-purpose20240507210641715100000003] module.service.aws_s3_bucket_server_side_encryption_configuration.general_purpose_encryption: Refreshing state... [id=analytics-dev-general-purpose20240507210641715100000003] module.service.aws_s3_bucket_lifecycle_configuration.general_purpose: Refreshing state... [id=analytics-dev-general-purpose20240507210641715100000003] module.service.data.aws_iam_policy_document.general_purpose_put_access: Reading... module.service.aws_ecs_task_definition.app: Refreshing state... [id=analytics-dev] module.service.data.aws_iam_policy_document.general_purpose_put_access: Read complete after 0s [id=225885815] module.service.aws_s3_bucket_policy.general_purpose: Refreshing state... [id=analytics-dev-general-purpose20240507210641715100000003] aws_sfn_state_machine.sprint_reports: Refreshing state... [id=arn:aws:states:us-east-1:315341936575:stateMachine:analytics-dev-sprint-reports] module.service.aws_ecs_service.app: Refreshing state... [id=arn:aws:ecs:us-east-1:315341936575:service/analytics-dev/analytics-dev] data.aws_iam_policy.migrator_db_access_policy[0]: Read complete after 2s [id=arn:aws:iam::315341936575:policy/analytics-dev-migrator-access] module.service.aws_iam_role_policy_attachment.migrator_db_access[0]: Refreshing state... [id=analytics-dev-migrator-20240328214813227300000004] data.aws_iam_policy.app_db_access_policy[0]: Read complete after 2s [id=arn:aws:iam::315341936575:policy/analytics-dev-app-access] module.service.aws_iam_role_policy_attachment.app_service_db_access[0]: Refreshing state... [id=analytics-dev-app-20240328214813228600000005] aws_scheduler_schedule.sprint_reports: Refreshing state... [id=analytics-dev-sprint-reports/analytics-dev-sprint-reports] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place -/+ destroy and then create replacement Terraform will perform the following actions: # aws_sfn_state_machine.sprint_reports will be updated in-place ~ resource "aws_sfn_state_machine" "sprint_reports" { ~ definition = jsonencode( { - StartAt = "ExecuteECSTask" - States = { - ExecuteECSTask = { - End = true - Parameters = { - Cluster = "arn:aws:ecs:us-east-1:315341936575:cluster/analytics-dev" - LaunchType = "FARGATE" - NetworkConfiguration = { - AwsvpcConfiguration = { - SecurityGroups = [ - "sg-0ed2e3fe9a482683c", ] - Subnets = [ - "subnet-0a5ea667d3751639f", - "subnet-068ede7dcfd9469ab", - "subnet-019f469ba97dc6ec7", ] } } - Overrides = { - ContainerOverrides = [ - { - Command = [ - "make", - "gh-data-export", - "sprint-reports", ] - Environment = [ - { - Name = "PY_RUN_APPROACH" - Value = "local" }, - { - Name = "SPRINT_FILE" - Value = "/tmp/sprint-data.json" }, - { - Name = "ISSUE_FILE" - Value = "/tmp/issue-data.json" }, - { - Name = "OUTPUT_DIR" - Value = "/tmp/" }, ] - Name = "analytics-dev" }, ] } - TaskDefinition = "arn:aws:ecs:us-east-1:315341936575:task-definition/analytics-dev:34" } - Resource = "arn:aws:states:::ecs:runTask.sync" - Type = "Task" } } } ) -> (known after apply) id = "arn:aws:states:us-east-1:315341936575:stateMachine:analytics-dev-sprint-reports" name = "analytics-dev-sprint-reports" tags = {} # (11 unchanged attributes hidden) # (3 unchanged blocks hidden) } # module.service.aws_ecs_service.app will be updated in-place ~ resource "aws_ecs_service" "app" { id = "arn:aws:ecs:us-east-1:315341936575:service/analytics-dev/analytics-dev" name = "analytics-dev" tags = {} ~ task_definition = "arn:aws:ecs:us-east-1:315341936575:task-definition/analytics-dev:34" -> (known after apply) # (15 unchanged attributes hidden) # (3 unchanged blocks hidden) } # module.service.aws_ecs_task_definition.app must be replaced -/+ resource "aws_ecs_task_definition" "app" { ~ arn = "arn:aws:ecs:us-east-1:315341936575:task-definition/analytics-dev:34" -> (known after apply) ~ arn_without_revision = "arn:aws:ecs:us-east-1:315341936575:task-definition/analytics-dev" -> (known after apply) ~ container_definitions = jsonencode( ~ [ ~ { ~ linuxParameters = { ~ capabilities = { - add = [] # (1 unchanged attribute hidden) } # (1 unchanged attribute hidden) } - mountPoints = [] name = "analytics-dev" ~ portMappings = [ ~ { - hostPort = 8000 - protocol = "tcp" # (1 unchanged attribute hidden) }, ] ~ secrets = [ ~ { name = "ANALYTICS_REPORTING_CHANNEL_ID" ~ valueFrom = "/analytics/dev/reporting-channel-id" -> "arn:aws:ssm:us-east-1:315341936575:parameter/analytics/dev/reporting-channel-id" }, ~ { name = "ANALYTICS_SLACK_BOT_TOKEN" ~ valueFrom = "/analytics/dev/slack-bot-token" -> "arn:aws:ssm:us-east-1:315341936575:parameter/analytics/slack-bot-token" }, ~ { name = "GH_TOKEN" ~ valueFrom = "/analytics/dev/github-token" -> "arn:aws:ssm:us-east-1:315341936575:parameter/analytics/github-token" }, ] - systemControls = [] - volumesFrom = [] # (8 unchanged attributes hidden) }, ] # forces replacement ) ~ id = "analytics-dev" -> (known after apply) ~ revision = 34 -> (known after apply) - tags = {} -> null # (12 unchanged attributes hidden) } # module.service.aws_iam_role_policy.task_executor will be updated in-place ~ resource "aws_iam_role_policy" "task_executor" { id = "analytics-dev-task-executor:analytics-dev-task-executor-role-policy" name = "analytics-dev-task-executor-role-policy" ~ policy = jsonencode( ~ { ~ Statement = [ # (8 unchanged elements hidden) { Action = [ "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability", ] Effect = "Allow" Resource = "arn:aws:ecr:us-east-1:315341936575:repository/simpler-grants-gov-analytics" Sid = "ECRPullAccess" }, ~ { ~ Resource = [ ~ "arn:aws:ssm:*:*:parameter/analytics/dev/slack-bot-token" -> "arn:aws:ssm:us-east-1:315341936575:parameter/analytics/slack-bot-token", ~ "arn:aws:ssm:*:*:parameter/analytics/dev/reporting-channel-id" -> "arn:aws:ssm:us-east-1:315341936575:parameter/analytics/github-token", ~ "arn:aws:ssm:*:*:parameter/analytics/dev/github-token" -> "arn:aws:ssm:us-east-1:315341936575:parameter/analytics/dev/reporting-channel-id", ] # (3 unchanged attributes hidden) }, ] # (1 unchanged attribute hidden) } ) # (2 unchanged attributes hidden) } Plan: 1 to add, 3 to change, 1 to destroy. ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now. ``` </details>

coilysiren added 7 commits April 19, 2024 15:30

opensearch spike

668d7d0

checkpoint-kai-1713808423

7419e02

docs

82ff894

checkpoint-kai-1713818731

80b3531

cmk

b982f46

remove docs

a89af33

Merge branch 'main' into kai/opensearch

7b5d055

github-actions bot added terraform infra labels Oct 2, 2024

coilysiren added 6 commits October 2, 2024 12:19

add lockfile

1b53aa0

fix username password stuff

eeb1cdb

resize

0b0c29d

move into a module

d9209cf

remove comments

2879634

password docs

3db0a7e

coilysiren changed the title ~~Kai/opensearch~~ Deploy Opensearch Oct 3, 2024

coilysiren mentioned this pull request Oct 3, 2024

Updates checkov #2364

Merged

coilysiren and others added 3 commits October 3, 2024 15:53

appease checkov

ccc7ec0

more details in comment

9944768

Merge branch 'main' into kai/opensearch

2ee4e7b

coilysiren marked this pull request as ready for review October 3, 2024 23:02

coilysiren requested review from chouinar and acouch October 3, 2024 23:02

coilysiren marked this pull request as draft October 3, 2024 23:15

chouinar reviewed Oct 4, 2024

View reviewed changes

infra/modules/opensearch/main.tf Outdated Show resolved Hide resolved

mxk0 added topic: infra Infrastructure related tickets and removed infra labels Oct 6, 2024

bump version

13f5fd8

github-actions bot added the infra label Oct 7, 2024

coilysiren added 10 commits October 9, 2024 16:11

Merge branch 'main' into kai/opensearch

9fdcccd

secrets refactor

3f84d01

remove plan

a382cee

fix horrible mistake

3157982

security enhancements

885a74a

appease checkov

e0b587d

extreme KMS

c162a3f

rename stuff

dbd0e06

workon checkov

bd98fc3

workon checkov

bc5aea0

github-actions bot added the ci/cd label Oct 10, 2024

coilysiren mentioned this pull request Oct 10, 2024

CKV2_AWS_5 fails to identify opensearch attachment bridgecrewio/checkov#6760

Open

workon kms

303dfe6

coilysiren marked this pull request as ready for review October 10, 2024 09:11

coilysiren requested a review from mdragon as a code owner October 10, 2024 09:11

coilysiren requested a review from chouinar October 10, 2024 09:11

mdragon previously approved these changes Oct 10, 2024

View reviewed changes

chouinar reviewed Oct 10, 2024

View reviewed changes

fix env vars

bb78f2e

coilysiren dismissed mdragon’s stale review via bb78f2e October 10, 2024 16:01

coilysiren requested a review from chouinar October 10, 2024 16:15

chouinar approved these changes Oct 10, 2024

View reviewed changes

coilysiren merged commit 57013e6 into main Oct 10, 2024
7 checks passed

coilysiren deleted the kai/opensearch branch October 10, 2024 17:44

coilysiren mentioned this pull request Oct 10, 2024

fix opensearch env vars #2451

Merged

coilysiren added a commit that referenced this pull request Oct 10, 2024

fix opensearch env vars (#2451)

43cceb4

fast-follow to #2341

This was referenced Oct 16, 2024

Frontend deploys broken Wednesday Oct 16th 2024 #2485

Closed

[Issue #2485] Swap everything to new secrets pattern #2486

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploy Opensearch #2341

Deploy Opensearch #2341

coilysiren commented Oct 2, 2024 •

edited

Loading

coilysiren commented Oct 10, 2024

mdragon left a comment

chouinar left a comment

chouinar Oct 10, 2024

coilysiren Oct 10, 2024

chouinar Oct 10, 2024

coilysiren Oct 10, 2024

coilysiren Oct 10, 2024

Deploy Opensearch #2341

Deploy Opensearch #2341

Conversation

coilysiren commented Oct 2, 2024 • edited Loading

Summary

Time to review: 10 ~ 30 mins

Changes proposed

Context for reviewers

Additional information

coilysiren commented Oct 10, 2024

mdragon left a comment

Choose a reason for hiding this comment

chouinar left a comment

Choose a reason for hiding this comment

chouinar Oct 10, 2024

Choose a reason for hiding this comment

coilysiren Oct 10, 2024

Choose a reason for hiding this comment

chouinar Oct 10, 2024

Choose a reason for hiding this comment

coilysiren Oct 10, 2024

Choose a reason for hiding this comment

coilysiren Oct 10, 2024

Choose a reason for hiding this comment

coilysiren commented Oct 2, 2024 •

edited

Loading