App level metrics monitoring with Prometheus #186

LDiazN · 2025-02-17T12:36:35Z

This PR adds support for scraping application level metrics straight from the ECS nodes, without going through the load balancer, solving the flapping behavior described in #104

To achieve this we had to solve the following problems:

Reach the cluster nodes in AWS, this was solved by Prometheus ec2 monitoring #182
Discover ECS tasks with their corresponding port and IP address. This was a bit more tricky, we solved it by adding a cronjob that runs a python script that will request ECS data from AWS using the boto3 client, and then storing that information into a Prometheus file based discovery compatible file

So, this PR will add:

A Python script to collect ECS tasks information from aws and writing it into a file
An Ansible configuration for creating and running this file with a cronjob
An update to the Nginx configuration used to proxy metrics requests from the monitoring host to the EC2 instances in AWS
Security groups configuration to allow traffic from the proxy host to the ECS cluster nodes
IAM credentials used for requesting tasks information from AWS

This PR solves ooni/backend#937 and ooni/backend#938 and is related to #104

…ration

…s proxy

github-actions · 2025-02-17T12:37:32Z

Terraform Run Output 🤖

Format and Style 🖌`success`

Initialization ⚙️`success`

Validation 🤖`success`

Validation Output


$ terraform validate
Success! The configuration is valid.

Plan 📖`success`

Plan: 0 to add, 2 to change, 0 to destroy.

Show Plan


$ terraform plan
data.dns_a_record_set.monitoring_host: Reading...
module.ooni_clickhouse_proxy.data.cloudinit_config.ooni_ec2: Reading...
random_id.artifact_id: Refreshing state... [id=8Ujqew]
module.adm_iam_roles.tls_private_key.oonidevops: Refreshing state... [id=b49a9fdb9f720320340226016efe24808dd68203]
random_password.prometheus_metrics_password: Refreshing state... [id=none]
module.ansible_inventory.local_file.ansible_inventory: Refreshing state... [id=b6de844ed8d384f890fa6f467502390de843f758]
module.ooni_clickhouse_proxy.data.cloudinit_config.ooni_ec2: Read complete after 0s [id=2022394177]
module.ansible_inventory.null_resource.ansible_update_known_hosts: Refreshing state... [id=236461505953331670]
data.dns_a_record_set.monitoring_host: Read complete after 0s [id=monitoring.ooni.org]
data.aws_ssm_parameter.clickhouse_readonly_url: Reading...
module.adm_iam_roles.aws_key_pair.oonidevops: Refreshing state... [id=oonidevops]
module.ooniapi_oonifindings.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonifindings-task-role]
module.ooniapi_oonifindings_deployer.data.aws_caller_identity.current: Reading...
aws_s3_bucket.oonith_codepipeline_bucket: Refreshing state... [id=codepipeline-oonith-eu-central-1-f148ea7b]
module.ooniapi_oonimeasurements_deployer.data.aws_caller_identity.current: Reading...
module.oonidevops_github_user.aws_secretsmanager_secret.oonidevops_github: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/github_user/access_key_json-9JTJgd]
module.ooniapi_user.aws_secretsmanager_secret.aws_access_key_id: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_access_key_id-EcXOBx]
module.ooniapi_ooniauth.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniauth-task-role]
module.ooniapi_user.aws_secretsmanager_secret.aws_secret_access_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_secret_access_key-L0DQDr]
module.ooniapi_oonimeasurements_deployer.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.ooniapi_reverseproxy_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-reverseproxy]
module.ooniapi_oonifindings_deployer.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.ooniapi_oonimeasurements.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-oonimeasurements]
aws_acm_certificate.ooniapi_frontend: Refreshing state... [id=arn:aws:acm:eu-central-1:905418398257:certificate/190205f1-392d-425c-a059-7006ca8c8c46]
module.ooniapi_ooniprobe_deployer.data.aws_caller_identity.current: Reading...
module.ooniapi_user.aws_ses_email_identity.ooniapi: Refreshing state... [[email protected]]
module.ooniapi_ooniprobe_deployer.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.ooniapi_reverseproxy.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-reverseproxy-task-role]
module.ooniapi_user.aws_iam_user.ooniapi: Refreshing state... [id=oonidevops-ooniapi]
module.ooniapi_reverseproxy.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-reverseproxy]
module.ooniapi_cluster.aws_cloudwatch_log_group.ooniapi_services: Refreshing state... [id=ooni-ecs-group/ooniapi-ecs-cluster]
data.aws_ssm_parameter.clickhouse_readonly_url: Read complete after 0s [id=/oonidevops/secrets/clickhouse_readonly_url]
module.ooniapi_ooniauth.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
module.oonidevops_github_user.aws_iam_user.oonidevops_github: Refreshing state... [id=oonidevops-github]
module.ooniapi_reverseproxy.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
module.ooniapi_oonirun.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonirun-task-role]
module.ooniapi_ooniprobe.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
module.ooniapi_reverseproxy_deployer.data.aws_caller_identity.current: Reading...
module.oonidevops_github_user.aws_iam_policy.oonidevops_github: Refreshing state... [id=arn:aws:iam::905418398257:policy/oonidevops-github-policy]
module.ooniapi_reverseproxy_deployer.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.adm_iam_roles.data.aws_iam_policy_document.assume_role: Reading...
module.adm_iam_roles.data.aws_iam_policy_document.assume_role: Read complete after 0s [id=367960279]
module.ooniapi_oonirun.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
module.ooniapi_ooniprobe.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-ooniprobe]
module.ooniapi_cluster.data.aws_ssm_parameter.ecs_optimized_ami: Reading...
module.ooni_monitoring.aws_iam_user.ooni_monitoring: Refreshing state... [id=oonidevops-monitoring]
module.ooniapi_oonirun_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-oonirun]
module.ooniapi_ooniauth.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 1s [id=arn:aws:ecs:eu-central-1:905418398257:task-definition/ooniapi-service-ooniauth-td:98/ooniapi-service-ooniauth]
module.ooniapi_oonimeasurements_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-oonimeasurements]
module.ooniapi_reverseproxy.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 1s [id=arn:aws:ecs:eu-central-1:905418398257:task-definition/ooniapi-service-reverseproxy-td:33/ooniapi-service-reverseproxy]
module.ooniapi_oonifindings.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
aws_s3_bucket.ooniapi_codepipeline_bucket: Refreshing state... [id=codepipeline-ooniapi-eu-central-1-f148ea7b]
module.ooniapi_cluster.data.aws_ssm_parameter.ecs_optimized_ami: Read complete after 0s [id=/aws/service/ecs/optimized-ami/amazon-linux-2/recommended]
module.ooniapi_ooniauth_deployer.data.aws_caller_identity.current: Reading...
module.ooniapi_ooniprobe.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 1s [id=arn:aws:ecs:eu-central-1:905418398257:task-definition/ooniapi-service-ooniprobe-td:90/ooniapi-service-ooniprobe]
module.ooniapi_oonimeasurements.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonimeasurements-task-role]
data.aws_availability_zones.available: Reading...
module.ooniapi_ooniprobe.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniprobe-task-role]
data.aws_ssm_parameter.jwt_secret: Reading...
module.ooniapi_oonirun.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 1s [id=arn:aws:ecs:eu-central-1:905418398257:task-definition/ooniapi-service-oonirun-td:93/ooniapi-service-oonirun]
module.ooni_clickhouse_proxy.data.aws_ssm_parameter.ubuntu_22_ami: Reading...
module.ooniapi_oonifindings.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 0s [id=arn:aws:ecs:eu-central-1:905418398257:task-definition/ooniapi-service-oonifindings-td:42/ooniapi-service-oonifindings]
module.ooniapi_oonifindings_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-oonifindings]
module.ooniapi_ooniauth_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-ooniauth]
module.ooniapi_ooniauth_deployer.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.ooniapi_ooniauth.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-ooniauth]
data.aws_availability_zones.available: Read complete after 0s [id=eu-central-1]
data.aws_ssm_parameter.do_token: Reading...
module.adm_iam_roles.aws_iam_policy.oonidevops: Refreshing state... [id=arn:aws:iam::905418398257:policy/OONIDevopsPolicy]
module.ooniapi_oonirun.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-oonirun]
data.aws_ssm_parameter.jwt_secret: Read complete after 0s [id=/oonidevops/secrets/ooni_services/jwt_secret]
aws_secretsmanager_secret.oonipg_url: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni-tier0-postgres/postgresql_url-w62CTZ]
data.aws_ssm_parameter.oonipg_url: Reading...
module.adm_iam_roles.aws_secretsmanager_secret.oonidevops_deploy_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/deploy_key-2ebqSe]
module.ooniapi_oonifindings.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-oonifindings]
module.ooni_clickhouse_proxy.data.aws_ssm_parameter.ubuntu_22_ami: Read complete after 0s [id=/aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id]
aws_secretsmanager_secret.prometheus_metrics_password: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni_services/prometheus_metrics_password-M8BbRw]
module.ooniapi_cluster.aws_iam_role.container_host: Refreshing state... [id=ooniapi-ecs-cluster-container-host-role]
module.ooniapi_ooniprobe_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-ooniprobe]
module.ooniapi_oonirun_deployer.data.aws_caller_identity.current: Reading...
module.ooniapi_reverseproxy_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-reverseproxy]
data.aws_ssm_parameter.do_token: Read complete after 0s [id=/oonidevops/secrets/digitalocean_access_token]
module.ooniapi_oonifindings.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonifindings-task-role:ooniapi-service-oonifindings-task-role]
module.ooniapi_ooniauth.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniauth-task-role:ooniapi-service-ooniauth-task-role]
module.ooniapi_user.aws_iam_access_key.ooniapi: Refreshing state... [id=AKIA5FTZELIYSK2XEVOT]
module.ooniapi_user.aws_iam_user_policy.ooniapi: Refreshing state... [id=oonidevops-ooniapi:oonidevops-ooniapi-policy]
module.ooniapi_oonirun_deployer.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.ooniapi_reverseproxy.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-reverseproxy-task-role:ooniapi-service-reverseproxy-task-role]
module.oonidevops_github_user.aws_iam_access_key.oonidevops_github: Refreshing state... [id=AKIA5FTZELIYXDN55SMS]
module.ooniapi_oonirun.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonirun-task-role:ooniapi-service-oonirun-task-role]
module.oonidevops_github_user.aws_iam_user_policy_attachment.oonidevops_github: Refreshing state... [id=oonidevops-github-20240313195612421500000001]
aws_route53_record.ooniapi_frontend_cert_validation["ooniauth.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__48cd4e71cee9930614228176b7deefb9.ooniauth.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["ooniprobe.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__a064be8aa084a037ff9fa5e3e541c87d.ooniprobe.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["oonirun.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__05c891caeb4509d4cd7f9c24d8b6dbd0.oonirun.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["api.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__cd4729fc0c282e771d056e719a7bdf4f.api.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["8.th.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__ef17825e5fd9713f596344bdd9626f5e.8.th.dev.ooni.io._CNAME]
module.ooni_monitoring.aws_iam_access_key.ooni_monitoring: Refreshing state... [id=AKIA5FTZELIYWULOT65S]
module.ooni_monitoring.aws_iam_user_policy.ooni_monitoring: Refreshing state... [id=oonidevops-monitoring:oonidevops-monitoring-policy]
data.aws_ssm_parameter.oonipg_url: Read complete after 0s [id=/oonidevops/secrets/ooni-tier0-postgres/postgresql_write_url]
module.ooniapi_oonirun_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-oonirun]
module.ooniapi_cluster.aws_ecs_cluster.main: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:cluster/ooniapi-ecs-cluster]
module.ooniapi_oonimeasurements_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-oonimeasurements]
module.ooniapi_oonimeasurements.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonimeasurements-task-role:ooniapi-service-oonimeasurements-task-role]
module.ooniapi_oonifindings_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-oonifindings]
module.ooniapi_ooniprobe.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniprobe-task-role:ooniapi-service-ooniprobe-task-role]
module.ooniapi_ooniauth_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-ooniauth]
module.adm_iam_roles.aws_iam_role.oonidevops: Refreshing state... [id=oonidevops]
module.adm_iam_roles.aws_secretsmanager_secret_version.oonidevops_deploy_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/deploy_key-2ebqSe|terraform-20240925140131946100000002]
module.ooniapi_cluster.aws_iam_role_policy.container_host: Refreshing state... [id=ooniapi-ecs-cluster-container-host-role:ooniapi-ecs-cluster-instance-role-policy]
module.ooniapi_cluster.aws_iam_instance_profile.container_host: Refreshing state... [id=ooniapi-ecs-cluster]
module.ooniapi_ooniprobe_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-ooniprobe]
data.aws_secretsmanager_secret_version.prometheus_metrics_password: Reading...
aws_secretsmanager_secret_version.prometheus_metrics_password: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni_services/prometheus_metrics_password-M8BbRw|terraform-20240314200140936700000008]
module.oonidevops_github_user.aws_secretsmanager_secret_version.oonidevops_github: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/github_user/access_key_json-9JTJgd|terraform-20240519071250187000000004]
module.ooniapi_user.aws_secretsmanager_secret_version.aws_secret_access_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_secret_access_key-L0DQDr|terraform-20240314200140914600000006]
module.ooniapi_user.aws_secretsmanager_secret_version.aws_access_key_id: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_access_key_id-EcXOBx|terraform-20240314200140918400000007]
module.ooni_monitoring.aws_ssm_parameter.ooni_monitoring_access_key: Refreshing state... [id=/oonidevops/secrets/ooni_monitoring/access_key]
module.ooni_monitoring.aws_ssm_parameter.ooni_monitoring_secret_key: Refreshing state... [id=/oonidevops/secrets/ooni_monitoring/secret_key]
aws_acm_certificate_validation.ooniapi_frontend: Refreshing state... [id=0001-01-01 00:00:00 +0000 UTC]
aws_codestarconnections_connection.oonidevops: Refreshing state... [id=arn:aws:codestar-connections:eu-central-1:905418398257:connection/6bd492f6-c11d-43ec-92b0-24c47700d528]
data.aws_secretsmanager_secret_version.prometheus_metrics_password: Read complete after 0s [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni_services/prometheus_metrics_password-M8BbRw|AWSCURRENT]
data.aws_secretsmanager_secret_version.deploy_key: Reading...
module.terraform_state_backend.data.aws_region.current: Reading...
module.terraform_state_backend.data.aws_region.current: Read complete after 0s [id=eu-central-1]
module.network.aws_vpc.main: Refreshing state... [id=vpc-0e382f3ad89286de9]
module.ooniapi_oonimeasurements.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-oonimeasurements-td]
module.ooniapi_reverseproxy.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-reverseproxy-td]
module.ooniapi_ooniprobe.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-ooniprobe-td]
data.aws_secretsmanager_secret_version.deploy_key: Read complete after 1s [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/deploy_key-2ebqSe|AWSCURRENT]
module.ooniapi_oonirun.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-oonirun-td]
module.ooniapi_oonifindings.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-oonifindings-td]
module.ooniapi_ooniauth.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-ooniauth-td]
module.ooni_th_droplet.data.cloudinit_config.ooni_th_docker: Reading...
module.ooni_th_droplet.data.cloudinit_config.ooni_th_docker: Read complete after 0s [id=1719060339]
module.ooni_th_droplet.digitalocean_droplet.ooni_th_docker[0]: Refreshing state... [id=459912318]
module.terraform_state_backend.aws_s3_bucket.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Reading...
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Read complete after 0s [id=2666303363]
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=oonidevops-dev-terraform-state-lock]
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Reading...
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Read complete after 0s [id=2666303363]
module.ooniapi_ooniauth_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-ooniauth-eu-central-1]
module.ooniapi_oonimeasurements_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-oonimeasurements-eu-central-1]
module.ooniapi_ooniprobe_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-ooniprobe-eu-central-1]
module.ooniapi_oonifindings_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-oonifindings-eu-central-1]
module.ooniapi_reverseproxy_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-reverseproxy-eu-central-1]
module.ooniapi_oonirun_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-oonirun-eu-central-1]
module.ooniapi_reverseproxy_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-reverseproxy]
module.ooniapi_oonimeasurements_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-oonimeasurements]
module.ooniapi_oonifindings_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-oonifindings]
module.ooniapi_ooniauth_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-ooniauth]
module.ooniapi_ooniprobe_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-ooniprobe]
module.ooniapi_oonirun_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-oonirun]
module.ooniapi_reverseproxy_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-reverseproxy]
module.ooniapi_oonifindings_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-oonifindings]
module.ooniapi_ooniauth_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-ooniauth]
module.ooniapi_oonimeasurements_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-oonimeasurements]
module.ooniapi_ooniprobe_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-ooniprobe]
module.ooniapi_oonirun_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-oonirun]
module.ooni_th_droplet.aws_route53_record.ooni_th["0"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_0.do.th.dev.ooni.io_A]
module.network.aws_internet_gateway.gw: Refreshing state... [id=igw-0c080e9b235ed29d1]
module.ooniapi_ooniprobe.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OproM-20250115122624346700000001/9f9264a4e53931d3]
module.ooniapi_reverseproxy.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OrevM-20250115122624347000000002/32c2f9b4e4d3b8c4]
module.ooniapi_ooniauth.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OautM-20250115122624347200000004/6e746a968782a49f]
module.ooni_clickhouse_proxy.aws_alb_target_group.ooni_ec2: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/oockpr20250116192249626700000002/2e9dada4dd22c268]
module.ooniapi_oonifindings.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OfinM-20250115122624350600000005/ad715c6e26dd616c]
module.ooniapi_cluster.aws_security_group.web: Refreshing state... [id=sg-0187eedfe39538357]
module.ooniapi_oonimeasurements.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OmeaM-20250116160254864500000001/4d88cb32eb2f381c]
module.ooni_clickhouse_proxy.aws_security_group.ec2_sg: Refreshing state... [id=sg-0903c108a44c922a5]
module.ooniapi_oonirun.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OrunM-20250115122624347100000003/17e1664b99b708a5]
module.network.aws_route_table.public: Refreshing state... [id=rtb-0ccb0852e6a365a95]
module.network.aws_subnet.private[0]: Refreshing state... [id=subnet-09314a43ec89d6331]
module.network.aws_subnet.private[1]: Refreshing state... [id=subnet-0b899a7ad10406d06]
module.network.aws_route_table.private: Refreshing state... [id=rtb-011463437da96c77b]
module.network.aws_subnet.public[0]: Refreshing state... [id=subnet-0e7a4478be988463f]
module.network.aws_subnet.public[1]: Refreshing state... [id=subnet-0b18966cccfc9d5ef]
module.terraform_state_backend.aws_s3_bucket_server_side_encryption_configuration.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.terraform_state_backend.aws_s3_bucket_versioning.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.terraform_state_backend.aws_s3_bucket_public_access_block.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_egress[0]: Refreshing state... [id=sgrule-1099643652]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_egress[1]: Refreshing state... [id=sgrule-1281654482]
module.network.aws_route_table_association.private[0]: Refreshing state... [id=rtbassoc-0e7933e6b804ff2c1]
module.network.aws_route_table_association.private[1]: Refreshing state... [id=rtbassoc-0c9cc0f117ef15fe7]
module.ooniapi_cluster.aws_security_group.container_host: Refreshing state... [id=sg-0aa6a97400b619de3]
module.oonipg.aws_security_group.pg: Refreshing state... [id=sg-005ca579eb9c08cda]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_ingress[1]: Refreshing state... [id=sgrule-3288936075]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_ingress[3]: Refreshing state... [id=sgrule-3953292375]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_ingress[2]: Refreshing state... [id=sgrule-3890696530]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_ingress[0]: Refreshing state... [id=sgrule-1921217342]
module.network.aws_route_table_association.public[1]: Refreshing state... [id=rtbassoc-08ab18165bf481054]
module.network.aws_route_table_association.public[0]: Refreshing state... [id=rtbassoc-0dbd7fb16801ee049]
module.terraform_state_backend.aws_s3_bucket_policy.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.ooniapi_cluster.aws_launch_template.container_host: Refreshing state... [id=lt-0e328a8671f870c64]
module.oonipg.aws_db_subnet_group.pg: Refreshing state... [id=ooni-tier0-postgres-dbsng]
module.ooni_clickhouse_proxy.aws_launch_template.ooni_ec2: Refreshing state... [id=lt-0855bc6373ff4c75b]
module.ooniapi_frontend.aws_alb.ooniapi: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:loadbalancer/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6]
module.terraform_state_backend.time_sleep.wait_for_aws_s3_bucket_settings[0]: Refreshing state... [id=2024-03-10T15:06:17Z]
module.terraform_state_backend.aws_s3_bucket_ownership_controls.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.ooniapi_cluster.aws_autoscaling_group.container_host: Refreshing state... [id=ooniapi-ecs-cluster20240310192644083800000003]
module.ooni_clickhouse_proxy.aws_instance.ooni_ec2: Refreshing state... [id=i-01f355c588856e904]
module.ooniapi_frontend.aws_alb_listener.ooniapi_listener_https: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd]
module.ooniapi_frontend.aws_alb_listener.ooniapi_listener_http: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/d9b2448464179cd1]
aws_route53_record.ooniapi_frontend_alt["ooniauth.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_ooniauth.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["8.th.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_8.th.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["ooniprobe.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_ooniprobe.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_main: Refreshing state... [id=Z055356431RGCLK3JXZDL_api.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["oonirun.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_oonirun.dev.ooni.io_A]
module.oonipg.aws_db_instance.pg: Refreshing state... [id=db-27N7Q6XIBNASFCOXN4N7C762L4]
module.ooniapi_frontend.aws_alb_listener_rule.ooniapi_th: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/01ee7503374be8ff]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniprobe_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/b436b91883ae7c86]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniprobe_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/2b09ed268181ba4f]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniauth_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/128c53ea760208fc]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonifindings_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/5a872da4cc6b9135]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniauth_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/cefeff0d8aa3118a]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonifindings_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/cdc4e8e8eabb56f2]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonirun_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/65e6f5e3aca0a4e5]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonirun_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/551c4128bb282fa4]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonimeasurements_rule_1[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/cd947c873f9a1408]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonimeasurements_rule_host[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/f3142766eb744476]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonimeasurements_rule_2[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/49f4e3329b9b4665]
aws_route53_record.postgres_dns: Refreshing state... [id=Z091407123AEJO90Z3H6D_postgres.dev.ooni.nu_CNAME]
data.aws_secretsmanager_secret_version.pg_login: Reading...
data.aws_secretsmanager_secret_version.pg_login: Read complete after 0s [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:rds!db-5fe27151-3a37-44e0-a5bd-3517363fa2e8-BDI0KI|AWSCURRENT]
aws_secretsmanager_secret_version.oonipg_url: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni-tier0-postgres/postgresql_url-w62CTZ|terraform-20250214100503095200000001]
module.ooni_clickhouse_proxy.aws_lb_target_group_attachment.oonibackend_proxy: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/oockpr20250116192249626700000002/2e9dada4dd22c268-20250116192309877300000006]
aws_route53_record.clickhouse_proxy_alias: Refreshing state... [id=Z055356431RGCLK3JXZDL_clickhouseproxy.dev.ooni.io_CNAME]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform planned the following actions, but then encountered a problem:

  # module.ooni_clickhouse_proxy.aws_instance.ooni_ec2 will be updated in-place
  ~ resource "aws_instance" "ooni_ec2" {
        id                                   = "i-01f355c588856e904"
        tags                                 = {
            "Environment" = "dev"
            "Name"        = "ooni-tier0-clickhouseproxy"
            "Repository"  = "https://github.com/ooni/devops"
        }
      ~ user_data                            = "bd31c4f073ccfc1109babe7ffeb3ecc1983bd61a" -> "93eac876eb1214179891aaf309d7d19fd41e0c7f"
        # (39 unchanged attributes hidden)

        # (9 unchanged blocks hidden)
    }

  # module.ooni_clickhouse_proxy.aws_launch_template.ooni_ec2 will be updated in-place
  ~ resource "aws_launch_template" "ooni_ec2" {
        id                                   = "lt-0855bc6373ff4c75b"
      ~ image_id                             = (sensitive value)
      ~ latest_version                       = 1 -> (known after apply)
        name                                 = "oonickprx-tmpl-20250116192253018000000003"
        tags                                 = {}
        # (16 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

Warning: Failed to decode resource from state

Error decoding
"module.ooniapi_oonifindings_deployer.aws_codepipeline.ooniapi" from prior
state: unsupported attribute "timeout_in_minutes"

Warning: Failed to decode resource from state

Error decoding "module.ooniapi_ooniprobe.aws_ecs_service.ooniapi_service"
from prior state: unsupported attribute "availability_zone_rebalancing"

Warning: Failed to decode resource from state

Error decoding "module.ooniapi_ooniauth_deployer.aws_codepipeline.ooniapi"
from prior state: unsupported attribute "timeout_in_minutes"

Warning: Failed to decode resource from state

Error decoding "module.ooniapi_oonirun.aws_ecs_service.ooniapi_service" from
prior state: unsupported attribute "availability_zone_rebalancing"

Warning: Failed to decode resource from state

Error decoding "module.ooniapi_ooniprobe_deployer.aws_codepipeline.ooniapi"
from prior state: unsupported attribute "timeout_in_minutes"

Warning: Failed to decode resource from state

Error decoding "module.ooniapi_oonifindings.aws_ecs_service.ooniapi_service"
from prior state: unsupported attribute "availability_zone_rebalancing"

Warning: Failed to decode resource from state

Error decoding
"module.ooniapi_oonimeasurements_deployer.aws_codepipeline.ooniapi" from
prior state: unsupported attribute "timeout_in_minutes"

Warning: Failed to decode resource from state

Error decoding "module.ooniapi_oonirun_deployer.aws_codepipeline.ooniapi"
from prior state: unsupported attribute "timeout_in_minutes"

Warning: Failed to decode resource from state

Error decoding
"module.ooniapi_reverseproxy_deployer.aws_codepipeline.ooniapi" from prior
state: unsupported attribute "timeout_in_minutes"

Warning: Failed to decode resource from state

Error decoding "module.ooniapi_ooniauth.aws_ecs_service.ooniapi_service" from
prior state: unsupported attribute "availability_zone_rebalancing"

Warning: Failed to decode resource from state

Error decoding
"module.ooniapi_oonimeasurements.aws_ecs_service.ooniapi_service" from prior
state: unsupported attribute "availability_zone_rebalancing"

Warning: Failed to decode resource from state

Error decoding "module.ooniapi_reverseproxy.aws_ecs_service.ooniapi_service"
from prior state: unsupported attribute "availability_zone_rebalancing"


Pusher	@LDiazN
Action	pull_request
Environment	dev
Workflow	.github/workflows/check_terraform.yml
Last updated	Tue, 18 Feb 2025 09:40:44 GMT

github-actions · 2025-02-17T12:38:08Z

Ansible Run Output 🤖

Ansible Playbook Recap 🔍

Ansible playbook output 📖`success`

Show Execution


$ ansible-playbook playbook.yml --check --diff -i ../tf/modules/ansible_inventory/inventories/inventory-dev.ini
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
[WARNING]: Could not match supplied host pattern, ignoring: monitoring.ooni.org
[WARNING]: Could not match supplied host pattern, ignoring: backend-
hel.ooni.org
[WARNING]: Could not match supplied host pattern, ignoring:
clickhouseproxy.dev.ooni.io
[WARNING]: Could not match supplied host pattern, ignoring: notebook1.htz-
fsn.prod.ooni.nu
[WARNING]: Could not match supplied host pattern, ignoring: data1.htz-
fsn.prod.ooni.nu
[WARNING]: Could not match supplied host pattern, ignoring: data3.htz-
fsn.prod.ooni.nu
[WARNING]: Could not match supplied host pattern, ignoring: openvpn-
server1.ooni.io

PLAY [Ensure all hosts are bootstrapped correctly] *****************************
skipping: no hosts matched

PLAY [Deploy monitoring host] **************************************************
skipping: no hosts matched

PLAY [Update monitoring config] ************************************************
skipping: no hosts matched

PLAY [Deploy ooni backend services] ********************************************
skipping: no hosts matched

PLAY [Deploy clickhouse proxy] *************************************************
skipping: no hosts matched

PLAY [Deploy oonidata clickhouse hosts] ****************************************
skipping: no hosts matched

PLAY [Deploy airflow frontend host] ********************************************
skipping: no hosts matched

PLAY [Setup OpenVPN server] ****************************************************
skipping: no hosts matched

PLAY [Deploy notebook host] ****************************************************
skipping: no hosts matched

PLAY RECAP *********************************************************************


Pusher	@LDiazN
Action	pull_request
Working Directory
Workflow	.github/workflows/check_ansible.yml
Last updated	Tue, 18 Feb 2025 09:40:55 GMT

ansible/roles/monitoring/defaults/main.yml

ansible/roles/monitoring/tasks/main.yml

ansible/roles/monitoring/templates/ecs-discovery.py

ansible/roles/prometheus/templates/prometheus.yml

hellais

Lgtm. Let's document the two things as future work and move forward with merging

LDiazN added 17 commits February 13, 2025 16:34

Add directory for prometheus file based discovery and initial configu…

29eb218

…ration

Add initial configuration to redirect ecs tasks scraping to prometheu…

b6f9718

…s proxy

Add ECS scraping script

cb728c3

Moved discovery script to monitoring role

f224c71

Add ecs-discovery script file to the server

a315b7a

merge conflict

86882c6

Add port to path parameters for proxy pass

91f52da

Update relabeling to send port in path

c23640e

Add file discovery of ecs tasks

9376ed1

Add basic auth settings to the application level metrics scraping

bbce398

Change job name for ecs tasks in dev

268c76d

Add SG for monitoring to ECS tasks ports

66bfc1f

Add cronjob to discover ecs services in monitoring server

2c9faca

Add permissions required for ECS discovering

d0a1bd0

Add sh files for ecs discovery settings

c8dc243

Ensure that boto3 is installed for the ecs-discovery script

9059895

Add user for service discovery task

e1ef1fc

LDiazN added the funder/otfcred2025 label Feb 17, 2025

LDiazN requested a review from hellais February 17, 2025 12:36

LDiazN self-assigned this Feb 17, 2025