Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Setup application level metrics from ECS #938

Closed
LDiazN opened this issue Feb 11, 2025 · 1 comment
Closed

Setup application level metrics from ECS #938

LDiazN opened this issue Feb 11, 2025 · 1 comment
Assignees
@LDiazN
Labels
funder/otfcred2025

Comments

@LDiazN
Copy link
Contributor

LDiazN commented Feb 11, 2025

After #937 is done, we have to:

  • setup the permissions for the ecs cluster nodes to accept traffic for scraping their metrics (from the proxy host) and
  • setting up the file based discovery for prometheus
@LDiazN LDiazN self-assigned this Feb 11, 2025
@LDiazN LDiazN mentioned this issue Feb 17, 2025
Merged
LDiazN added a commit to ooni/devops that referenced this issue Feb 18, 2025
@LDiazN
App level metrics monitoring with Prometheus (#186)
03168e2 
This PR adds support for scraping application level metrics straight
from the ECS nodes, without going through the load balancer, solving the
flapping behavior described in #104

To achieve this we had to solve the following problems: 

- Reach the cluster nodes in AWS, this was solved by
#182
- Discover ECS tasks with their corresponding port and IP address. This
was a bit more tricky, we solved it by adding a cronjob that runs a
python script that will request ECS data from AWS using the `boto3`
client, and then storing that information into a [Prometheus file based
discovery](https://prometheus.io/docs/guides/file-sd/) compatible file

So, this PR will add: 
- A Python script to collect ECS tasks information from aws and writing
it into a file
- An Ansible configuration for creating and running this file with a
cronjob
- An update to the Nginx configuration used to proxy metrics requests
from the monitoring host to the EC2 instances in AWS
- Security groups configuration to allow traffic from the proxy host to
the ECS cluster nodes
- IAM credentials used for requesting tasks information from AWS 

This PR solves ooni/backend#937 and
ooni/backend#938 and is related to
#104
@LDiazN
Copy link
Contributor Author

LDiazN commented Feb 18, 2025

closed with ooni/devops#186

@LDiazN LDiazN closed this as completed Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LDiazN LDiazN

Labels
funder/otfcred2025
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@LDiazN