cloudwatch-agent doesn't appear to be working with IMDSv2 w/ auth enabled #249
Comments
|
Running into the same issue. Have not really been able to find any other guidance or leads on how to solve this. |
|
Apologies guys, I actually opened this ticket with AWS Support - EKS team and got some GREAT help.. I've copied their commands here for others in the hope it helps. Issue: Cloudwatch -> Insights wasn't showing performance metrics for pods in the node-group with IMDSv2
Once i updated the count to 2 my performance metrics immediately started showing up in insights.. BUT the cloudwatch_logs were still not showing... Issue: Cloudwatch -> log groups no node-group or applications logs from node-groups with the IMDSv2 In the end I updated created the role/attached the policy/created the service account via Kubectl manifest and then updated the aws-cw-fluent-bit configmap to set |
JohnPolansky
changed the title
|
Closing this ticket as the steps provided solved my issue.. Thanks! |
|
how can we check the hop-limit and further change it for an imdsv2 ec2 server? |
Hey All,
We've been using the
amazon-cloudwatch-agentfor awhile now and so far we have been loving it. We set it up using the documentation found here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.htmlHowever during a security audit we recently discovered that all our AWS (EKS) EC2 instances are running IMDS with open/unauthenticated access. As this is a security vulnerability we wanted to remove the concern. To that end we updated our terraform EC2 instance templates to switch the IMDS interface to authenticated and 1 hop
This worked as expected the IMDS interface is still open on HTTPS and required authentication which we assume should work through the service account provisioned for the cluster. We currently have 2 node-groups setup NodeGroupA was configured for IMDs https/auth/1 hop and NodeGroupB was left the default http/open access. However after we made the change, we didn't notice until several days later that all the
cloudwatch-agentson the NodeGroupA started failing continuouslyThis has resulted in a total lose of cloudwatch logs for any pods in the NodeGroupA and all CloudInsight Performance data is also being lost for the same nodegroup. We are hoping someone can help us with this issue, as it would seem best practice not to leave the IMDs with open access. We've temporarily enabled
aws-for-fluent-biton the cluster to at least ensure our cloudwatch logs keep coming in but we'd like to stick with your tool if we can.Here are the logs for a standard failure on NodeGroupA:
We noticed the lines about failing to connect to IMDS and then it mentioned
Detected the instance is OnPremwhich after some google search we found this issue: aws-samples/amazon-cloudwatch-container-insights#56, so we attempted to edit the daemonset config and add theRUN_IN_AWS=Truethe logs did change to indicate it picked up the option, but we are still getting failures:We appreciate any insights this work is being done on AWS Region us-east-2 and we are seeing it in two different accounts that we enabled the IMDS w/ auth.
The text was updated successfully, but these errors were encountered: