-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Agent fails with credential errors, cannot use IAM Roles for Service Accounts (IRSA) or EKS Pod Identities with EKS Addon #119
Comments
Hi @AaronFriel, Have a few questions that would help us look into your issue.
|
Hey @jefchien thanks for getting back to me. Yeah, this is the TypeScript code used with Pulumi IaC to deploy EKS CloudWatch. // This configures IRSA with an `AssumeRoleWithWebIdentity` - confirmed working with other addons including:
// * AWS EFS CSI Driver
// * Karpenter Controller
const role = IamServiceAccountRole(`${clusterPetName}-cloudwatch-observability`, {
namespaceName: 'amazon-cloudwatch',
serviceAccountName: 'cloudwatch-agent',
});
new aws.iam.RolePolicyAttachment(`${clusterPetName}-cloudwatch-observability-agent`, {
policyArn: 'arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy',
role,
});
new aws.iam.RolePolicyAttachment(`${clusterPetName}-cloudwatch-observability-xray`, {
policyArn: 'arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess',
role,
});
const addonVersion = aws.eks.getAddonVersionOutput({
addonName: 'amazon-cloudwatch-observability',
kubernetesVersion: clusterVersion,
mostRecent: true,
});
const addon = new aws.eks.Addon(
`${clusterPetName}-cloudwatch-observability`,
{
clusterName,
addonName: addonVersion.addonName,
addonVersion: addonVersion.version,
serviceAccountRoleArn: role.arn,
preserve: false,
},
{ dependsOn },
); Yes, here are the agent logs. I've formatted these for readability:
The only way to set apiVersion: v1
items:
- apiVersion: cloudwatch.aws.amazon.com/v1alpha1
kind: AmazonCloudWatchAgent
metadata:
annotations:
pulumi.com/patchForce: "true"
creationTimestamp: "2024-04-01T08:21:38Z"
generation: 5
labels:
app.kubernetes.io/managed-by: amazon-cloudwatch-agent-operator
name: cloudwatch-agent
namespace: amazon-cloudwatch
resourceVersion: "3839446"
uid: 542fecd4-0368-4ab1-8d8b-e7e5ad47c538
spec:
config: '{"agent":{"region":"us-west-2"},"logs":{"metrics_collected":{"app_signals":{"hosted_in":"opal-quokka-6860d02"},"kubernetes":{"cluster_name":"opal-quokka-6860d02","enhanced_container_insights":true}}},"traces":{"traces_collected":{"app_signals":{}}}}'
env:
+ - name: RUN_WITH_IRSA
+ value: true
- name: K8S_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName This edit enables the CloudWatch Agent to succeed. However, there is no way to guarantee that this change is persistent, because the |
Do you have IMDS disabled or a hop limit set to 1? This seems like a similar issue to aws/amazon-cloudwatch-agent#1101 where the agent thinks it is |
Yes, because that is considered best practice - but also because if node IMDS is enabled, it is not using pod identity, it's using node identity. If the CloudWatch Agent does not work with IMDS hop limit set to 1, what is this section doing? I think the answer is "nothing"? |
Any updates on this? It's preventing us from using it. |
We're working on providing an alternative to IMDS. You can track that here aws/amazon-cloudwatch-agent#1101. |
Users may wish to run the CloudWatch Agent using pod-based IAM roles, using the IRSA or EKS Pod Identities technologies. Recently (PR below) this was enabled when an environment variable is set on the agent pod,
RUN_WITH_IRSA=true
, and this enables the agent to utilize the default provider chain for AWS authentication.However, the EKS Addon for AWS CloudWatch Observability creates a managed
AmazonCloudWatchAgent
configuration, making it unsafe - there is no guarantee it won't be overridden - to add environment variables.Background
CloudWatch Agent PR:
Expected behavior
Running the EKS Addon for AWS CloudWatch Observability with pod-based IAM should work by default.
Actual behavior
The agent fails, and there is no knob available to users to ensure the agent works.
Proposal
Either of these solutions would address this:
AmazonCloudWatchAgent
custom resourcecwagentconfig.json
, which is managed by the add-on should accept a configuration key to enable the RUN_WITH_IRSA mode.The text was updated successfully, but these errors were encountered: