[f5-cloud-failover] Status: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1 Retries left: 0

[sd224646]

Do you already have an issue opened with F5 support?

no.

Description

Getting below error on AWS F5 and failover not working
[f5-cloud-failover] Status: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1 Retries left: 0

Environment information

For bugs, enter the following information:

• Cloud Failover Extension Version: 1.15.0
• BIG-IP version: 17.1.0.3

Severity Level

2

Severity: <Fill in level: 1 through 5>

Severity level definitions:

1. Severity 1 (Critical) : Defect is causing systems to be offline and/or nonfunctional. immediate attention is required.
2. Severity 2 (High) : Defect is causing major obstruction of system operations.
3. Severity 3 (Medium) : Defect is causing intermittent errors in system operations.
4. Severity 4 (Low) : Defect is causing infrequent interuptions in system operations.
5. Severity 5 (Trival) : Defect is not causing any interuptions to system operations, but none-the-less is a bug.

[mikeshimkus]

Hi @sd224646 This error usually happens when the IAM role/policy isn't assigned to the BIG-IP instances. Can you verify that a role with the correct policy permissions is assigned per https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/aws.html#create-and-assign-an-iam-role

[sd224646]

Thanks, you are correct, I had created IAM role/policy and missed to attach on BIG-IP instances.
Further I can see now failover is working but failover will happen once below errors count goes to 0. Please review and confirm if any finetune required on IAM.

Existing IAM permissions:
"ec2:DescribeInstances",
"ec2:DescribeInstanceStatus",
"ec2:DescribeAddresses",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeNetworkInterfaceAttribute",
"ec2:DescribeRouteTables",
"s3:ListAllMyBuckets",
"ec2:AssociateAddress",
"ec2:DisassociateAddress",
"ec2:AssignPrivateIpAddresses",
"ec2:UnassignPrivateIpAddresses",
"ec2:ReplaceRoute",
"ec2:CreateRoute",
"ec2:DeleteRoute",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface"

error:

Sun, 22 Oct 2023 13:44:27 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 49
Sun, 22 Oct 2023 13:44:32 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 48
Sun, 22 Oct 2023 13:44:37 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 47
Sun, 22 Oct 2023 13:44:42 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 46
Sun, 22 Oct 2023 13:44:47 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 45
Sun, 22 Oct 2023 13:44:52 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 44

[mikeshimkus]

You only have one S3 permission, but seven are required. These are the permissions you will need: https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/aws.html#create-and-assign-an-iam-role

[sd224646]

Below is updated IAM policy with error. its retry for approx. 40 times then will do failover.
Note: after 40 retry failover is success. need to know why this is trying 40 retry

{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeInstanceStatus",
"ec2:DescribeSubnets",
"ec2:DescribeAddresses",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeNetworkInterfaceAttribute",
"ec2:DescribeRouteTables",
"s3:ListAllMyBuckets",
"ec2:AssociateAddress",
"ec2:DisassociateAddress",
"ec2:AssignPrivateIpAddresses",
"ec2:UnassignPrivateIpAddresses",
"ec2:ReplaceRoute",
"ec2:CreateRoute",
"ec2:DeleteRoute",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"logs:CreateLogStream",
"logs:CreateLogGroup",
"logs:PutLogEvents"
],
"Resource": "",
"Effect": "Allow"
},
{
"Action": [
"sts:AssumeRole"
],
"Resource": "arn:aws:iam::123456789123:role/F5_cfe_IAM_Role",
"Effect": "Allow"
},
{
"Action": [
"ec2:CreateRoute",
"ec2:ReplaceRoute"
],
"Resource": [
"arn:aws:ec2:us-east-1:123456789123:route-table/rtb-1111111111111111",
"arn:aws:ec2:us-east-1:123456789123:route-table/rtb-2222222222222222",
"arn:aws:ec2:us-east-1:123456789123:route-table/rtb-33333333333333333"
],
"Condition": {
"StringEquals": {
"ec2:ResourceTag/f5_cloud_failover_label": "Big-IP-F5"
}
},
"Effect": "Allow"
},
{
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:GetBucketTagging"
],
"Resource": "arn:aws:s3:::s3-f5-cfe",
"Effect": "Allow"
},
{
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::s3-f5-cfe/",
"Effect": "Allow"
}
]
}

Mon, 23 Oct 2023 17:25:20 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. User: arn:aws:sts::123456789123:assumed-role/F5_cfe_IAM_Role/i-aaaaaaaaaaaaaaaa is not authorized to perform: ec2:DescribeSubnets because no identity-based policy allows the ec2:DescribeSubnets action Retries left: 38
Mon, 23 Oct 2023 17:25:25 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. User: arn:aws:sts::123456789123:assumed-role/F5_cfe_IAM_Role/i-aaaaaaaaaaaaaaaa is not authorized to perform: ec2:DescribeSubnets because no identity-based policy allows the ec2:DescribeSubnets action Retries left: 37
.
.Will go till retry left 0
.
.
Mon, 23 Oct 2023 17:28:27 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. User: arn:aws:sts::123456789123:assumed-role/F5_cfe_IAM_Role/i-aaaaaaaaaaaaaaaa is not authorized to perform: ec2:DescribeSubnets because no permissions boundary allows the ec2:DescribeSubnets action Retries left: 1
Mon, 23 Oct 2023 17:28:32 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. User: arn:aws:sts::123456789123:assumed-role/F5_cfe_IAM_Role/i-aaaaaaaaaaaaaaaa is not authorized to perform: ec2:DescribeSubnets because no permissions boundary allows the ec2:DescribeSubnets action Retries left: 0

[mikeshimkus]

The error message is clear, the policy does not have the DescribeSubnets permission. It looks like the resource for that permission in your policy is blank. Check out https://github.com/F5Networks/f5-aws-cloudformation-v2/blob/8afb5e28975f42e70f64d2808735f91bd2663f44/examples/modules/access/access.yaml#L916 for an example from our CloudFormation templates.

[sd224646]

Thanks, I will modify IAM and will come back with upadate.

[sd224646]

Have verified again resource for IAM in your policy is not blank. not sure why "" is not showing i above comment.
further I am using () and allow for"ec2:DescribeSubnets". not sure why I am getting error saying You are not authorized to perform this operation.
Could you please any working full IAM. my purpose is to replace destination nic in route table.

[sd224646]

ok got it. looking like Github comment section not allow "asterisk". same i can see in my above comments.

[mikeshimkus]

The example shown above is the full working policy: https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/aws.html#iam-role-example-declaration

Your policy looks quite different from what's documented (you do not need sts:AssumeRole, for example), I would create a policy based on the example above and try again.

[sd224646]

No Luck, first statement ""BigIpHighAvailabilityAccessRole"" itself not accepted by IAM. then all "\asterisk" also not accepted.

[mikeshimkus]

That's the policy name, I don't think the editor will allow you to use it in the policy itself. The asterisks need quotes around them, that's a typo.

Can you update your existing policy to match our example?