[WIP] New Feature: `assume_role` connection config #1845

brittandeyoung · 2023-07-14T17:56:26Z

I am new to contributing to this project so some guidance to make sure I follow the correct procedures would be appreciated. This pull request is currently a Work In Progress. I am attempting to add the ability to use the assume_role in the connection configuration in order to leverage role assumption when configuring a connection to an AWS account instead of having to provide a profile or a static set of credentials.

I did not see any integration tests related to the creating and testing of the AWS connection itself for me to run.

Closes: #1844

Example config:

connection "aws" {
  plugin = "aws"
  type        = "aggregator"
  connections = ["aws_sandbox", "aws_ci", "aws_pre_prod"]
}

connection "aws_sandbox" {
  plugin      = "aws" 
  assume_role = {
    role_arn = "arn:aws:iam::111111111111:role/steampipe"
    duration = "300s"
    external_id = "testing"
    session_name = "steampipe"
  } 
  regions     = ["us-east-1", "us-west-2"]
}

I was able to use this config to run successful queries

rajlearner17 · 2023-07-15T06:24:30Z

@brittandeyoung Thanks for trying out Steampipe!
We appreciate your interest in contributing to the community 👍

As you are still working on it, please check the following docs for your reference

Writing Plugins
AssumeRole - I am not sure if this is the one you are looking for in your PR?

… role arn, duration, external_id, and session_name

brittandeyoung · 2023-07-19T14:27:45Z

@rajlearner17 There have been more details in the issue around the reason for these suggested changes. The biggest reason is that it would be nice to avoid having to both manage the ./aws/config file and the aws.spc file when setting up connections. It would be ideal to be able to manage all of this within the steampipe aws connection file.

I have updated the PR with setting these configurations at the root config, but am looking at some suggestions on how we can make this a config block.

brittandeyoung · 2023-09-17T20:35:46Z

For awareness, I have the assume role configuration working. The last piece is the optional items under the assume_role config. More details are in the issue. With the current code in the PR, assuming roles works, but the optional fields duration, external_id and session_name do not work.

cbruno10 · 2023-09-19T19:34:04Z

aws/service.go

@@ -1708,7 +1709,7 @@ func getBaseClientForAccount(ctx context.Context, d *plugin.QueryData) (*aws.Con
 // If we expire the cache regularly we are causing SSO sessions to end
 // prematurely, and causing the AWS SDK to refresh credentials more often
 // using the IDMS service etc.
-var getBaseClientForAccountCached = plugin.HydrateFunc(getBaseClientForAccountUncached).Memoize(memoize.WithTtl(time.Hour * 24 * 30))
+var getBaseClientForAccountCached = plugin.HydrateFunc(getBaseClientForAccountUncached).Memoize(memoize.WithTtl(time.Second * 10))


@brittandeyoung Is there a reason you shortened the cache time, was this used just for testing?

@cbruno10 Yes this change was made for testing, I have reverted this change.

cbruno10 · 2023-09-19T19:34:55Z

config/aws.spc

@@ -27,6 +27,14 @@ connection "aws" {
  # from an AWS credential file with the `profile` argument:
  #profile = "myprofile"

+  # If no credentials are specified, the plugin will use the AWS credentials


Is this example still correct, or does it follow the nested structure now?

@cbruno10 Thank you, I had forgot to update the example after changing to a map for configuration. The example has now been updated.

cbruno10 · 2023-09-19T19:36:25Z

aws/connection_config.go

-	IgnoreErrorCodes      []string `cty:"ignore_error_codes"`
-	EndpointUrl           *string  `cty:"endpoint_url"`
-	S3ForcePathStyle      *bool    `cty:"s3_force_path_style"`
+	Regions               []string          `hcl:"regions,optional"`


@brittandeyoung Referencing your most recent issue comment #1844 (comment), you mention using cty, but here in the code I see all fields using hcl instead.

How does the code currently behave? Does it not work in some scenarios?

@cbruno10 In the current state, the role_arn loads in properly and assuming the role works as expected as it has the cty tag.

Currently the remainder of the fields to not have the cty tag. Without this tag, these items DO NOT load in from the config. This means these values are nil.

If I add the cty tag, the values do load in, but then are no longer optional and must be provided. If you do not provide them all, steampipe crashes and fails to load. If i attempt to add the optional tag, steampipe crashes when attempting to load the config.

Any tips on direction from here is appreciated

We were able to reproduce the above issue -

Error: failed to start plugin 'hub.steampipe.io/plugins/turbot/aws@latest': failed to start 'hub.steampipe.io/plugins/turbot/aws@latest': failed to parse connection config for connection 'aws': Unsuitable value type: Unsuitable value: attribute "external_id" is required

This could be an HCL parsing error - https://github.com/turbot/steampipe-plugin-sdk/blob/main/plugin/connection_config_schema.go#L88

I have raised a Steampipe SDK issue for the same with the details.

@brittandeyoung, please let us know if you are getting any other issues than the above.

@bigdatasourav Great thanks! That was the only issue i was running into. I will keep an eye on this issue.

…ig map

opsanatoliy · 2023-09-21T19:20:49Z

@brittandeyoung could you please tell me how you replace aws plugin on your local version to run?

brittandeyoung · 2023-09-21T19:21:21Z

@brittandeyoung could you please tell me how you replace aws plugin on your local version to run?

@opsanatoliy Using the install make target. So:

make install

cbruno10 · 2023-10-20T05:29:44Z

aws/connection_config.go

 }

-var ConfigSchema = map[string]*schema.Attribute{


@brittandeyoung Why did you remove this along with Schema: ConfigSchema, in plugin.go?

@cbruno10 This was done in order to support the map for assume_role as the schema does not support map objects. The idea was to use tagging to dynamically pull in the settings.

I had this working with root level configurations instead of a map and utilizing the defined schema. Would this pull request be better to be reverted to using root level configurations instead of a map configuration?

Thanks for highlighting this; I have raised a Steampipe SDK issue to track.

@bigdatasourav Great! This getting added to the SDK will likely enable me to complete this PR with working code. I will be sure to follow this issue.

github-actions · 2023-12-25T23:32:15Z

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

misraved · 2024-01-05T07:58:51Z

Closing the PR for now, since it is waiting on the resolution of the following SDK issues -
turbot/steampipe-plugin-sdk#692
turbot/steampipe-plugin-sdk#691

brittandeyoung added 4 commits July 14, 2023 13:49

Amend connection_config, Add AssumeRole

77d81ed

Amend service, Add AssumeRole

09d5cda

Amend config/aws, add assume_role note

78fbeb9

Amend service, resolve assume_role

e9dd113

rajlearner17 requested review from misraved and removed request for misraved July 15, 2023 06:24

brittandeyoung added 3 commits July 19, 2023 10:23

aws: Amend connection_config, Add top level configurations for assume…

ac953c4

… role arn, duration, external_id, and session_name

aws: Amend service, Add building of AssumeRoleOptions

c5a3b51

config: Amend aws.spc, add example for simple role assumption

ea8422b

brittandeyoung changed the title ~~[WIP] New Feature: assume_role connection config~~ New Feature: assume_role connection config Jul 19, 2023

config: Amend aws.spc, re word example of role assumption

1286e83

brittandeyoung changed the title ~~New Feature: assume_role connection config~~ [WIP] New Feature: assume_role connection config Jul 21, 2023

brittandeyoung added 6 commits September 16, 2023 10:36

aws: Amend connection_config, move assume_role to block config

b0dbf12

aws: Amend plugin, remove config Schema

a9f0107

aws: Amend service, Add Assume role dynamic configuration

9bb3ed5

Merge branch 'main' into feature/assume_role-connection-config

9750cc0

aws: Amend connection_config, update config for assume_role

f7affb1

aws: service, working assume role

b9c5973

aws: plugin, remove un needed commented code

3c8898f

cbruno10 reviewed Sep 19, 2023

View reviewed changes

brittandeyoung added 2 commits September 21, 2023 11:01

config: Update aws.spc, Update assume_role config example to use conf…

c8f7c67

…ig map

aws: Amend service, revert caching change for testing

f6610e0

cbruno10 reviewed Oct 20, 2023

View reviewed changes

bigdatasourav mentioned this pull request Oct 25, 2023

Can we add support for map/object in configSchema turbot/steampipe-plugin-sdk#691

Open

github-actions bot added the stale No recent activity has been detected on this issue/PR and it will be closed label Dec 25, 2023

ParthaI removed the stale No recent activity has been detected on this issue/PR and it will be closed label Dec 27, 2023

misraved closed this Jan 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] New Feature: `assume_role` connection config #1845

[WIP] New Feature: `assume_role` connection config #1845

brittandeyoung commented Jul 14, 2023 •

edited

Loading

rajlearner17 commented Jul 15, 2023 •

edited

Loading

brittandeyoung commented Jul 19, 2023 •

edited

Loading

brittandeyoung commented Sep 17, 2023

cbruno10 Sep 19, 2023

brittandeyoung Sep 21, 2023

cbruno10 Sep 19, 2023

brittandeyoung Sep 21, 2023

cbruno10 Sep 19, 2023

brittandeyoung Sep 21, 2023

bigdatasourav Oct 25, 2023

bigdatasourav Oct 25, 2023

brittandeyoung Oct 26, 2023

opsanatoliy commented Sep 21, 2023

brittandeyoung commented Sep 21, 2023

cbruno10 Oct 20, 2023

brittandeyoung Oct 23, 2023

bigdatasourav Oct 25, 2023

brittandeyoung Oct 25, 2023 •

edited

Loading

github-actions bot commented Dec 25, 2023

misraved commented Jan 5, 2024

[WIP] New Feature: assume_role connection config #1845

[WIP] New Feature: assume_role connection config #1845

Conversation

brittandeyoung commented Jul 14, 2023 • edited Loading

rajlearner17 commented Jul 15, 2023 • edited Loading

brittandeyoung commented Jul 19, 2023 • edited Loading

brittandeyoung commented Sep 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opsanatoliy commented Sep 21, 2023

brittandeyoung commented Sep 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brittandeyoung Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Dec 25, 2023

misraved commented Jan 5, 2024

[WIP] New Feature: `assume_role` connection config #1845

[WIP] New Feature: `assume_role` connection config #1845

brittandeyoung commented Jul 14, 2023 •

edited

Loading

rajlearner17 commented Jul 15, 2023 •

edited

Loading

brittandeyoung commented Jul 19, 2023 •

edited

Loading

brittandeyoung Oct 25, 2023 •

edited

Loading