Support for credential_process in ~/.aws/config

[kennu]

This is a Feature Proposal

Description

AWS CLI supports the credential_process option in ~/.aws/config profiles, which allows you to store credentials somewhere else than the plain ~/.aws/credentials file. For instance, you can create a script that fetches them from 1Password using the 1Password CLI.

Serverless Framework doesn't support this so if you use this feature, you will get a Profile does not exist error when deploying. I know supporting this may not be quite trivial, but it might be very convenient.

Additional Data

• Serverless Framework Version you're using: 1.26.1
• Operating System: macOS
• Stack Trace:
• Provider Error messages:

[meatherly]

Just needs to be merged aws/aws-sdk-js#1923. Then we can implement it on serverless

[StevenACoffman]

This has been merged in aws/aws-sdk-js#2559 and is available with aws-sdk-js v2.474.0 through v2.429.0

Would appreciate if this got picked up.

[mungojam]

It would be lovely to see this progress again (sorry I can't help at the moment). Currently I have to keep switching my config files between using credential_process for all other SDKs and tools and using temporary credentials when calling serverless.

[thomasmichaelwallace]

Just another +1.

credential_process seems to be a good work around for #7567 (see https://github.com/benkehoe/aws-sso-credential-process).

At the moment I'm trying to understand why the serverless framework uses it's own approach to credentials; which seems to be the reason that credential_process is supported by the aws-sdk out-of-the-box, but doesn't work here.

[flyinprogrammer]

but seriously, why aren't we using the default CredentialProviderChain ??

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CredentialProviderChain.html
https://github.com/aws/aws-sdk-js/blob/f1c689ffbca0d21d7bd359d5d4a5f1bd2b498a8b/lib/node_loader.js#L58-L70

[thomasmichaelwallace]

@flyinprogrammer - I'm actually trying to get together a PR to see about using it.

However, because sls hasn't used the default chain, there's a few additional cases I'm working through.

For comment, it's looking something like this:

    // Tweak the default provider chain to match serverless extensions:
    // https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CredentialProviderChain.html#defaultProviders-property
    const credentialProvider = new AWS.CredentialProviderChain([
      function () { return new AWS.EnvironmentCredentials('AWS'); },
      function () { return new AWS.EnvironmentCredentials('AMAZON'); },
      // add serverless specific AWS_${STAGE} environment
      function () { return new AWS.EnvironmentCredentials(environment); },
      // prefer serverless configured profile and readline token fn for ini:
      function () { return new AWS.SharedIniFileCredentials({ profile, tokenCodeFn, filename }); },
      // function () { return new AWS.ECSCredentials(); }
      // prefer serverless configured profile for process:
      function () { return new AWS.ProcessCredentials({ profile }); },
      // function () { return new AWS.TokenFileWebIdentityCredentials(); },
      // function () { return new AWS.EC2MetadataCredentials(); }
      // support serverless config credentials
      function () { return new AWS.Credentials(providerCredentials) },
    ]);

[dougmoscrop]

Looking for feedback on a first pass at using the provider chain here: https://github.com/dougmoscrop/serverless/commit/49cd19aa4b1954aa0441fbebeca8e9291bf635fe

[drissamri]

Is there any workaround currently for using AWS SSO until this is done?

[thomasmichaelwallace]

If you've only got one profile, you can use this: https://github.com/benkehoe/aws-export-credentials

[flyinprogrammer]

@drissamri i've been using aws-vault like so:

aws-vault exec profile_name -- sls ...

which just causes sls to rely on environment variables existing.

[emilhdiaz]

@thomasmichaelwallace I installed aws-sso-credential-process through pipx: pipx install aws-sso-credential-process, configured that against my desired aws profile in ~/.aws/config, and finally wrote this bash script (aws-sso) to wrap any shell commands that do not support AWS CLI v2 SSO profiles yet:

#!/bin/bash
set -euE -o pipefail

CREDS=$(aws-sso-credential-process --profile "${AWS_PROFILE}")

(
  unset AWS_PROFILE &&
  AWS_ACCESS_KEY_ID=$(echo "${CREDS}" | jq -r '.AccessKeyId') \
  AWS_SECRET_ACCESS_KEY=$(echo "${CREDS}" | jq -r '.SecretAccessKey') \
  AWS_SESSION_TOKEN=$(echo "${CREDS}" | jq -r '.SessionToken') \
  AWS_DEFAULT_REGION=$(aws configure list | grep region | awk '{print $2}') \
  "$@"
)
SUBSHELL_STATUS=$?

if [ $SUBSHELL_STATUS -eq 0 ]; then
exit 0
else
echo "aws-sso command failed."
exit 1
fi

To use:

aws-sso sls deploy

[thomasmichaelwallace]

☝️ it's an interesting solution - the main struggle we have is that we rely on provider.profile a lot, where as these effectively create a single profile configuration controlled outside of serverless.

I'm actually planning on picking up @dougmoscrop 's implementation and working that into a PR today.

So hopefully native serverless support for this is in the future...

[thomasmichaelwallace]

☝️ watch this space.

[thomasmichaelwallace]

So - there's quite a lot of refactoring planned for/around the files that need to be touched by this PR.

Unfortunately I don't have the time to look at these (#8441, #8444, #8445, #8446 - for anyone that does.) Once they're clear I'll come back and get the PR back up off the ground again.

I need this feature, now, however, and so I'll be maintaining a patch with the core (support for credential_process) code.

For anyone else who wants to use this patch in the interim - I'll be maintaining it (with some instructions) here: https://gist.github.com/thomasmichaelwallace/5ef97b1fbadf8df2bca21bebafd2dd7e

[medikoo]

For reference: Work on that has been started here: #8415 which was closed temporarily, until dependent issues, as listed below are addressed

• Test: Refactor lib/plugins/aws/invokeLocal/index.test.js #8441 - Test: Refactor lib/plugins/aws/invokeLocal/index.test.js
• Refactor: Seclude "lib/utils/log.js" (general logging) util #8444 - Refactor: Seclude lib/utils/log.js (general logging) util
• Refactor: Seclude "lib/utils/aws/request.js" (AWS request logic) util #8445 - Refactor: Seclude lib/utils/aws/request.js (AWS request logic) util
• Test: Refactor lib/plugins/aws/provider/awsProvider.test.js #8446 - Test: Refactor lib/plugins/aws/provider/awsProvider.test.js

Any help is addressing above will be appreciated

[thomasmichaelwallace]

For anyone following - I did a resync of my patch to the latest (v2.30.3) version,

While I've got everything back in sync - @medikoo - is the plan still to block on the issues mentioned in November?

[medikoo]

While I've got everything back in sync - @medikoo - is the plan still to block on the issues mentioned in November?
@medikoo

@thomasmichaelwallace yes I believe nothing changed, to welcome this enhancement safely, first we need to improve our tests (it's quite sensitive change)

[thomasmichaelwallace]

And again - re-synced for v2.61.0 for those who want to try the new ARM lambdas (patch-package compatible .patch here)

@medikoo - I see that the majority of the issues that were blocking this becoming a PR again are solved. Is it worth submitting this patch as a PR?

It's also worth appreciating that the assumption the serverless framework currently makes about AWS credentials (that they are synchronous and static) isn't true, and the majority of work I do when re-syncing the fixes is to refactor the tests to reflect this (see my branch ./test/unit/lib/plugins/aws/provider.test.js).

[medikoo]

@thomasmichaelwallace as problem with AWS credentials resolution in Framework is bigger than simply lack of support for credential_process we've decided to cover this issue in context of #9290

Any help in driving that forward is ofc welcome, still as it touches sensitive parts, it needs to be done with care, and best if no implementation happens until we have full agreement on spec level.

[thomasmichaelwallace]

@medikoo is there nothing I can say/do to convince you otherwise? AWS SSO is nearly 4 years old now.

• The patch passes all unit and integration tests
• Over the last year there's been work to improve (Refactor: Seclude "lib/utils/aws/request.js" (AWS request logic) util #8445 and Test: Refactor lib/plugins/aws/provider/awsProvider.test.js #8446) the extent and reliability of these tests (which was the initial concern that blocked the original PR)
• We've been using it on our production pipeline for over a year (100s of deployments, and use with community plugins like webpack, appsync, custom domains, etc.)

#9290 looks great (I'm sure people had their reasons, but I've also found it weird that Serverless has its own credentials resolution process) but also looks abandoned with no action for over a month, and no discussion. More worryingly it looks like it would result in a set of breaking changes that would be gated behind a major release, and it doesn't look like it'd be in time for v3. Although you will know much the roadmap better than I, it seems a fair bet that waiting for v4 means this issue (that has a solution) remains open for at least another year...

n.b. As it happens the approach used does address a lot of the points in #9290 as it uses the aws-sdk's built in credential resolution process; just one configured to support the slightly different resolution order the serverless decided to use. So it would represent an incremental movement towards what's being discussed.

[brandon-fryslie]

It's been years and standard AWS credentials support is still broken :/ I appreciate all the hard work the maintainers have been putting into the project, but I wish one of them had to deal with this issue so they would realize how much time it is wasting for many users of the tool

[mungojam]

In case it hasn't been mentioned, you can use awsume as a workaround for this. It supports credential_process and will set the env vars for serverless to pick up when run.

[thomasmichaelwallace]

Similarly I'll be upgrading my patch for v3 some time next week.

It makes Serverless re-use the standard aws-sdk credential chain (rather then their own one).

This does has the advantage of enabling credential_process &c. so that you can use all the other tooling that's popped up around SSO (my favourite is https://github.com/benkehoe/aws-sso-credential-process).

I admit I prefer the patch over using awsume because I need to do a lot of profile switching and it lets me use AWS_PROFILE and the serverless.yml / profile without any additional steps.

Honestly, I don't think they'll be any appetite for it, but I can try and post it as a PR for #9290.

[mnapoli]

Hey everyone, just wanted to reassure you that we are aware of this issue, and that we are keeping this in mind in our roadmap. Unfortunately breaking the current behavior is hard, but we have ideas for our roadmap and we are actively trying to imagine how this issue would fit into it.

To be transparent and honest, this is for the long term, don't expect a resolution in the coming weeks. But we are thinking about this.

[thomasmichaelwallace]

so

In the interim I give you all the serverless-better-credentials plugin with:

• built-in SSO support (if your aws cli works, then this should too!)
• support for credential_process
• respect for all the environment flags the aws-sdk-js@v2 supports.

https://github.com/thomasmichaelwallace/serverless-better-credentials

(n.b. this does mean that I won't be maintaining the patch any more, but hopefully the plugin will be ~x1000 more accessible to people.)

[kferrone]

I am really struggling with this issue. Hope this issue is resolved and realeased ASAP.

My profile:

[profile myprofile]
region = 'us-west-2'
output = 'json'
credential_process = 'mycli jit aws'

With this var set for the specific workspace:

export AWS_CONFIG_FILE="${CONF_DIR}/aws"

I can't seem to escape this error:

Cannot resolve serverless.yml: Variables resolution errored with:
  - Cannot resolve variable at "provider.iam.role": AWS profile "myprofile" doesn't seem to be configured

For compliance reasons, I can't make an IAM user let alone long lived credentials. Seems like serverless really wants the access key and secret key.

So I try this to set those env vars using a temp IAM sesion:

for i in `mycli jit aws -q '{AWS_ACCESS_KEY_ID: AccessKeyId, AWS_SECRET_ACCESS_KEY: SecretAccessKey, AWS_SESSION_TOKEN: SessionToken, AWS_REGION: Region}' -o env`; do export $i; done

This still doesn't work. It kind of seems it does not support the session token maybe, or does it not support using env vars? Maybe because I didn't put it in the aws credentials file? If so can I at least set AWS_SHARED_CREDENTIALS_FILE? I have to do this because each of my client projects are in their own directories and I manage nearly 100 accounts, I really don't want to pollute or even use ~/.aws/credentials or ~/.aws/config.

Then I tried using serverless-better-credentials from @thomasmichaelwallace. Unfortunately not even this worked for me, not sure why though, I just get the same error as everything else I try. I'll have to open an issue there too.

What am I missing here? I'm kind of sick of trialing and erroring something that should be trivial.

Edit: I think I found the issue. My global version was different than what was in package.json. Does the newer version actually support all of this? If so why is this issue still open?

Edit: Newer version does. Why is this still open? Hope customer doesn't mind upgrading.

[czubocha]

Hey everyone, closing this as Serverless Framework v4 uses the default AWS credential provider chain, which includes support for credential_process, AWS SSO, and other standard mechanisms defined in ~/.aws/config.

Thanks to everyone who contributed and shared workarounds here!