get secret as environnement variable from aws secretsmanager

[MedRAQAOUI]

Hi,

How to get secret as environnement variable from aws secretsmanager, is there an extension to do that ?

The need is to allow adding configuration inside the application.properties or application.yml at startup available for application boot.

Thank's

Best regards,
RAQAOUI MOHAMED

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[scrocquesel]

Currently, there is no configuration source for aws secretsmanager, but you can use the client directly to retrieve the secret value. See https://quarkiverse.github.io/quarkiverse-docs/quarkus-amazon-services/dev/amazon-secretsmanager.html

You should be able to build your own ConfigSourceFactory like

public class SecretsManagerConfigSourceProvider implements ConfigSourceFactory {
    

    @Inject
    SecretsManagerClient secretsManagerClient;

    @Override
    public Iterable<ConfigSource> getConfigSources(final ConfigSourceContext context) {
        List<ConfigSource> result = new ArrayList<>();
        // Retrieve all secrets using pagination and stream
        Map<String, String> secretsMap = secretsManagerClient.listSecretsPaginator().stream()
                .flatMap(response -> response.secretList().stream())
                .collect(Collectors.toUnmodifiableMap(SecretListEntry::name, secret -> getSecretValue(secretsManagerClient, secret.arn())));

        result.add(new SecretsManagerConfigSource(secretsMap, 0));

        return result;
    }

    private static String getSecretValue(SecretsManagerClient secretsManager, String secretArn) {
        GetSecretValueRequest getSecretValueRequest = GetSecretValueRequest.builder()
                .secretId(secretArn)
                .build();
        GetSecretValueResponse getSecretValueResponse = secretsManager.getSecretValue(getSecretValueRequest);
        return getSecretValueResponse.secretString();
    }

    private static final class SecretsManagerConfigSource extends MapBackedConfigSource {

        public SecretsManagerConfigSource(Map<String, String> propertyMap, int ordinal) {
            super("SecretsManagerConfigSource", propertyMap, ordinal);
        }
    }

    @Override
    public OptionalInt getPriority() {
        return OptionalInt.of(290);
    }
}

[madocx]

@scrocquesel any suggestions for using this secrets extension with something like the reactive-pg-client quarkus extension that requires credentials available during STATIC_INIT? The above approach I suspect will result in the same issue where the SecretsManagerClient is not yet initialized during STATIC_INIT as its not available until RUNTIME_INIT.

I don't want to speak for OP, but I suspect that's why they're looking for a way to read secrets and inject as environment variable.

[scrocquesel]

reactive-pg-client

@madocx The Draft PR above is an attempt to have the config read during STATIC_INIT. Tests demonstrate that it can be used to feed the other sdk clients produced at runtime.

I kept the PR as a draft because it has not been tested thoroughly in real use cases, and I don't know if the filter configuration part matches the way AWS secrets are generally used.
If you're willing to test it in your project and provide valuable feedback, it is something I can finally merge.

[MedRAQAOUI]

Hi @scrocquesel ,

for me it work fine with @StaticInitSafe annotation :
@StaticInitSafe public class IotSecretsManagerConfigSourceFactory implements ConfigSourceFactory {

And also the dependencies below :
implementation platform('software.amazon.awssdk:bom:2.19.29') implementation 'software.amazon.awssdk:secretsmanager' implementation 'software.amazon.awssdk:sts'

And also WEB Identity Provider and right role on aws EKS .

Best reguards,
RAQAOUI MOHAMED

[madocx]

Excellent, thank you @scrocquesel! I'll implement it today and give it a whirl.

[madocx]

@scrocquesel, briefly looking through the draft PR I have an initial concern I figured I'd pass along. It looks like you're using the same approach as the example above where you're paginating all secrets in the account and retrieving them. If I misunderstood this please correct me. I don't have an understanding of much of the code changes in the PR but tried to identify the meat of where it was retrieving the secrets.

Assuming my interpretation is correct, this is not the most efficient way to do this, as you're billed per secret retrieval by AWS. In larger organizations there may be hundreds if not thousands of secrets in an account. Ideally, this functionality would be configurable to only retrieve the secrets in a specific config list.

[MedRAQAOUI]

@madocx

Yes by providing a secret name that want to pull , we can be more precise for secret that we want :

`

@StaticInitSafe
public class IotSecretsManagerConfigSourceFactory implements  ConfigSourceFactory {


    public static final String VERSION_STAGE = "AWSCURRENT";

    /** The ordinal is set to < 100 (which is the default) so that this config source is retrieved from last. */
    private static final int SECRET_MANAGER_ORDINAL = 280;

    private static final String CONFIG_SOURCE_NAME = "com.loxam.aws.secretsmanager.config";



    @Override
    public Iterable<ConfigSource> getConfigSources(ConfigSourceContext context) {
        final ConfigValue value = context.getValue("secretname");
        if (value == null || value.getValue() == null) {
            return Collections.emptyList();
        }

         return Collections.singletonList(new IotSecretsManagerConfigSource(VERSION_STAGE,SECRET_MANAGER_ORDINAL,CONFIG_SOURCE_NAME,value.getValue(),SecretsManagerClient.create()));


    }

    @Override
    public OptionalInt getPriority() {
        return OptionalInt.of(290);
    }
}

`

[gilday]

Ideally, this functionality would be configurable to only retrieve the secrets in a specific config list.

Yet, this introduces another level of configuration. Might there be a solution that balances cost concerns while avoiding the extra configuration? What if the extension listed all the secrets, but it lazily retrieved the value of each secret? In this case, the extension will incur a charge for each page of secrets it retrieves, but it only incurs a charge for retrieving the secret value when it's actually needed.

[madocx]

My concern with that would be adding to cold start time in lambdas. I'm all for avoiding unneeded configuration, but I'm not sure this is the place for that. I guess as long as that functionality is able to be disabled with optional configuration.

[gilday]

If I understand, the proposed configuration restricts the secrets available to this extension to only those in some allowed list, so that the function only retrieves the secrets that it actually uses (for cost reasons).

Is that duplicative to what one might achieve with IAM configuration? That is, if my account has hundreds of secrets, but my Lambda function only needs 4 of those secrets, then I should only permit the IAM role that my function assumes to access only the 4 secrets it needs. It's my understanding that an API call to list secrets from this function would only retrieve the 4 secrets it needs. Furthermore, addressing this concern with IAM permissions ultimately leads to a better "least privilege" outcome.

[madocx]

Oh that's an interesting thought I haven't considered. Looking at https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html it looks like this would be the ListSecrets action. Would need to test to see if that can be restricted to specific resources. The table seems to imply that its pretty binary, either you can list them all, or list none.

To be clear, this is a different action than the one needed to retrieve the secret value, which is of course restricted to specific resources (secrets).

[scrocquesel]

@scrocquesel, briefly looking through the draft PR I have an initial concern I figured I'd pass along. It looks like you're using the same approach as the example above where you're paginating all secrets in the account and retrieving them. If I misunderstood this please correct me. I don't have an understanding of much of the code changes in the PR but tried to identify the meat of where it was retrieving the secrets.

Assuming my interpretation is correct, this is not the most efficient way to do this, as you're billed per secret retrieval by AWS. In larger organizations there may be hundreds if not thousands of secrets in an account. Ideally, this functionality would be configurable to only retrieve the secrets in a specific config list.

That's why from what I remember I tried to add filtering.

[scrocquesel]

Also, looking at the hashicorp vault extension, they also provide a way to fetch an application properties from the vault and add it as a config source.
Is there a way to distinguish the type/mime type of a secret ?

[gilday]

I don't see anything like type/mime in the "list secrets" response.

[madocx]

Not that I'm aware of. Secret value will always be returned as a string value via the AWS api, even if a json structure (in which case it will be escaped). I think its reasonable to expect users to handle their string values accordingly.

[madocx]

@MedRAQAOUI looking at your ConfigFactory more closely, I'm not sure if you intended this or not, but you're bypassing the quarkus extension managed SecretsManagerClient. I suspect if you try injecting the client rather than doing a create you'll get an error since its currently not available during STATIC_INIT and is why the PR @scrocquesel is working on is necessary.

The main downside to this approach is you lose the other benefits of the extension - mainly the configuration options exposed for both sync and async http clients, and credential providers. For your use case this may not be a problem, but I do believe its non-optimal.

[gilday]

you're bypassing the quarkus extension managed SecretsManagerClient. I suspect if you try injecting the client rather than doing a create you'll get an error since it's currently not available during STATIC_INIT

I ran into the same issue while trying to implement a ConfigSource for AWS Secrets Manager, and that's how I came upon @scrocquesel 's work 🙂.

[MedRAQAOUI]

@madocx this code was before @scrocquesel's work , I'm the requester of this PR

[scrocquesel]

you're bypassing the quarkus extension managed SecretsManagerClient. I suspect if you try injecting the client rather than doing a create you'll get an error since it's currently not available during STATIC_INIT

I ran into the same issue while trying to implement a ConfigSource for AWS Secrets Manager, and that's how I came upon @scrocquesel 's work 🙂.

Yes, the whole purpose of the PR is to provide the client to the configsource with the same config experience as other extension.

Also, to reduce cost, we may need to cache the retrieved values because config source can be called multiple times. How should we handle value changes ? But, I ran out of time and left it for a future PR.

[madocx]

Also, to reduce cost, we may need to cache the retrieved values because config source can be called multiple times. How should we handle value changes ? But, I ran out of time and left it for a future PR.

All other implementations I've seen for secrets handle this with a TTL configuration. Rotating secrets should allow for enough overlap between them that any TTL value is not a problem. I suggest any TTL be optional though, as there are many use cases where secret rotation is not utilized due to technical limitations of other systems.

So perhaps default of indefinite cache, with configurable TTL for rotation?

[pragmasoft-ua]

@MedRAQAOUI @scrocquesel I noticed this issue is still open. We need the same functionality to read the configuration, but from SSM parameter store.

I'm thinking about joining our efforts to contribute a separate quarkiverse extension for this in spirit of https://docs.powertools.aws.dev/lambda/java/utilities/parameters/ to read from both ssm and secretsmanager.

Also I was thinking about https://openfeature.dev/ integration extension sometimes, so we may need reading from the AppConfig as well.

If you don't mind, I'll register new extension request for this on quarkiverse

[MedRAQAOUI]

@pragmasoft-ua for me reading from aws secretsmanager only needs to implement these 2 interfaces:

• io.smallrye.config.common.AbstractConfigSource
• io.smallrye.config.ConfigSourceFactory
  And to have AWS authentification managed out of java code by the cloud (in our side AWS EKS) you need to add aws sts dependancy :
• software.amazon.awssdk:sts
  In addition to WEB IDENTITY PROVIDER mechanism well configured on AWS EKS (declaration of aws IAM role that identify the aws EKS cluster).

If you want the code of AbstractConfigSource and ConfigSourceFactory implementation I can share it with you.

[scrocquesel]

@MedRAQAOUI @scrocquesel I noticed this issue is still open. We need the same functionality to read the configuration, but from SSM parameter store.

I'm thinking about joining our efforts to contribute a separate quarkiverse extension for this in spirit of https://docs.powertools.aws.dev/lambda/java/utilities/parameters/ to read from both ssm and secretsmanager.

Also I was thinking about https://openfeature.dev/ integration extension sometimes, so we may need reading from the AppConfig as well.

If you don't mind, I'll register new extension request for this on quarkiverse

It could be a good idea. This extension is mainly designed to provide clients and has been designed to offer clients at the RUNTIME_INIT phase. Unfortunately, ConfigSourceFactory requires everything to be set up earlier. I attempted to share the same spirit of being able to choose the client implementation (sync/async/url/netty/apache/crt), but it resulted in a lot of duplicated code, and I don't think it's really necessary.
I believe that having a simpler extension focused only on providing the configuration would be easier to implement. Nonetheless the PR is still opened for contribution. I don't use AWS myself, so I have no added value regarding STS.