Is it possible to configure both a specific timeout for our DynamoDB operations (e.g. GetItem) AND another one for the provide_credentials part?

[Oliboy50]

Context

• Still the same context as in this previous discussion (i.e. call DynamoDB with Web Identity credentials from a backend application).
• I read this RFC about timeout configuration but I could not find an answer to my question there.

Question

Currently, we're doing a GetItem operation without any retry and with a single attempt timeout of 300 ms.

Our GetItem operation takes an average of 3 ms so it would be a good idea for us to decrease our timeout from 300 ms to something like 50 ms to avoid our server to go OutOfMemory when something will go wrong (e.g. all requests to DynamoDB timeouts for whatever reasons... just because shit happens... so our server keeps stacking tokio Tasks into RAM during 300 ms for nothing)...

BUT unfortunately, we can't decrease our current timeout configuration because it would also impact the timeout configuration of the "provide_credentials" requests that the SDK makes internally (probably to the AWS STS service) when the Web Identity token expires... and this request takes more than 200 ms sometimes 😿.

So, my question is: Is it possible to configure both a specific timeout for our DynamoDB operations (i.e. GetItem) AND another one for the internal/magical call to STS? And if it is, can someone share some code to show how it would look like, please?

[jdisanti]

Hmmm. This is definitely not ideal. I've cut an issue to track making this use-case better.

For now, I think you should be able to hack around this with the following:

    use aws_config::default_provider::credentials;

    let sdk_config = aws_config::from_env()
        .timeout_config(my_timeout_config()) // or `.http_connector(...)`
        // Explicitly set credentials to the default provider to prevent it from
        // acquiring the overridden HTTP connector above with the stricter timeout
        .credentials_provider(credentials::default_provider().await)
        .load()
        .await;

If you need to programmatically set the region, this gets a little more complicated:

    use aws_config::default_provider::credentials::DefaultCredentialsChain;

    let region = "us-east-1";
    aws_config::from_env()
        .region(region)
        .timeout_config(my_timeout_config()) // or `.http_connector(...)`
        // Explicitly set credentials to the default provider to prevent it from
        // acquiring the overridden HTTP connector above with the stricter timeout
        .credentials_provider(
            DefaultCredentialsChain::builder()
                .region(region)
                .build()
                .await,
        )
        .load()
        .await;

[Oliboy50]

Thank you!

I saw in the RFC that the SDK also supports environment variables to configure timeouts (i.e. AWS_CONNECT_TIMEOUT, AWS_API_CALL_ATTEMPT_TIMEOUT, etc.)...
And I think that I would use such environment variables if they would apply only to the "provide_credentials" part... because the timeout for this operation should be always the same no matter the "client" operations.

BTW, I just realized that I didn't give a code example to show how we currently configure the timeouts for each DynamoDB operation, so here you go:

    let sdk_config = aws_config::load_from_env().await;

    let dynamodb_get_item_client = aws_sdk_dynamodb::Client::from_conf(
        aws_sdk_dynamodb::config::Builder::from(&sdk_config)
            .retry_config(aws_config::retry::RetryConfig::disabled())
            .timeout_config(
                aws_config::timeout::TimeoutConfig::builder()
                    .operation_attempt_timeout(Duration::from_millis(300))
                    .build(),
            )
            .build(),
    );

    let dynamodb_put_item_client = aws_sdk_dynamodb::Client::from_conf(
        aws_sdk_dynamodb::config::Builder::from(&sdk_config)
            .timeout_config(
                aws_config::timeout::TimeoutConfig::builder()
                    .operation_attempt_timeout(Duration::from_millis(300))
                    .build(),
            )
            .build(),
    );

So we're currently also using 300 ms for the PutItem operation because of the "provide_credentials" thing... but what we would like to have at the end, would be something like 300 ms for "provide_credentials", 50 ms for GetItem and 100 ms for PutItem.

[Oliboy50]

@jdisanti before I mark this discussion as resolved (since you created an issue about it), have you noticed my last message where I post my current AWS SDK usage?
I'm not so sure to understand how your first message would help me regarding my current SDK usage.
Thanks

[jdisanti]

I saw in the RFC that the SDK also supports environment variables to configure timeouts (i.e. AWS_CONNECT_TIMEOUT, AWS_API_CALL_ATTEMPT_TIMEOUT, etc.)...

That RFC is outdated and those env vars no longer exist since they weren't standard across the other language SDKs.

I'm not so sure to understand how your first message would help me regarding my current SDK usage.

By explicitly setting the credentials provider, it will use separate connector timeout settings than what are given to the builder. If you don't explicitly set the credentials provider, then the timeout settings given to the config builder feed into the default that gets created.

[Oliboy50]

That RFC is outdated and those env vars no longer exist

I see 👌

By explicitly setting the credentials provider, it will use separate connector timeout settings than what are given to the builder.

I understand that but, if you look at my piece of code, you should see that I don't give timeout settings to the aws_config builder. Instead I set timeout settings on the aws_sdk_dynamodb client builder, so the credentials_provider should already has its own timeout settings (the aws_config default ones), don't you think so?

Anyway, do you know if this would be possible to not refresh credentials while initiating a request to DynamoDB, but instead refresh credentials from another thread every 15min for instance (so that my application callers do not have to wait 300ms sometimes instead of 5ms)?

[jdisanti]

I understand that but, if you look at my piece of code, you should see that I don't give timeout settings to the aws_config builder. Instead I set timeout settings on the aws_sdk_dynamodb client builder, so the credentials_provider should already has its own timeout settings (the aws_config default ones), don't you think so?

Yeah, that sounds right to me.

Anyway, do you know if this would be possible to not refresh credentials while initiating a request to DynamoDB, but instead refresh credentials from another thread every 15min for instance (so that my application callers do not have to wait 300ms sometimes instead of 5ms)?

You're welcome to file a feature request for this.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.