Reduce Powertools Cold Start Impact

[barreeeiroo]

I'm building a fully serverless application, but I've noticed that my Lambdas suffer a huge cold-start impact on every AWS library. Despite enabling both Function Concurrency at the Function level and Provisioned Concurrency at the Alias level, I saw no difference in terms of first-request performance.

For example, see this Authorizer trace:

As you can see, accessing Secrets using AWS Powertools is taking way more than expected, impacting the actual Function execution time (even after the Lambda cold start finished).
Also, I've seen this issue affecting not only the AWS Powertools library but also PynamoDB. As you can see, the first call for each Table results in that "initialization" time, being way more than it should be. Subsequent calls are shorter and meet the expected time.

Another example is the Function trace (after the Authorizer finishes):

Again, initializing the GetParameter call from AWS Powertools is taking way too much time.

I'd like to know how to reduce these initialization times for AWS API calls. It is not an issue only with AWS Powertools, but also with any library invoking these APIs (like PynamoDB).
Thanks!

[heitorlessa]

Hey @barreeeiroo - that's unusual, at most, AWS SDK would incur ~200ms for creating a SDK Client + TLS Handshake (Session creation), though I've seen north of 300 lately if using a newer SDK version.

X-Ray SDK will add at worst 300-400ms (uses AWS SDK + some non-optimised path).

This should be amortised by a warm start. Higher memory functions have a lower cold start because of the CPU Slicing Lambda does, so you have more room for cryptography operations AWS SDK and TLS require -- Lambda PowerTuning can be a pragmatic way to find what optimum looks like.

Powertools being a thin wrapper, specially in Parameters, it'll only inherit what SDK and X-Ray does, and slow down on call stacks and control flows (~10-50ms at worst).

That being said, we can always improve and we will stop at nothing to achieve it given the bandwidth.

For actionable steps, would you be able to share a sample repo without sensitive data so we can try to reproduce? Please keep the same memory, runtime, AWS Region (round trip latency), settings you're using and such.

Until then, I'd suggest using a mix of Tuna (cold start profiler), and CodeGuru or any line profiler like pybenchmark locally.

There was a discussion in AWS SDK to ship compact models which would increase cold start by at least 50ms per client (decompress model at client creation) - your sample will help us track down dependencies, and make a PR to improve any non-optimal areas.

Meanwhile, if you can tell us what your dependencies look like (I'll exclude PynamoDB to keep it simple), what your function memory setting and event source are, we can setup a POC on our side.

PS: This week is tough, but also happy to setup a call and profile this through with you (aws-lambda-powertools-feedback@), and we'll send you a link with slots available.

Thanks a lot!

[heitorlessa]

Additional material if you prefer reading first to understand boto initialisation in more detail with numbers

https://www.tecracer.com/blog/2021/02/how-boto3-impacts-the-cold-start-times-of-your-lambda-functions.html

https://ben11kehoe.medium.com/boto3-sessions-and-why-you-should-use-them-9b094eb5ca8e

You can then create a single boto3 session at the global scope, and pass to the get_secret Powertools API via boto3_session parameter - this will amortize both PynamoDB and Powertools calls, since a boto3 default session is cached, and we can create new clients from an existing session.

https://awslabs.github.io/aws-lambda-powertools-python/2.7.1/utilities/parameters/#customizing-boto-configuration

Another common issue to this is STS where the default endpoint is in US East, but won't get to that now until profiling and debugging logs are available to repro.

[barreeeiroo]

Hi @heitorlessa, thanks for such detailed responses!
My current week is also pretty busy, so no worries.

I'll try through this weekend the profilers you mentioned (Tuna ad CodeGuru) and better identify the root causes. I'll also try to replicate the issue with some "simpler" code so I can provide a snippet.

Just to provide an initial idea of the conditions, it is using 128MB, Python 3.9, x86 architecture and Spain (eu-south-2) region. The event source is an API Gateway Proxy integration and API Token Authorizer integration (no v2, the classic ones).
The dependencies are the following:

aws-lambda-powertools[all]~=2.5.0
boto3~=1.26.41
boto3-stubs[s3,stepfunctions,sts]~=1.26.41
cachetools~=5.3.0
datamodel-code-generator~=0.14.1
holidays~=0.18
msal~=1.20.0
pydantic[email,dotenv]~=1.10.4
pyjwt~=2.6.0
pynamodb~=5.3.4
pytest~=7.2.0
pytest-cov~=4.0.0
python-dateutil~=2.8.2
requests~=2.28.1

You can then create a single boto3 session at the global scope, and pass to the get_secret Powertools API via boto3_session parameter - this will amortize both PynamoDB and Powertools calls, since a boto3 default session is cached, and we can create new clients from an existing session.

That sounds like a possible solution. I'll also investigate if I can pass an already existing client to PynamoDB, as it seems like they use one connection per Table.

Another common issue to this is STS where the default endpoint is in US East, but won't get to that now until profiling and debugging logs are available to repro.

Just found out about this. I'll try setting the AWS_STS_REGIONAL_ENDPOINTS env var and see if anything changes.

I'll investigate through the weekend and let you know about the findings. Thanks!

[barreeeiroo]

Uh, I have to say I was very curious about Lambda PowerTuning (as I recall hearing a few months ago), so I went to test it out now. And it seems like the main issue was those 128MB of RAM...

Lambda PowerTuning suggested increasing it to around 512MB-1024MB (the impact on cost was nearly no difference as per the invocation duration).
After increasing it to 1024MB, I could see that on a cold start, there was still a 2s-2.5s initialization (expected), but the invocation time was considerably reduced from 4.5s in the Authorizer to 0.6s and from 7.5s in the Lambda to 1.0s, which is a huge gain!

I'll keep investigating through the weekend.

[barreeeiroo]

Hey, so I've been playing a bit with some Lambda configurations and other tracer tools, and I was able to confirm the main issue was the memory size. In terms of Powertools, I was able to reduce that initialization time of 1s to around 250ms as you mentioned in your post.
Now I believe it's a matter of optimizing the code and calls, as most of the issue is now on DynamoDB as their Model class uses one connection per Table (rather than a global connection), resulting in ~200ms * n_tables called by each Lambda function.
Thanks a lot for your help!