Proper Client Use and Configuration for Scale

[eblackw2]

I’m getting really slow performance when sending messages to SQS from my rust Lambda as I test with a KafkaEvent containing a realistic number of records. When I send one or two records, the messages individually send in 30-50ms. When I use a full sample payload with about 100 records, each message takes about 400ms – 1.3s. I’m guessing that it has something to do with how we use the client because I was seeing a similar issue with requesting a secret until realizing I only needed to request it once. Perhaps we need to tamper down the batch size from the Kafka process because it’s just using the default, but that seems like a big slow down regardless.

I was hoping someone could help us with best practices on this. The AWS chat representative pointed me to an article on horizontal scaling that suggested using more clients or clients with more threads. I tried instantiating a new client for every message and saw no improvement, and I don’t see a way to increase the number of threads that a client has to use.

[jdisanti]

The Rust SDK clients are intended to be reused (even across threads), and client creation is expensive, so you definitely don't want to create a new client for each message. You should create the client once on Lambda start-up so that it becomes part of the cold start time.

Some questions:

• Are you compiling in release mode?
• How large, approximately, are these messages?
• When you receive a payload of 100 records, are you kicking off a separate SQS call for each message in a tight loop? If so, you may want to use a semaphore to rate limit the calls. Without one, the async runtime will attempt to do all of them simultaneously, which can cause performance issues and even lead to server-side throttling.

[eblackw2]

• Yes, this is getting compiled in release mode
• Looking at the SQS monitoring graphs, it looks like the messages are normally around .8 - 1.3 kb
• We're handling each record in a thread and doing some decoding/decrypting/transforming before sending, but that processing happens pretty quickly which means that yeah, a lot of the sends are happening on top of each other. I've introduced a semaphore which does appear to have helped some. I'm going to tinker with the permit number to see what works best. Do you have any suggestions based on limitations the client may have? Or are the limiting factors more going to be decided by the lambda and SQS configuration?

[ymwjbxxq]

Hi,

Inside the main function you should initialize the client;

#[tokio::main]
async fn main() -> Result<(), Error> {
   let sqs_client = aws_sdk_sqs::Client::new(&config);
   let app_client = MyDIStruct::builder()
            .sqs_client(sqs_client)
            .build();

    lambda_http::run(service_fn(|event: Request| {
        my_handler(&app_client, event)
    })).await?;
    Ok(())
}

and now you can pass it to the handler and you use it from there.

About SQS:

• If you have many messages I would say use the send_message_batch for better latency.
• If you need to use the send_message I got throttled but sending thousands at the same time (different runtime)

About Latency:
you can setup each client with some timeout, for example:

let timeout_config = timeout::Config::new()
        .with_api_timeouts(
            timeout::Api::new()
                .with_call_timeout(TriState::Set(Duration::from_millis(150)))
                .with_call_attempt_timeout(TriState::Set(Duration::from_millis(50))),
    let config = aws_config::from_env()
        .retry_config(RetryConfig::new().with_max_attempts(3))
        .timeout_config(timeout_config)
        .load()
        .await;