S3TransferManager directoryDownload downloads incomplete files without failing

[bugrabenturk]

Describe the bug

Sometimes S3TransferManager does not actually download the complete file and it does not marks that as failed transfer instead it marks as complete transfer, it does not even throws any exception.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Should download the complete file, if it does not downloads the complete file should trigger as failed download or throw the corresponded exception.

Current Behavior

Does not downloads the complete file and does not triggers as failed download instead marks as download completed without throwing any exception.

First I thought that it's happening because of the assuming request's token expiration, since its set to 3600 secs (1 hour), i thought that when the download starts then the token expires and somehow it does fails the transfer but still marks as completed.

So based on that idea, i've created the session right before every transfer action, so it still did not help and problem still occurred.

However, when I have checked the logs i couldn't be able to see much but this is the last part thats been downloaded, rest of the parts are not downloaded at all.
[DEBUG] [2024-09-26T15:39:52Z] [00007fbe09e24b38] [S3MetaRequest] - id=0x7fbe07ff2300: 132 out of 998 parts have completed.

I'm attaching the log file, since it was very big i'm adding the part that its stopped downloading at the part of 132.

svhf.log

Reproduction Steps

We're using STSClient to gather temporary AWS Session credentials with assuming to the corresponding role and then doing the download operation

`

                stsClient = StsClient.builder()
                    .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)))
                    .build()

                AssumeRoleRequest assumeRoleRequest = AssumeRoleRequest.builder()
                    .roleArn(roleArn)
                    .roleSessionName(roleSessionName)
                    .externalId(externalId)
                    .durationSeconds(3600)
                    .build();

                AssumeRoleResponse assumeRoleResponse = stsClient.assumeRole(assumeRoleRequest);
                Credentials stsCredentials = assumeRoleResponse.credentials();

                awsSessionCredentials = AwsSessionCredentials.create(
                        stsCredentials.accessKeyId(),
                        stsCredentials.secretAccessKey(),
                        stsCredentials.sessionToken()
                );

        s3AsyncClient = S3AsyncClient.crtBuilder()
                .credentialsProvider(StaticCredentialsProvider.create(awsSessionCredentials))
                .region(Region.EU_WEST_1)
                .maxConcurrency(10)
                .targetThroughputInGbps(5.0)
                .minimumPartSizeInBytes(80 * SizeConstant.MB)
                .retryConfiguration(r -> r.numRetries(5))
                .build();

        s3TransferManager = S3TransferManager.builder()
                .s3Client(s3AsyncClient)
                .build();`


       DirectoryDownload directoryDownload = s3TransferManager.downloadDirectory(DownloadDirectoryRequest.builder()
                .destination(assetDir)
                .bucket(amazonS3URI.getBucket())
                .listObjectsV2RequestTransformer(builder -> builder.prefix(amazonS3URI.getKey()))
                .build());

        CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();

`

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.28.6

AWS CRT version used

0.31.1

JDK version used

21

Operating System and version

Alpine Linux v3.20