Multipart upload fails when using SHA1 checksum

[maloshuk]

Describe the bug

I'm trying to use SHA1 checksums when uploading files to amazon S3 bucket. It works when I upload small files. But with large files and Multipart Upload, process fails to complete.

Expected Behavior

Upload to AWS S3 bucket with SHA1 checksum algorithm enabled ends with success.

Current Behavior

Upload fails with 400 response code:

400 Bad Request

The upload was created using a sha1 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request.

Relevant part of the stack trace:

An exception occurred while completing a multipart upload: Error executing "CompleteMultipartUpload" on "https://BUCKET.s3.amazonaws.com/contents/myfile.dat?uploadId=ID"; AWS HTTP error: Client error: `POST https://BUCKET.s3.amazonaws.com/contents/myfile.dat?uploadId=ID` resulted in a `400 Bad Request` response:
<Error><Code>InvalidRequest</Code><Message>The upload was created using a sha1 checksum. The complete request must inclu (truncated...)
 InvalidRequest (client): The upload was created using a sha1 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request. - <Error><Code>InvalidRequest</Code><Message>The upload was created using a sha1 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request.</Message><RequestId>VS9J72F799MSGTG1</RequestId><HostId>Vqgw2uJseIdwDBKKay3xT15fsADJLhd7bpqW+0k6denCuisH1pQzKKrKWYQMMQqcND144szIHVw=</HostId></Error> {"userId":25,"exception":"[object] (Aws\\S3\\Exception\\S3MultipartUploadException(code: 0): An exception occurred while completing a multipart upload: Error executing \"CompleteMultipartUpload\" on \"https://BUCKET.s3.amazonaws.com/contents/myfile.dat?uploadId=ID\"; AWS HTTP error: Client error: `POST https://BUCKET.s3.amazonaws.com/contents/myfile.dat?uploadId=ID` resulted in a `400 Bad Request` response:
<Error><Code>InvalidRequest</Code><Message>The upload was created using a sha1 checksum. The complete request must inclu (truncated...)
 InvalidRequest (client): The upload was created using a sha1 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request. - <Error><Code>InvalidRequest</Code><Message>The upload was created using a sha1 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request.</Message><RequestId>VS9J72F799MSGTG1</RequestId><HostId>Vqgw2uJseIdwDBKKay3xT15fsADJLhd7bpqW+0k6denCuisH1pQzKKrKWYQMMQqcND144szIHVw=</HostId></Error> at ROOT\\vendor\\aws\\aws-sdk-php\\src\\Multipart\\AbstractUploadManager.php:147)
[stacktrace]
#0 ROOT\\vendor\\aws\\aws-sdk-php\\src\\Multipart\\AbstractUploadManager.php(156): Aws\\Multipart\\AbstractUploadManager->transformException(Object(Aws\\S3\\Exception\\S3Exception))
#1 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(204): Aws\\Multipart\\AbstractUploadManager->Aws\\Multipart\\{closure}(Object(Aws\\S3\\Exception\\S3Exception))
#2 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(153): GuzzleHttp\\Promise\\Promise::callHandler(2, Object(Aws\\S3\\Exception\\S3Exception), NULL)
#3 ROOT\\vendor\\guzzlehttp\\promises\\src\\TaskQueue.php(48): GuzzleHttp\\Promise\\Promise::GuzzleHttp\\Promise\\{closure}()
#4 ROOT\\vendor\\guzzlehttp\\guzzle\\src\\Handler\\CurlMultiHandler.php(159): GuzzleHttp\\Promise\\TaskQueue->run()
#5 ROOT\\vendor\\guzzlehttp\\guzzle\\src\\Handler\\CurlMultiHandler.php(184): GuzzleHttp\\Handler\\CurlMultiHandler->tick()
#6 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(248): GuzzleHttp\\Handler\\CurlMultiHandler->execute(true)
#7 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(224): GuzzleHttp\\Promise\\Promise->invokeWaitFn()
#8 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(269): GuzzleHttp\\Promise\\Promise->waitIfPending()
#9 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(226): GuzzleHttp\\Promise\\Promise->invokeWaitList()
#10 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(269): GuzzleHttp\\Promise\\Promise->waitIfPending()
#11 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(226): GuzzleHttp\\Promise\\Promise->invokeWaitList()
#12 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()
#13 ROOT\\vendor\\guzzlehttp\\promises\\src\\Coroutine.php(67): GuzzleHttp\\Promise\\Promise->wait()
#14 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(248): GuzzleHttp\\Promise\\Coroutine->GuzzleHttp\\Promise\\{closure}(true)
#15 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(224): GuzzleHttp\\Promise\\Promise->invokeWaitFn()
#16 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(269): GuzzleHttp\\Promise\\Promise->waitIfPending()
#17 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(226): GuzzleHttp\\Promise\\Promise->invokeWaitList()
#18 ROOT\\vendor\\guzzlehttp\\promises\\src\\Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()
#19 ROOT\\vendor\\aws\\aws-sdk-php\\src\\S3\\S3ClientTrait.php(35): GuzzleHttp\\Promise\\Promise->wait()
#20 ROOT\
outes\\web.php(542): Aws\\S3\\S3Client->upload('ws-company-55-6...', 'contents/myfile...', Resource id #8, 'private', Array)

Reproduction Steps

/* create s3 client  */
$s3Client = new \Aws\S3\S3Client( 
  [
         'version'=> 'latest',
         'region' => REGION,
         'credentials' => array(
             'key' => AWS_KEY,
             'secret' => AWS_SECRET
         )
 ]);
/* open file stream large enough to initiate multipart upload ( 33 MB in my example)  */
$stream = fopen("d:/temp/G320.MOV", "r");
/* send file to S3  with SHA1 checksum enabled */
$s3client->upload('ws-company-55-63fccce1bcddb',  'contents/myfile.dat',  $stream, 'private', 
            // pass params to enable SHA1 checksum calculation
            ['params' => [
                              "ChecksumAlgorithm" => "SHA1" 
                         ]
             ]
);

Possible Solution

No response

Additional Information/Context

No response

SDK version used

3.261.0

Environment details (Version of PHP (php -v)? OS name and version, etc.)

php 7.4.9, Windows 10, Laravel 8

[yenfryherrerafeliz]

Hi @maloshuk, thanks for opening this issue. I can confirm that this is a bug. The reason for this behavior is because the checksum parameter needs to be provided in the parts that are being sent in the complete upload command, but we can see here that just the partNumber and the eTag parameters are given.
Basically, instead of:

protected function handleResult(CommandInterface $command, ResultInterface $result)
{
    $this->getState()->markPartAsUploaded($command['PartNumber'], [
        'PartNumber' => $command['PartNumber'],
        'ETag'       => $this->extractETag($result),
    ]);
}

We should have something that would look as follow:

protected function handleResult(CommandInterface $command, ResultInterface $result)
{
    $this->getState()->markPartAsUploaded($command['PartNumber'], [
        'PartNumber' => $command['PartNumber'],
        'ETag'       => $this->extractETag($result),
        'ChecksumSHA1' => $result['ChecksumSHA1']
    ]);
}

Actually, we would need to review which other parameters we should consider to include when completing the upload.

I will mark this issue as needs-review so we can address this further.

Thanks!

[yenfryherrerafeliz]

Hi @maloshuk, this has been fixed by the following PR. Please feel free of opening a new issue for anything else that we may help with.

Thanks!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.