Write File to S3 results in Exception

[uwegeercken]

Description

A description of what the bug is.
I have a basic flow where Objects are read from S3, processed and written back to S3 to a different bucket using a Write File Step.

The Write File Step is configured as shown below:

The following error occurs in the Write File Step when running the flow:
`The request you provided implies functionality that is not implemented. (Service: S3, Status Code: 501, Request ID: 1643103651280481, Extended Request ID: 12525043)

INFO:

at: /home/boxuser/development/tweakstreet/pseudonimize_s3_files.dfl:Write to File: S3 Storage Grid

code: RUNTIME_ERROR

message: The request you provided implies functionality that is not implemented. (Service: S3, Status Code: 501, Request ID: 1643103651280481, Extended Request ID: 12525043)

value: nil

ERROR:

code: software.amazon.awssdk.services.s3.model.S3Exception

message: The request you provided implies functionality that is not implemented. (Service: S3, Status Code: 501, Request ID: 1643103651280481, Extended Request ID: 12525043)

TRACE:

software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)

software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106)

software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84)

software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42)

software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94)

software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$6(BaseClientHandler.java:270)

software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)

software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)

software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)

software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)

software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)

software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)

software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)

software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)

software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)

software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)

software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)

software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)

software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)

software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)

software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)

software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)

software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)

software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)

software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)

software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)

software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)

software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)

software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)

software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)

software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:135)

software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:161)

software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:114)

software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169)

software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95)

software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)

software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)

software.amazon.awssdk.services.s3.DefaultS3Client.uploadPart(DefaultS3Client.java:10018)

com.twineworks.tweakstreet.library.storage.s3.s3upload.StreamTransferManager.uploadStreamPart(StreamTransferManager.java:549)

com.twineworks.tweakstreet.library.storage.s3.s3upload.StreamTransferManager.complete(StreamTransferManager.java:356)

com.twineworks.tweakstreet.library.storage.s3.s3upload.S3OutputStream.close(S3OutputStream.java:41)

java.base/sun.nio.cs.StreamEncoder.implClose(Unknown Source)

java.base/sun.nio.cs.StreamEncoder.close(Unknown Source)

java.base/java.io.OutputStreamWriter.close(Unknown Source)

java.base/java.io.BufferedWriter.close(Unknown Source)

com.twineworks.tweakstreet.features.core.steps.writefile.WriteFileStep.closeFile(WriteFileStep.java:123)

com.twineworks.tweakstreet.features.core.steps.writefile.WriteFileStep.applySettings(WriteFileStep.java:63)

com.twineworks.tweakstreet.features.core.steps.writefile.WriteFileStep.processRow(WriteFileStep.java:99)

com.twineworks.tweakstreet.engine.plugins.extensions.steps.passthrough.PassThroughAdapterProcessor.work(PassThroughAdapterProcessor.java:78)

com.twineworks.tweakstreet.engine.exec.data.MapShell.work(MapShell.java:358)

com.twineworks.tweakstreet.engine.exec.data.ExecutableDataFlow$DataFlowWorker.run(ExecutableDataFlow.java:829)

java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

java.base/java.util.concurrent.FutureTask.run(Unknown Source)

java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

java.base/java.util.concurrent.FutureTask.run(Unknown Source)

java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

java.base/java.lang.Thread.run(Unknown Source)`

I have tried to write the files in binary mode, but same issue. The credentials for incomming and outgoing objects are the same, so I don't think there is a problem with that.

Environment

• Tweakstreet Version 1.19.6
• Operating system and version: Fedora 35
• Additional environment information such as database or driver versions