Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support to customize httpClientBuilder for s3 client #50

Merged
merged 25 commits into from
Jan 17, 2025

Conversation

yidian1997
Copy link
Contributor

@yidian1997 yidian1997 commented Nov 26, 2024

Pull Request

What

Add a new parameter S3_SDK_HTTP_CLIENT_BUILDER_CONFIG
This is a class to customize the httpClient used by the s3 client

Why

Our use case is:
We have a minio server with https endpoint. And the minio server uses self-signed certificate to do the TLS.
When our kafka client uses this lib to put object to that minio, we need to make sure the CA certificate of the minio server should in the truststore used by the s3 client. This is the reason we want to customize the httpClient.

Another limitation is that we need add the CA certificate to the truststore in the code instead of using the keytool command line. Since the command line is hard to automate in our use case. So finally I choose add this support in the this lib.

How

  1. Add the S3_SDK_HTTP_CLIENT_BUILDER_CONFIG class parameter
  2. Add method getAmazonSdkHttpClientBuilderInstance() to get the object of SdkHttpClient.Builder by reflection
  3. Add some logic in the createAmazonS3Client() method to config the SdkHttpClient.Builder

Test

I added this parameter with DefaultTlsTrustManagersProvider in the UT and it works fine.

@yidian1997 yidian1997 marked this pull request as ready for review December 5, 2024 03:46
@yidian1997 yidian1997 marked this pull request as draft December 5, 2024 03:47
@yidian1997 yidian1997 marked this pull request as ready for review December 5, 2024 06:20
@yidian1997 yidian1997 changed the title Add support to customize tlsTrustManagersProvider for s3 client Add support to customize httpClientBuilder for s3 client Dec 28, 2024
@yidian1997
Copy link
Contributor Author

Hi @philipp94831 I update the PR according to your comments. Can you take a look when you have time. Thanks!

Copy link
Member

@philipp94831 philipp94831 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should do the trick and then we are good to merge :-)
Remove_NoSdkHttpClientBuilder.patch

@philipp94831 philipp94831 merged commit 379eed2 into bakdata:master Jan 17, 2025
@philipp94831
Copy link
Member

Thank you very much for your contribution and patience. I create a release later today

@yidian1997
Copy link
Contributor Author

Also thanks for your review, that is really helpful!

@philipp94831
Copy link
Member

2.10.0 should be available

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants