Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(datasource): supports creating and testing OSS data sources #3879

Open
wants to merge 19 commits into
base: dev/4.3.x
Choose a base branch
from

Conversation

guowl3
Copy link
Collaborator

@guowl3 guowl3 commented Nov 20, 2024

What type of PR is this?

type-feature

What this PR does / why we need it:

  1. Supports creating OSS datasources but is only used for the target datasource of the archive task
  2. Supports testing OSS datasources

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Additional documentation e.g., usage docs, etc.:


@guowl3 guowl3 added the type-feature The functionality to be implemented label Nov 20, 2024
@guowl3 guowl3 added this to the ODC 4.3.3 milestone Nov 20, 2024
@guowl3 guowl3 self-assigned this Nov 20, 2024
@@ -97,7 +99,9 @@ public ConnectionTestResult test(@NotNull @Valid TestConnectionReq req) {
if (req.getAccountType() == ConnectionAccountType.SYS_READ) {
connectionConfig.setDefaultSchema(null);
}
return test(connectionConfig);
return connectionConfig.getDialectType() == DialectType.FILE_SYSTEM
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this switch is not easy to extend, if a new type involved, it may missed to change this code.
can we designed a new way to force the user feel this method?

*/

@Component
public class FileSystemConnectionTesting {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing means in do test process. it's a verb, use noun is more suitable

@Component
public class FileSystemConnectionTesting {

private static final String COS_ENDPOINT_REGEX = "cos\\.(\\w+-\\w+)\\.myqcloud\\.com";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as this regex will be compiled to Pattern. declare pattern, please confirm Pattern is thread safe

return ConnectionTestResult.success(config.getType());
} catch (CloudException e) {
if (e.getCause() instanceof OSSException) {
OSSException cause = (OSSException) e.getCause();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aware of npe and class cast exception

public PutObjectResult putObject(String bucketName, String key, InputStream in, ObjectMetadata metadata)
throws CloudException {
return callAmazonMethod("Put object", () -> {
com.amazonaws.services.s3.model.ObjectMetadata objectMetadata = toS3(metadata);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this package prefix needed?

private String getEndPointRegex(ConnectType type) {
switch (type) {
case COS:
return COS_ENDPOINT_REGEX;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as you have so many switch case around your code, reconstruct may needed.
let enum's member hold this fields. or put what programmer must implement to enums class to aware if you want to add some type ,you should do like this way

@guowl3 guowl3 requested a review from kiko-art as a code owner December 16, 2024 09:08
@guowl3 guowl3 requested a review from CHLK as a code owner December 17, 2024 04:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-feature The functionality to be implemented
Projects
Status: In Test
Development

Successfully merging this pull request may close these issues.

2 participants