Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement](Nereids) Support create routine load command #43930

Merged
merged 17 commits into from
Nov 27, 2024
Merged

[Improvement](Nereids) Support create routine load command #43930

merged 17 commits into from
Nov 27, 2024

Conversation

LiBinfeng-01
Copy link
Collaborator

@LiBinfeng-01 LiBinfeng-01 commented Nov 14, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #42805

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

1 similar comment
@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

2 similar comments
@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01 LiBinfeng-01 force-pushed the support_load_command branch 2 times, most recently from 77eac8c to 9480305 Compare November 21, 2024 12:26
@LiBinfeng-01
Copy link
Collaborator Author

run buildall

starocean999
starocean999 reviewed Nov 22, 2024
View reviewed changes
.../src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateRoutineLoadInfo.java Outdated
}

private void checkDBTable(ConnectContext ctx) throws AnalysisException {
dbName = labelName.getDbName();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to create a new LabelNameInfo class and implement analyze or validate method like TableNameInfo. Because LabelNameInfo may be used in other commands, and the validate can be called in multiple place.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

starocean999
starocean999 reviewed Nov 22, 2024
View reviewed changes
.../src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateRoutineLoadInfo.java
}
}
}
routineLoadDesc = new RoutineLoadDesc(columnSeparator, lineDelimiter, importColumnsStmt,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to create a new RoutineLoadInfo class and remove the dependence of ImportColumnsStmt, ImportWhereStmt. Just keep Expression as its member variable. And add setRoutineLoadDesc(RoutineLoadInfo info) into RoutineLoadJob to use the new class.(you can do it in next pr)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it

starocean999
starocean999 reviewed Nov 22, 2024
View reviewed changes
.../src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateRoutineLoadInfo.java
List<String> nameParts = Lists.newArrayList();
nameParts.add(dbName);
nameParts.add(tableName);
Plan unboundRelation = new UnboundRelation(StatementScopeIdGenerator.newRelationId(), nameParts);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bind an Expression and translate to legacy Expr is used very often. Better to put this into a new ExpressionToExprUtil class

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the initialize of translation differs a lot, so it is hard to extract one common analyzer and translator. In next pr using it I would extract common codes

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run feut

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run feut

1 similar comment
@LiBinfeng-01
Copy link
Collaborator Author

run feut

@LiBinfeng-01
Copy link
Collaborator Author

run cloud_p0

1 similar comment
@LiBinfeng-01
Copy link
Collaborator Author

run cloud_p0

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 27, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@starocean999 starocean999 merged commit d3276d9 into apache:master Nov 27, 2024
23 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 approved these changes

@englefly englefly englefly approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Enhancement] (nereids)implement createRoutineLoadJobCommand in nereids
4 participants
@LiBinfeng-01 @doris-robot @englefly @starocean999