Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](paimon)Optimize the storage location of the serialized paimon table #44274

Merged
merged 2 commits into from
Nov 25, 2024
Merged

[opt](paimon)Optimize the storage location of the serialized paimon table #44274

merged 2 commits into from
Nov 25, 2024

Conversation

wuwenchi
Copy link
Contributor

What problem does this PR solve?

Related PR: #43167

Previously, the serialized paimon table was placed in TPaimonFileDesc, which resulted in each split storing a serialized paimon table. It will wasted a lot of space if a table has lots of split.
Now change the storage location of the serialized paimon table so that all splits of a table can share the data.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wuwenchi
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.01% (9899/26042)
Line Coverage: 29.19% (82781/283568)
Region Coverage: 28.33% (42515/150071)
Branch Coverage: 24.89% (21551/86592)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2666998c527f12686c4708895b1707456d61e24a_2666998c527f12686c4708895b1707456d61e24a/report/index.html

@wuwenchi
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.02% (9898/26035)
Line Coverage: 29.20% (82786/283537)
Region Coverage: 28.33% (42518/150055)
Branch Coverage: 24.90% (21556/86584)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4ebaebdc73f1ddd5f779932a15d0b7450801a2da_4ebaebdc73f1ddd5f779932a15d0b7450801a2da/report/index.html

@wuwenchi
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.01% (9897/26035)
Line Coverage: 29.20% (82780/283540)
Region Coverage: 28.33% (42508/150055)
Branch Coverage: 24.90% (21558/86584)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4bcf74246333f9d7566c6c1a2feb372a1168c33e_4bcf74246333f9d7566c6c1a2feb372a1168c33e/report/index.html

@wuwenchi
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.03% (9902/26038)
Line Coverage: 29.22% (82845/283539)
Region Coverage: 28.34% (42527/150055)
Branch Coverage: 24.90% (21563/86582)
Coverage Report: http://coverage.selectdb-in.cc/coverage/56e3cc571ac9f223e311e6ee14cfe3622fe0a6f2_56e3cc571ac9f223e311e6ee14cfe3622fe0a6f2/report/index.html

@wuwenchi
Copy link
Contributor Author

run p0

@wuwenchi
Copy link
Contributor Author

run cloud_p0

@wuwenchi
Copy link
Contributor Author

run external

morningman
morningman reviewed Nov 22, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java Outdated
@@ -152,6 +158,9 @@ private void setPaimonParams(TFileRangeDesc rangeDesc, PaimonSplit paimonSplit)
if (split != null) {
// use jni reader
fileDesc.setPaimonSplit(encodeObjectToString(split));
if (!serializedTable.isPresent()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to init this serializedTable in doInitialize().

@wuwenchi
Copy link
Contributor Author

run buildall

@wuwenchi
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.31% (9979/26048)
Line Coverage: 29.43% (83515/283806)
Region Coverage: 28.58% (42975/150354)
Branch Coverage: 25.17% (21834/86742)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c48addb87a6f19165b4e19bc0bfff6b1537a3410_c48addb87a6f19165b4e19bc0bfff6b1537a3410/report/index.html

morningman
morningman approved these changes Nov 25, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 25, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

suxiaogang223
suxiaogang223 approved these changes Nov 25, 2024
View reviewed changes
Copy link
Contributor

@suxiaogang223 suxiaogang223 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@morningman morningman merged commit 3c8d752 into apache:master Nov 25, 2024
29 of 32 checks passed
wuwenchi added a commit to wuwenchi/doris_new that referenced this pull request Nov 27, 2024
@wuwenchi
[opt](paimon)Optimize the storage location of the serialized paimon t…
23b814b 
…able (apache#44274)

Related PR: apache#43167

Previously, the serialized paimon table was placed in `TPaimonFileDesc`,
which resulted in each split storing a serialized paimon table. It will
wasted a lot of space if a table has lots of split.
Now change the storage location of the serialized paimon table so that
all splits of a table can share the data.
wuwenchi added a commit to wuwenchi/doris_new that referenced this pull request Nov 27, 2024
@wuwenchi
[opt](paimon)Optimize the storage location of the serialized paimon t…
f203f03 
…able (apache#44274)

Related PR: apache#43167

Previously, the serialized paimon table was placed in `TPaimonFileDesc`,
which resulted in each split storing a serialized paimon table. It will
wasted a lot of space if a table has lots of split.
Now change the storage location of the serialized paimon table so that
all splits of a table can share the data.
This was referenced Nov 27, 2024
Merged
Merged
morningman pushed a commit that referenced this pull request Nov 27, 2024
@wuwenchi
[opt](paimon)Optimize the storage location of the serialized paimon t…
88cfaed 
…able for 2.1 (#44274) (#44660)

bp: #44274
morningman pushed a commit that referenced this pull request Nov 27, 2024
@wuwenchi
[opt](paimon)Optimize the storage location of the serialized paimon t…
fbc103a 
…able for 3.0 (#44274) (#44664)

bp: #44274
@wuwenchi wuwenchi mentioned this pull request Dec 3, 2024
Merged
16 tasks
morningman pushed a commit that referenced this pull request Dec 6, 2024
@wuwenchi
[opt](paimon)Remove unnecessary fields (#44906)
0f74869 
### What problem does this PR solve?

Related PR: #44274

Problem Summary:

We have already set the serialization table in `TFileScanRangeParams`,
so there is no need to repeat it for each split.
github-actions bot pushed a commit that referenced this pull request Dec 6, 2024
@wuwenchi
[opt](paimon)Remove unnecessary fields (#44906)
8c4c203 
### What problem does this PR solve?

Related PR: #44274

Problem Summary:

We have already set the serialization table in `TFileScanRangeParams`,
so there is no need to repeat it for each split.
wuwenchi added a commit to wuwenchi/doris_new that referenced this pull request Dec 6, 2024
@wuwenchi
[opt](paimon)Remove unnecessary fields (apache#44906)
ec5a9d4 
### What problem does this PR solve?

Related PR: apache#44274

Problem Summary:

We have already set the serialization table in `TFileScanRangeParams`,
so there is no need to repeat it for each split.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@suxiaogang223 suxiaogang223 suxiaogang223 approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wuwenchi @doris-robot @morningman @suxiaogang223