-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[performance](load) increase max_broker_concurrency to 100 #44929
[performance](load) increase max_broker_concurrency to 100 #44929
Conversation
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
run buildall |
run buildall |
TPC-H: Total hot run time: 39654 ms
|
TPC-DS: Total hot run time: 197613 ms
|
ClickBench: Total hot run time: 32.89 s
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
run buildall |
TPC-H: Total hot run time: 40259 ms
|
TPC-DS: Total hot run time: 197463 ms
|
ClickBench: Total hot run time: 33.27 s
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Increase default `max_broker_concurrency` to 100 to improve broker load performance. This option will affect the max number of scan / sink instances allowed in a broker load. ``` parallel instance = min(max_broker_concurrency, source file size / min_bytes_per_broker_scanner, num backends * load_parallelism) ``` S3 load time of tpcds_1000g catalog_sales: * before: 438s * after: 225s
Increase default `max_broker_concurrency` to 100 to improve broker load performance. This option will affect the max number of scan / sink instances allowed in a broker load. ``` parallel instance = min(max_broker_concurrency, source file size / min_bytes_per_broker_scanner, num backends * load_parallelism) ``` S3 load time of tpcds_1000g catalog_sales: * before: 438s * after: 225s
…#44929 (#45238) Cherry-picked from #44929 Co-authored-by: Kaijie Chen <[email protected]>
What problem does this PR solve?
Increase default
max_broker_concurrency
to 100 to improve broker load performance.This option will affect the max number of scan / sink instances allowed in a broker load.
S3 load time of tpcds_1000g catalog_sales:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)