Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[performance](load) increase max_broker_concurrency to 100 #44929

Merged
merged 4 commits into from
Dec 10, 2024

Conversation

kaijchen
Copy link
Contributor

@kaijchen kaijchen commented Dec 3, 2024

What problem does this PR solve?

Increase default max_broker_concurrency to 100 to improve broker load performance.
This option will affect the max number of scan / sink instances allowed in a broker load.

parallel instance = min(max_broker_concurrency,
                        source file size / min_bytes_per_broker_scanner,
                        num backends * load_parallelism)

S3 load time of tpcds_1000g catalog_sales:

  • before: 438s
  • after: 225s

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen kaijchen marked this pull request as ready for review December 6, 2024 07:00
@kaijchen kaijchen marked this pull request as draft December 6, 2024 07:01
@kaijchen kaijchen marked this pull request as ready for review December 9, 2024 08:35
@kaijchen kaijchen changed the title [performance](load) increase max_broker_concurrency to 24 [performance](load) increase max_broker_concurrency to 100 Dec 9, 2024
@kaijchen
Copy link
Contributor Author

kaijchen commented Dec 9, 2024

run buildall

@kaijchen
Copy link
Contributor Author

kaijchen commented Dec 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39654 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a92c919b510d3fa6678641ac5088c470c5388322, data reload: false

------ Round 1 ----------------------------------
q1	17569	7410	7237	7237
q2	2038	193	166	166
q3	10725	1063	1123	1063
q4	10570	743	687	687
q5	7603	2744	2654	2654
q6	239	147	144	144
q7	1006	614	590	590
q8	9238	1859	1906	1859
q9	6627	6526	6522	6522
q10	7034	2269	2292	2269
q11	484	277	252	252
q12	416	221	218	218
q13	17760	3021	2984	2984
q14	241	206	212	206
q15	564	529	524	524
q16	661	590	582	582
q17	961	522	484	484
q18	7189	6695	6562	6562
q19	1346	981	973	973
q20	449	180	175	175
q21	4016	3220	3196	3196
q22	384	313	307	307
Total cold run time: 107120 ms
Total hot run time: 39654 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7220	7202	7208	7202
q2	324	235	230	230
q3	2890	2797	2903	2797
q4	2057	1764	1817	1764
q5	5641	5696	5642	5642
q6	219	141	139	139
q7	2228	1780	1827	1780
q8	3348	3555	3474	3474
q9	9048	8981	9071	8981
q10	3606	3538	3512	3512
q11	594	503	513	503
q12	825	626	634	626
q13	13036	3205	3203	3203
q14	294	266	272	266
q15	591	527	535	527
q16	691	635	629	629
q17	1843	1638	1596	1596
q18	8271	7803	7719	7719
q19	1697	1608	1512	1512
q20	2126	1893	1857	1857
q21	5665	5360	5378	5360
q22	636	567	560	560
Total cold run time: 72850 ms
Total hot run time: 59879 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197613 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a92c919b510d3fa6678641ac5088c470c5388322, data reload: false

query1	1250	952	939	939
query2	6229	2041	2035	2035
query3	10981	4491	4396	4396
query4	67253	28522	23517	23517
query5	4883	445	454	445
query6	424	185	172	172
query7	5459	303	299	299
query8	310	235	241	235
query9	8477	2723	2731	2723
query10	443	262	262	262
query11	16917	15177	15870	15177
query12	175	102	104	102
query13	1473	460	410	410
query14	10359	7232	7609	7232
query15	204	193	195	193
query16	6750	525	486	486
query17	1021	582	562	562
query18	1994	306	295	295
query19	204	159	153	153
query20	117	109	145	109
query21	208	101	104	101
query22	4925	4705	4588	4588
query23	35565	34314	34445	34314
query24	5505	2572	2436	2436
query25	474	395	375	375
query26	637	159	152	152
query27	1945	283	288	283
query28	4479	2517	2491	2491
query29	654	421	410	410
query30	214	151	147	147
query31	1018	836	841	836
query32	65	53	52	52
query33	454	303	287	287
query34	945	504	535	504
query35	874	776	781	776
query36	1088	975	1008	975
query37	117	66	78	66
query38	4451	4370	4489	4370
query39	1521	1559	1483	1483
query40	205	100	99	99
query41	42	43	42	42
query42	115	97	104	97
query43	541	511	500	500
query44	1190	822	818	818
query45	184	174	165	165
query46	1158	736	752	736
query47	2073	1968	1972	1968
query48	433	312	332	312
query49	744	408	414	408
query50	834	397	396	396
query51	7424	7252	7076	7076
query52	97	84	85	84
query53	255	174	175	174
query54	490	383	378	378
query55	79	72	75	72
query56	258	231	238	231
query57	1241	1140	1143	1140
query58	219	210	203	203
query59	3296	3127	3137	3127
query60	265	238	238	238
query61	116	114	119	114
query62	836	665	661	661
query63	210	189	195	189
query64	1383	662	620	620
query65	3285	3234	3190	3190
query66	693	302	350	302
query67	16516	15926	15745	15745
query68	3903	561	543	543
query69	452	250	247	247
query70	1206	1145	1144	1144
query71	355	261	242	242
query72	6184	4089	4063	4063
query73	765	368	363	363
query74	10240	9002	9024	9002
query75	3417	2668	2671	2668
query76	1947	1082	1076	1076
query77	546	269	265	265
query78	10531	9559	9524	9524
query79	1346	604	584	584
query80	864	433	424	424
query81	481	237	237	237
query82	1322	115	114	114
query83	252	142	138	138
query84	280	66	73	66
query85	879	352	307	307
query86	357	297	312	297
query87	4730	4607	4641	4607
query88	3552	2224	2163	2163
query89	419	288	289	288
query90	2023	181	183	181
query91	139	105	164	105
query92	63	49	48	48
query93	1764	530	540	530
query94	819	294	271	271
query95	349	240	249	240
query96	617	286	273	273
query97	2951	2633	2695	2633
query98	215	194	198	194
query99	1720	1303	1338	1303
Total cold run time: 318932 ms
Total hot run time: 197613 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a92c919b510d3fa6678641ac5088c470c5388322, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.08	0.08
query4	1.61	0.11	0.11
query5	0.42	0.41	0.44
query6	1.18	0.72	0.68
query7	0.02	0.01	0.02
query8	0.04	0.03	0.02
query9	0.60	0.51	0.52
query10	0.57	0.55	0.56
query11	0.14	0.11	0.11
query12	0.12	0.11	0.12
query13	0.61	0.61	0.59
query14	2.83	2.82	2.80
query15	0.91	0.82	0.83
query16	0.39	0.41	0.38
query17	1.05	1.02	1.04
query18	0.22	0.21	0.20
query19	1.93	1.82	2.02
query20	0.01	0.00	0.01
query21	15.36	0.58	0.55
query22	2.91	2.24	1.82
query23	17.21	0.94	0.77
query24	2.70	1.29	1.68
query25	0.24	0.13	0.07
query26	0.55	0.14	0.13
query27	0.05	0.05	0.05
query28	10.18	1.10	1.08
query29	12.55	3.23	3.24
query30	0.25	0.07	0.07
query31	2.86	0.38	0.37
query32	3.28	0.47	0.46
query33	2.97	3.00	3.12
query34	17.02	4.48	4.43
query35	4.47	4.50	4.53
query36	0.68	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.74 s
Total hot run time: 32.89 s

Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 9, 2024
Copy link
Contributor

github-actions bot commented Dec 9, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Dec 9, 2024

PR approved by anyone and no changes requested.

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40259 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 03cf22587e5adcd0057a5541ef97db73f50d524a, data reload: false

------ Round 1 ----------------------------------
q1	17569	7703	7336	7336
q2	2041	171	175	171
q3	10692	1086	1190	1086
q4	10568	741	839	741
q5	7617	2752	2756	2752
q6	239	145	148	145
q7	994	655	595	595
q8	9239	1877	1941	1877
q9	6697	6642	6570	6570
q10	7028	2307	2286	2286
q11	481	264	256	256
q12	427	222	223	222
q13	17784	3071	3062	3062
q14	249	214	203	203
q15	576	538	522	522
q16	674	601	593	593
q17	1005	619	476	476
q18	7245	6811	6824	6811
q19	1349	974	1053	974
q20	460	180	182	180
q21	4021	3290	3093	3093
q22	385	319	308	308
Total cold run time: 107340 ms
Total hot run time: 40259 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7301	7290	7275	7275
q2	330	231	226	226
q3	2946	2936	2966	2936
q4	2048	1840	1864	1840
q5	5653	5674	5686	5674
q6	231	144	142	142
q7	2330	1872	1780	1780
q8	3415	3535	3543	3535
q9	9006	9076	9078	9076
q10	3588	3578	3593	3578
q11	622	508	507	507
q12	824	648	632	632
q13	12524	3194	3213	3194
q14	300	282	273	273
q15	573	539	527	527
q16	687	651	650	650
q17	1882	1639	1624	1624
q18	8290	7880	7667	7667
q19	1742	1478	1532	1478
q20	2151	1911	1876	1876
q21	5738	5569	5591	5569
q22	638	550	591	550
Total cold run time: 72819 ms
Total hot run time: 60609 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197463 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 03cf22587e5adcd0057a5541ef97db73f50d524a, data reload: false

query1	1241	928	955	928
query2	6220	2156	2151	2151
query3	11092	4582	4549	4549
query4	67058	29501	23590	23590
query5	5006	461	446	446
query6	410	192	180	180
query7	5631	306	294	294
query8	334	242	235	235
query9	9107	2731	2716	2716
query10	454	250	251	250
query11	17581	15467	15846	15467
query12	152	116	105	105
query13	1530	439	414	414
query14	10755	7154	7502	7154
query15	215	178	198	178
query16	7346	476	512	476
query17	1070	590	571	571
query18	1808	296	307	296
query19	193	153	151	151
query20	118	109	108	108
query21	214	102	96	96
query22	4852	4525	4523	4523
query23	34647	34345	34431	34345
query24	5623	2520	2445	2445
query25	478	385	389	385
query26	643	158	146	146
query27	1908	286	288	286
query28	4496	2499	2469	2469
query29	673	435	437	435
query30	215	160	151	151
query31	997	850	871	850
query32	67	56	54	54
query33	405	284	282	282
query34	934	518	547	518
query35	925	746	765	746
query36	1096	966	961	961
query37	123	80	68	68
query38	4460	4399	4426	4399
query39	1514	1505	1485	1485
query40	209	96	97	96
query41	50	43	43	43
query42	105	97	102	97
query43	546	520	507	507
query44	1205	830	841	830
query45	186	175	177	175
query46	1178	749	738	738
query47	2072	1896	1932	1896
query48	414	318	328	318
query49	727	398	405	398
query50	868	397	409	397
query51	7480	7253	7046	7046
query52	97	85	92	85
query53	257	188	180	180
query54	526	399	408	399
query55	79	75	75	75
query56	237	228	227	227
query57	1248	1117	1086	1086
query58	219	225	211	211
query59	3167	2981	3235	2981
query60	272	246	234	234
query61	103	98	102	98
query62	792	660	654	654
query63	214	189	181	181
query64	1337	653	630	630
query65	3260	3219	3197	3197
query66	698	289	299	289
query67	15839	15508	15680	15508
query68	4073	573	562	562
query69	413	256	243	243
query70	1123	1108	1103	1103
query71	356	251	251	251
query72	6591	4229	4000	4000
query73	755	354	367	354
query74	10194	9042	9001	9001
query75	3381	2672	2663	2663
query76	1905	1038	1144	1038
query77	491	273	267	267
query78	10492	9406	9428	9406
query79	1464	606	607	606
query80	988	422	431	422
query81	494	251	226	226
query82	1161	118	120	118
query83	219	146	144	144
query84	283	76	67	67
query85	878	303	296	296
query86	341	304	297	297
query87	4731	4643	4598	4598
query88	3590	2196	2171	2171
query89	418	294	282	282
query90	2043	190	190	190
query91	139	105	103	103
query92	66	49	50	49
query93	1733	544	541	541
query94	811	294	291	291
query95	353	244	240	240
query96	674	284	290	284
query97	2846	2678	2664	2664
query98	221	196	194	194
query99	1661	1313	1335	1313
Total cold run time: 319864 ms
Total hot run time: 197463 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 03cf22587e5adcd0057a5541ef97db73f50d524a, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.04	0.03
query3	0.23	0.08	0.07
query4	1.61	0.10	0.11
query5	0.42	0.41	0.43
query6	1.18	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.52
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.70	2.73	2.71
query15	0.90	0.83	0.83
query16	0.38	0.38	0.39
query17	1.09	1.07	1.09
query18	0.22	0.21	0.21
query19	1.98	1.90	2.04
query20	0.02	0.01	0.01
query21	15.35	0.61	0.58
query22	2.66	2.05	2.43
query23	16.95	0.79	0.86
query24	2.94	2.00	1.28
query25	0.29	0.11	0.31
query26	0.33	0.12	0.13
query27	0.04	0.04	0.04
query28	9.92	1.11	1.08
query29	12.59	3.26	3.25
query30	0.25	0.06	0.07
query31	2.86	0.38	0.38
query32	3.27	0.46	0.45
query33	2.99	3.07	3.04
query34	16.98	4.48	4.51
query35	4.52	4.48	4.64
query36	0.65	0.47	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.99 s
Total hot run time: 33.27 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit df90de9 into apache:master Dec 10, 2024
26 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 10, 2024
Increase default `max_broker_concurrency` to 100 to improve broker load
performance.
This option will affect the max number of scan / sink instances allowed
in a broker load.

```
parallel instance = min(max_broker_concurrency,
                        source file size / min_bytes_per_broker_scanner,
                        num backends * load_parallelism)
```

S3 load time of tpcds_1000g catalog_sales:
* before: 438s
* after: 225s
dataroaring pushed a commit that referenced this pull request Dec 11, 2024
Increase default `max_broker_concurrency` to 100 to improve broker load
performance.
This option will affect the max number of scan / sink instances allowed
in a broker load.

```
parallel instance = min(max_broker_concurrency,
                        source file size / min_bytes_per_broker_scanner,
                        num backends * load_parallelism)
```

S3 load time of tpcds_1000g catalog_sales:
* before: 438s
* after: 225s
dataroaring pushed a commit that referenced this pull request Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.4-merged kind/behavior-changed reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants