Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc tcache #38185

Merged
merged 4 commits into from
Aug 21, 2024
Merged

[opt](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc tcache #38185

merged 4 commits into from
Aug 21, 2024

Conversation

xinyiZzz
Copy link
Contributor

  1. Modify thedefault JEMALLOC_CONF, reduce muzzy_decay_ms and dirty_decay_ms, and modify the parameters of the regular dump heap profile for better understanding.
  2. support flush Jemalloc tcache, when process memory exceed limit and tcache is larger than 1G, the limit of tcache is the number of pages, not the total number of page bytes.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40837 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b007da380cf4bc89be25f7b9bf7b9c7697e3bd7f, data reload: false

------ Round 1 ----------------------------------
q1	17660	4684	4336	4336
q2	2033	195	189	189
q3	10458	1190	1080	1080
q4	10172	787	782	782
q5	7549	2734	2710	2710
q6	219	135	136	135
q7	965	606	606	606
q8	9217	2134	2152	2134
q9	9029	6647	6652	6647
q10	8760	3875	3879	3875
q11	459	250	251	250
q12	391	227	235	227
q13	18327	3049	3082	3049
q14	276	248	243	243
q15	538	500	500	500
q16	502	389	379	379
q17	998	631	653	631
q18	8139	7641	7606	7606
q19	1624	1470	1477	1470
q20	701	332	334	332
q21	5175	3354	4015	3354
q22	354	305	302	302
Total cold run time: 113546 ms
Total hot run time: 40837 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4431	4294	4275	4275
q2	384	262	280	262
q3	3082	2830	2927	2830
q4	2044	1816	1780	1780
q5	5729	5622	5602	5602
q6	233	138	131	131
q7	2291	1923	1978	1923
q8	3317	3432	3461	3432
q9	8984	8965	9049	8965
q10	4227	4028	3934	3934
q11	601	504	510	504
q12	835	649	639	639
q13	11573	3277	3212	3212
q14	312	310	294	294
q15	548	486	484	484
q16	506	434	444	434
q17	1819	1531	1527	1527
q18	8275	7952	7970	7952
q19	1751	1584	1635	1584
q20	2117	1939	1896	1896
q21	5362	5048	4953	4953
q22	591	611	518	518
Total cold run time: 69012 ms
Total hot run time: 57131 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176387 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b007da380cf4bc89be25f7b9bf7b9c7697e3bd7f, data reload: false

query1	906	364	369	364
query2	6424	1904	1898	1898
query3	6652	210	218	210
query4	28374	17773	17490	17490
query5	3618	488	483	483
query6	272	176	163	163
query7	4590	299	296	296
query8	245	193	192	192
query9	8508	2512	2520	2512
query10	431	317	299	299
query11	12513	10148	10223	10148
query12	119	88	84	84
query13	1644	392	387	387
query14	9513	8001	8947	8001
query15	284	162	172	162
query16	7417	445	463	445
query17	1642	581	547	547
query18	1267	286	298	286
query19	339	156	159	156
query20	92	83	88	83
query21	212	127	130	127
query22	4644	4495	4220	4220
query23	34606	34443	33922	33922
query24	10794	3068	2964	2964
query25	688	405	400	400
query26	1058	156	163	156
query27	2461	299	289	289
query28	7123	2108	2106	2106
query29	898	666	643	643
query30	256	154	152	152
query31	966	811	737	737
query32	100	55	58	55
query33	754	353	353	353
query34	912	504	533	504
query35	872	789	772	772
query36	1138	994	1014	994
query37	165	86	91	86
query38	2920	2978	2823	2823
query39	882	827	810	810
query40	201	121	117	117
query41	45	43	46	43
query42	122	99	96	96
query43	521	484	478	478
query44	1210	741	739	739
query45	188	160	158	158
query46	1095	766	724	724
query47	1886	1800	1828	1800
query48	378	298	306	298
query49	824	411	416	411
query50	776	396	421	396
query51	6844	6771	6682	6682
query52	101	93	88	88
query53	355	294	285	285
query54	916	465	472	465
query55	75	74	78	74
query56	279	271	285	271
query57	1160	1081	1045	1045
query58	248	256	258	256
query59	2916	2757	2746	2746
query60	305	273	276	273
query61	97	98	101	98
query62	780	671	651	651
query63	320	292	289	289
query64	9525	2365	1704	1704
query65	3183	3107	3150	3107
query66	751	331	344	331
query67	15627	15280	15403	15280
query68	4536	563	556	556
query69	479	358	386	358
query70	1166	1120	1099	1099
query71	363	323	285	285
query72	7001	5072	5622	5072
query73	745	335	345	335
query74	6131	5756	5862	5756
query75	3457	2751	2750	2750
query76	2722	917	897	897
query77	471	334	311	311
query78	9729	9116	10895	9116
query79	1019	538	530	530
query80	700	490	491	490
query81	556	226	225	225
query82	465	146	135	135
query83	188	163	172	163
query84	235	87	87	87
query85	855	316	362	316
query86	325	323	303	303
query87	3319	3188	3247	3188
query88	3100	2513	2498	2498
query89	488	384	382	382
query90	1855	196	196	196
query91	132	105	102	102
query92	58	52	49	49
query93	997	516	505	505
query94	787	302	302	302
query95	418	326	323	323
query96	585	280	285	280
query97	3231	3101	3080	3080
query98	211	211	199	199
query99	1637	1275	1289	1275
Total cold run time: 274988 ms
Total hot run time: 176387 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b007da380cf4bc89be25f7b9bf7b9c7697e3bd7f, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.70	0.08	0.07
query5	0.49	0.49	0.48
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.04	0.05
query9	0.56	0.50	0.50
query10	0.55	0.54	0.55
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.61	0.59	0.59
query14	0.77	0.78	0.79
query15	0.86	0.82	0.82
query16	0.35	0.37	0.36
query17	0.96	1.05	1.05
query18	0.22	0.22	0.22
query19	1.86	1.73	1.77
query20	0.01	0.00	0.01
query21	15.39	0.75	0.66
query22	4.93	7.88	1.41
query23	18.32	1.33	1.28
query24	2.10	0.24	0.22
query25	0.16	0.08	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.22
query28	13.28	1.04	1.00
query29	12.57	3.39	3.34
query30	0.25	0.06	0.06
query31	2.88	0.40	0.38
query32	3.24	0.48	0.47
query33	2.97	2.95	2.95
query34	17.12	4.46	4.43
query35	4.50	4.48	4.46
query36	0.65	0.46	0.49
query37	0.18	0.16	0.16
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.16	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.64 s
Total hot run time: 30.51 s

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40827 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6fd0733d3e6e2a0d0a16fec984ea006b7b63dc0b, data reload: false

------ Round 1 ----------------------------------
q1	17996	4630	4450	4450
q2	2214	191	190	190
q3	10451	1201	1158	1158
q4	10183	912	824	824
q5	7518	2837	2687	2687
q6	224	140	145	140
q7	986	611	617	611
q8	9216	2138	2133	2133
q9	8876	6623	6586	6586
q10	8807	3900	3908	3900
q11	451	238	246	238
q12	410	238	228	228
q13	17762	3055	3059	3055
q14	268	233	244	233
q15	532	489	482	482
q16	492	386	375	375
q17	997	666	673	666
q18	8135	7620	7578	7578
q19	1659	1437	1339	1339
q20	677	314	329	314
q21	4985	3416	3348	3348
q22	349	292	297	292
Total cold run time: 113188 ms
Total hot run time: 40827 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4417	4416	4283	4283
q2	391	272	271	271
q3	3088	2842	2848	2842
q4	1937	1647	1610	1610
q5	5336	5364	5384	5364
q6	227	135	136	135
q7	2216	1790	1777	1777
q8	3266	3379	3343	3343
q9	8435	8413	8476	8413
q10	4028	3768	3797	3768
q11	592	501	483	483
q12	795	615	609	609
q13	12196	3036	3030	3030
q14	302	273	280	273
q15	517	479	488	479
q16	477	433	436	433
q17	1794	1491	1472	1472
q18	7750	7491	7361	7361
q19	1698	1531	1545	1531
q20	2065	1828	1801	1801
q21	5113	4766	4824	4766
q22	606	527	538	527
Total cold run time: 67246 ms
Total hot run time: 54571 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176822 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6fd0733d3e6e2a0d0a16fec984ea006b7b63dc0b, data reload: false

query1	919	372	368	368
query2	6449	1929	1873	1873
query3	6670	213	214	213
query4	24692	17650	17506	17506
query5	4181	483	510	483
query6	289	180	154	154
query7	4587	282	286	282
query8	242	208	208	208
query9	8460	2436	2433	2433
query10	447	291	282	282
query11	11413	10128	10224	10128
query12	141	87	83	83
query13	1650	401	398	398
query14	10413	7820	7815	7815
query15	276	169	162	162
query16	8173	495	439	439
query17	1680	595	549	549
query18	2023	285	282	282
query19	320	156	159	156
query20	91	83	82	82
query21	205	130	127	127
query22	4393	4134	4225	4134
query23	34111	33258	35691	33258
query24	11624	2868	2907	2868
query25	535	396	406	396
query26	1133	151	149	149
query27	2767	279	290	279
query28	7492	1989	1997	1989
query29	859	633	639	633
query30	292	151	154	151
query31	974	771	778	771
query32	103	57	56	56
query33	793	362	362	362
query34	907	497	512	497
query35	885	730	722	722
query36	1087	986	939	939
query37	208	84	83	83
query38	2916	2813	2779	2779
query39	879	807	847	807
query40	281	126	122	122
query41	48	49	46	46
query42	118	103	106	103
query43	525	470	496	470
query44	1268	760	756	756
query45	190	169	165	165
query46	1118	774	731	731
query47	1919	1814	1810	1810
query48	378	302	316	302
query49	1172	440	443	440
query50	783	399	408	399
query51	6778	6796	6794	6794
query52	105	91	91	91
query53	362	283	295	283
query54	899	464	465	464
query55	81	74	77	74
query56	322	301	315	301
query57	1171	1107	1079	1079
query58	266	385	262	262
query59	2992	2832	2819	2819
query60	310	279	288	279
query61	99	94	105	94
query62	828	654	666	654
query63	328	288	286	286
query64	10442	2277	1748	1748
query65	3252	3149	3159	3149
query66	1260	341	347	341
query67	15678	15201	15353	15201
query68	6405	542	541	541
query69	662	455	373	373
query70	1239	1108	1150	1108
query71	525	291	299	291
query72	6885	5682	5887	5682
query73	811	341	334	334
query74	6301	5712	5739	5712
query75	4218	2708	2740	2708
query76	4469	902	937	902
query77	753	331	321	321
query78	9782	10132	11110	10132
query79	1894	534	531	531
query80	920	489	476	476
query81	549	225	219	219
query82	277	131	138	131
query83	201	167	172	167
query84	275	136	90	90
query85	1405	319	301	301
query86	428	290	314	290
query87	3290	3239	3183	3183
query88	3544	2524	2509	2509
query89	492	393	373	373
query90	1976	204	199	199
query91	132	102	98	98
query92	72	52	51	51
query93	1101	496	501	496
query94	1199	306	300	300
query95	414	327	330	327
query96	591	318	277	277
query97	3205	3098	3145	3098
query98	220	210	198	198
query99	1729	1245	1266	1245
Total cold run time: 284216 ms
Total hot run time: 176822 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.06 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6fd0733d3e6e2a0d0a16fec984ea006b7b63dc0b, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.47	0.48
query6	1.12	0.73	0.74
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.49	0.50
query10	0.56	0.55	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.60	0.59
query14	0.75	0.78	0.80
query15	0.85	0.82	0.82
query16	0.35	0.38	0.38
query17	1.05	1.00	1.06
query18	0.23	0.21	0.22
query19	1.94	1.83	1.83
query20	0.01	0.01	0.01
query21	15.40	0.75	0.66
query22	4.44	6.25	2.07
query23	18.24	1.47	1.26
query24	2.16	0.24	0.22
query25	0.15	0.08	0.09
query26	0.30	0.21	0.21
query27	0.45	0.22	0.23
query28	13.34	1.02	1.00
query29	12.64	3.29	3.29
query30	0.25	0.06	0.05
query31	2.86	0.40	0.38
query32	3.30	0.47	0.46
query33	2.93	2.97	2.96
query34	16.89	4.43	4.40
query35	4.44	4.40	4.44
query36	0.66	0.48	0.48
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 110.1 s
Total hot run time: 31.06 s

yiguolei
yiguolei previously approved these changes Jul 23, 2024
View reviewed changes
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@xinyiZzz xinyiZzz marked this pull request as draft July 28, 2024 05:59
@xinyiZzz xinyiZzz marked this pull request as ready for review August 13, 2024 12:26
@xinyiZzz xinyiZzz force-pushed the 20240720_fix_jemalloc branch from 6fd0733 to c172d89 Compare August 14, 2024 07:17
@github-actions github-actions bot added doing and removed approved Indicates a PR has been approved by one committer. labels Aug 14, 2024
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38312 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c172d8970a57c301875b0bb736671b91b67ce8a9, data reload: false

------ Round 1 ----------------------------------
q1	17613	4475	4313	4313
q2	2007	189	175	175
q3	11739	1068	1124	1068
q4	10514	707	700	700
q5	7783	2844	2851	2844
q6	226	137	135	135
q7	960	598	609	598
q8	9350	2122	2088	2088
q9	7182	6553	6572	6553
q10	6990	2244	2212	2212
q11	472	245	249	245
q12	399	223	220	220
q13	18782	3115	3065	3065
q14	294	245	237	237
q15	538	482	488	482
q16	497	418	394	394
q17	996	725	736	725
q18	7328	6963	6877	6877
q19	1389	1146	1047	1047
q20	693	360	340	340
q21	4245	2994	3102	2994
q22	1105	1028	1000	1000
Total cold run time: 111102 ms
Total hot run time: 38312 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4269	4278	4269
q2	370	274	280	274
q3	2943	2731	2725	2725
q4	1935	1696	1643	1643
q5	5678	5714	5786	5714
q6	231	137	134	134
q7	2210	1867	1813	1813
q8	3271	3524	3443	3443
q9	8902	8875	8789	8789
q10	3582	3417	3360	3360
q11	622	525	517	517
q12	837	636	646	636
q13	14745	3240	3260	3240
q14	346	296	305	296
q15	538	494	482	482
q16	515	439	465	439
q17	1871	1549	1539	1539
q18	8182	7897	8040	7897
q19	1764	1677	1572	1572
q20	2181	1935	1940	1935
q21	5749	5593	5548	5548
q22	1137	1059	1061	1059
Total cold run time: 71972 ms
Total hot run time: 57324 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190781 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c172d8970a57c301875b0bb736671b91b67ce8a9, data reload: false

query1	1252	905	880	880
query2	6292	1932	1901	1901
query3	10621	3996	3962	3962
query4	59763	24500	23151	23151
query5	5677	520	522	520
query6	446	167	178	167
query7	5854	286	302	286
query8	287	204	215	204
query9	8809	2489	2488	2488
query10	491	278	268	268
query11	18267	15121	15223	15121
query12	159	98	103	98
query13	1555	392	385	385
query14	11080	7824	6897	6897
query15	237	184	180	180
query16	7178	448	449	448
query17	1252	590	567	567
query18	1420	296	306	296
query19	299	146	143	143
query20	123	106	109	106
query21	205	98	100	98
query22	4580	4517	4641	4517
query23	34258	33495	33399	33399
query24	5963	2885	2826	2826
query25	529	377	372	372
query26	695	155	153	153
query27	1809	275	272	272
query28	3692	2044	2030	2030
query29	651	404	397	397
query30	238	154	147	147
query31	935	763	764	763
query32	78	53	54	53
query33	455	310	276	276
query34	865	463	476	463
query35	866	712	694	694
query36	1047	975	922	922
query37	146	80	77	77
query38	4083	3848	3869	3848
query39	1447	1391	1400	1391
query40	200	116	114	114
query41	45	44	43	43
query42	115	95	93	93
query43	519	472	465	465
query44	1129	736	743	736
query45	196	162	164	162
query46	1103	764	747	747
query47	1886	1809	1760	1760
query48	367	292	290	290
query49	756	430	462	430
query50	810	402	414	402
query51	6758	6811	6688	6688
query52	98	88	88	88
query53	267	184	180	180
query54	564	461	458	458
query55	77	74	75	74
query56	288	254	255	254
query57	1209	1104	1042	1042
query58	279	223	242	223
query59	3113	2973	2750	2750
query60	286	262	272	262
query61	98	95	97	95
query62	773	666	665	665
query63	220	187	183	183
query64	3841	2233	1710	1710
query65	3211	3171	3189	3171
query66	682	340	334	334
query67	15227	14946	15101	14946
query68	3057	574	599	574
query69	405	269	270	269
query70	1081	1156	1112	1112
query71	388	278	269	269
query72	2535	2176	2064	2064
query73	707	327	320	320
query74	9229	8916	8896	8896
query75	3406	2709	2720	2709
query76	1520	1034	979	979
query77	553	318	310	310
query78	9696	8989	9076	8989
query79	1046	540	525	525
query80	679	505	508	505
query81	458	231	225	225
query82	291	135	135	135
query83	173	150	151	150
query84	252	78	80	78
query85	670	275	276	275
query86	301	280	294	280
query87	4378	4287	4231	4231
query88	2944	2285	2291	2285
query89	384	292	278	278
query90	1935	195	192	192
query91	119	98	98	98
query92	62	52	50	50
query93	1036	532	532	532
query94	717	306	348	306
query95	324	255	269	255
query96	584	268	267	267
query97	3237	3104	3052	3052
query98	217	215	201	201
query99	1507	1279	1239	1239
Total cold run time: 301612 ms
Total hot run time: 190781 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c172d8970a57c301875b0bb736671b91b67ce8a9, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.08
query5	0.50	0.50	0.48
query6	1.13	0.72	0.74
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.55	0.48	0.48
query10	0.53	0.54	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.61	0.58
query14	0.77	0.77	0.81
query15	0.90	0.82	0.82
query16	0.37	0.39	0.37
query17	1.01	1.06	1.04
query18	0.23	0.22	0.24
query19	1.97	1.87	1.72
query20	0.01	0.01	0.01
query21	15.39	0.76	0.67
query22	3.74	7.12	1.78
query23	18.29	1.41	1.27
query24	2.10	0.24	0.22
query25	0.15	0.08	0.09
query26	0.31	0.21	0.22
query27	0.45	0.24	0.22
query28	13.31	1.01	1.00
query29	12.64	3.33	3.35
query30	0.24	0.05	0.05
query31	2.90	0.40	0.39
query32	3.24	0.49	0.48
query33	2.98	3.00	2.98
query34	17.13	4.41	4.36
query35	4.39	4.42	4.44
query36	0.66	0.47	0.49
query37	0.19	0.17	0.16
query38	0.15	0.17	0.15
query39	0.05	0.03	0.03
query40	0.16	0.13	0.12
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.63 s
Total hot run time: 30.75 s

@xinyiZzz
Copy link
Contributor Author

run buildall

@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38278 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b448e83d192b9a24e54fa5156f81c0aad7c2805b, data reload: false

------ Round 1 ----------------------------------
q1	18198	4603	4442	4442
q2	3119	190	170	170
q3	11073	1180	1151	1151
q4	10578	721	702	702
q5	8129	2889	2874	2874
q6	224	144	143	143
q7	985	611	595	595
q8	9359	2069	2078	2069
q9	7365	6524	6546	6524
q10	7047	2161	2201	2161
q11	449	248	257	248
q12	391	220	221	220
q13	17766	3054	3016	3016
q14	285	242	239	239
q15	536	480	483	480
q16	520	414	387	387
q17	994	732	745	732
q18	7381	6948	6832	6832
q19	1391	1010	1146	1010
q20	669	349	344	344
q21	4203	2936	3038	2936
q22	1140	1013	1003	1003
Total cold run time: 111802 ms
Total hot run time: 38278 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4389	4293	4308	4293
q2	387	271	267	267
q3	2924	2650	2626	2626
q4	1900	1630	1648	1630
q5	5397	5394	5430	5394
q6	214	131	131	131
q7	2111	1714	1728	1714
q8	3196	3355	3369	3355
q9	8424	8368	8426	8368
q10	3427	3167	3182	3167
q11	595	494	512	494
q12	797	607	617	607
q13	11530	3029	3004	3004
q14	320	277	281	277
q15	521	483	483	483
q16	466	420	426	420
q17	1782	1501	1478	1478
q18	7833	7537	7640	7537
q19	1669	1424	1648	1424
q20	2057	1809	1779	1779
q21	5524	5296	5248	5248
q22	1114	1052	1021	1021
Total cold run time: 66577 ms
Total hot run time: 54717 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186407 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b448e83d192b9a24e54fa5156f81c0aad7c2805b, data reload: false

query1	898	363	361	361
query2	6448	1997	1964	1964
query3	6654	213	221	213
query4	27125	23403	23099	23099
query5	4209	504	507	504
query6	277	162	169	162
query7	4574	295	293	293
query8	239	213	203	203
query9	8482	2483	2456	2456
query10	453	270	263	263
query11	17840	14909	14955	14909
query12	144	99	102	99
query13	1646	382	367	367
query14	9519	7359	7340	7340
query15	265	171	175	171
query16	8156	452	422	422
query17	1659	573	563	563
query18	2131	289	292	289
query19	338	149	146	146
query20	115	106	103	103
query21	208	102	105	102
query22	4366	4093	4103	4093
query23	34025	33638	33325	33325
query24	11231	2920	2843	2843
query25	628	394	396	394
query26	1493	157	162	157
query27	2791	276	281	276
query28	7348	2044	2048	2044
query29	952	413	420	413
query30	305	151	161	151
query31	999	759	768	759
query32	96	57	59	57
query33	762	296	289	289
query34	967	475	478	475
query35	869	758	709	709
query36	1084	939	960	939
query37	158	87	88	87
query38	3993	3851	3816	3816
query39	1421	1392	1378	1378
query40	269	117	118	117
query41	51	46	50	46
query42	116	99	100	99
query43	528	475	482	475
query44	1238	743	745	743
query45	192	166	163	163
query46	1118	717	741	717
query47	1864	1785	1785	1785
query48	378	308	301	301
query49	1115	452	437	437
query50	818	414	419	414
query51	6819	6711	6686	6686
query52	100	87	87	87
query53	255	180	178	178
query54	1020	458	458	458
query55	79	77	80	77
query56	300	266	259	259
query57	1191	1077	1072	1072
query58	240	220	252	220
query59	3055	2964	2672	2672
query60	300	278	272	272
query61	118	117	237	117
query62	850	654	650	650
query63	222	185	187	185
query64	6294	2271	1769	1769
query65	3284	3144	3165	3144
query66	1392	358	332	332
query67	15631	15175	15140	15140
query68	5799	550	564	550
query69	584	373	287	287
query70	1204	1123	1123	1123
query71	490	271	269	269
query72	6681	2341	2054	2054
query73	808	324	326	324
query74	9183	8826	8730	8730
query75	3830	2724	2721	2721
query76	3649	1075	985	985
query77	718	331	309	309
query78	9753	9000	9087	9000
query79	1640	535	550	535
query80	823	491	486	486
query81	580	223	224	223
query82	1000	137	136	136
query83	255	148	145	145
query84	225	75	74	74
query85	1520	318	281	281
query86	471	273	301	273
query87	4393	4294	4221	4221
query88	4139	2361	2345	2345
query89	377	290	283	283
query90	1823	213	189	189
query91	120	98	95	95
query92	67	49	53	49
query93	1943	538	539	538
query94	897	290	295	290
query95	352	269	268	268
query96	598	273	272	272
query97	3205	3057	3054	3054
query98	217	215	198	198
query99	1716	1251	1256	1251
Total cold run time: 292765 ms
Total hot run time: 186407 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b448e83d192b9a24e54fa5156f81c0aad7c2805b, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.05
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.52	0.50
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.06	0.05	0.04
query9	0.55	0.48	0.48
query10	0.52	0.53	0.55
query11	0.16	0.12	0.11
query12	0.14	0.12	0.13
query13	0.60	0.60	0.59
query14	0.76	0.76	0.79
query15	0.85	0.81	0.82
query16	0.36	0.36	0.38
query17	1.02	1.02	0.97
query18	0.22	0.22	0.22
query19	1.80	1.84	1.80
query20	0.02	0.01	0.01
query21	15.40	0.76	0.67
query22	3.97	7.16	1.87
query23	18.31	1.41	1.32
query24	2.10	0.24	0.21
query25	0.15	0.08	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.22
query28	13.30	1.00	1.00
query29	12.58	3.32	3.28
query30	0.24	0.06	0.04
query31	2.89	0.40	0.39
query32	3.26	0.47	0.46
query33	2.94	2.98	2.99
query34	17.00	4.35	4.44
query35	4.38	4.41	4.38
query36	0.66	0.51	0.47
query37	0.20	0.16	0.16
query38	0.15	0.14	0.15
query39	0.06	0.03	0.03
query40	0.16	0.13	0.13
query41	0.10	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.42 s
Total hot run time: 30.74 s

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 17, 2024
wangbo
wangbo approved these changes Aug 20, 2024
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit 81160b2 into apache:master Aug 21, 2024
26 of 30 checks passed
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Aug 23, 2024
@xinyiZzz
[opt](memory) Modify thedefault JEMALLOC_CONF and support flush Jemal…
edd5e56 
…loc tcache (apache#38185)

1. Modify thedefault JEMALLOC_CONF, reduce `muzzy_decay_ms` and
`dirty_decay_ms`, and modify the parameters of the regular dump heap
profile for better understanding.
2. support flush Jemalloc tcache, when process memory exceed limit and
tcache is larger than 1G, the limit of `tcache` is the number of pages,
not the total number of page bytes.
@xinyiZzz xinyiZzz mentioned this pull request Aug 23, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 23, 2024
@xinyiZzz
[branch-2.1](memory) Modify thedefault JEMALLOC_CONF and support flus…
baf5b71 
…h Jemalloc tcache (#39829)

pick #38185
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Dec 16, 2024
@xinyiZzz
[opt](memory) Modify thedefault JEMALLOC_CONF and support flush Jemal…
3ab28c8 
…loc tcache (apache#38185)

1. Modify thedefault JEMALLOC_CONF, reduce `muzzy_decay_ms` and
`dirty_decay_ms`, and modify the parameters of the regular dump heap
profile for better understanding.
2. support flush Jemalloc tcache, when process memory exceed limit and
tcache is larger than 1G, the limit of `tcache` is the number of pages,
not the total number of page bytes.
@xinyiZzz xinyiZzz mentioned this pull request Dec 16, 2024
Merged
16 tasks
yiguolei pushed a commit that referenced this pull request Dec 25, 2024
@xinyiZzz
[pick](branch-3.0) pick #38185 #42059 (#45092)
0ca6c38 
### What problem does this PR solve?

pick
[opt](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc
tcache #38185
[opt](memory) Support Jemalloc heap profile start at runtime and
automatically generate dot #42059
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wangbo wangbo wangbo approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.4-merged doing reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@xinyiZzz @doris-robot @yiguolei @wangbo