[opt](memory) Support Jemalloc heap profile start at runtime and automatically generate dot #42059

xinyiZzz · 2024-10-17T16:36:50Z

Now Doris needs to restart the BE process every time to analyze the Jemalloc heap profile, and dump the profile file first, and then use jeprof to analyze the profile file to generate a dot graph.

This PR will allow the jemalloc heap profiler to be dynamically start or stop during BE runtime, and can directly return to the profile dot graph.

start heap profiler

curl http://{be_ip}:8040/jeheap/active/true

heap profiler started
Jemalloc will only track and sample the memory allocated and freed after the heap profiler started, it cannot analyze the memory allocated and freed before. Therefore, dumping the heap profile immediately after start heap profiler may prompt `No nodes to print`. If you want to analyze the memory that has been allocated in the past, you can only restart the BE process and start heap profiler immediately.

stop heap profiler

curl http://{be_ip}:8040/jeheap/active/false

heap profiler stoped

dump heap profile and format to dot

curl http://{be_ip}:8040/jeheap/dump

perl: warning: Setting locale failed.
 ......
addr2line: Dwarf Error
 ......
digraph
.......
-------------------------------------------------------
Copy the text after `digraph` in the above output to http://www.webgraphviz.com to generate a dot graph.
If there are many errors: `addr2line: Dwarf Error`,Reference Documentation: https://doris.apache.org/community/developer-guide/debug-tool/#4-qa

only dump heap profile

curl http://{be_ip}:8040/jeheap/dump_only

jemalloc heap dump success, dump file path:xxxx

doris-robot · 2024-10-17T16:36:54Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

xinyiZzz · 2024-10-17T16:37:34Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/runtime/memory/heap_profiler.cpp

doris-robot · 2024-10-17T18:06:00Z

TPC-H: Total hot run time: 41567 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 875adb5dc6bcd2db55442430a6c9628004a7b059, data reload: false

------ Round 1 ----------------------------------
q1	17572	7476	7300	7300
q2	2019	293	292	292
q3	11932	1088	1194	1088
q4	10568	902	908	902
q5	7779	3135	3116	3116
q6	245	154	156	154
q7	1039	634	610	610
q8	9362	2006	2021	2006
q9	6723	6483	6453	6453
q10	7088	2444	2471	2444
q11	438	234	250	234
q12	417	222	225	222
q13	17779	2971	3030	2971
q14	247	211	210	210
q15	576	524	511	511
q16	650	574	602	574
q17	995	593	565	565
q18	7371	6810	6686	6686
q19	1345	899	1008	899
q20	473	189	190	189
q21	4076	3186	3157	3157
q22	1063	984	1000	984
Total cold run time: 109757 ms
Total hot run time: 41567 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7281	7273	7359	7273
q2	338	232	241	232
q3	3080	3006	3096	3006
q4	2153	1876	1828	1828
q5	5783	5757	5893	5757
q6	249	152	148	148
q7	2278	1797	1853	1797
q8	3444	3506	3543	3506
q9	9069	8978	8965	8965
q10	3630	3621	3561	3561
q11	590	488	498	488
q12	864	628	618	618
q13	9463	3239	3196	3196
q14	300	277	282	277
q15	586	538	526	526
q16	706	651	645	645
q17	1878	1638	1629	1629
q18	8444	7784	7743	7743
q19	1722	1550	1601	1550
q20	2151	1954	1865	1865
q21	5675	5537	5442	5442
q22	1175	1061	1032	1032
Total cold run time: 70859 ms
Total hot run time: 61084 ms

doris-robot · 2024-10-17T18:18:05Z

TPC-DS: Total hot run time: 193054 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 875adb5dc6bcd2db55442430a6c9628004a7b059, data reload: false

query1	839	421	411	411
query2	6278	2090	2062	2062
query3	8679	194	197	194
query4	34322	23724	23637	23637
query5	3501	498	476	476
query6	278	186	166	166
query7	4186	301	298	298
query8	307	255	238	238
query9	9576	2747	2743	2743
query10	462	291	274	274
query11	17935	15294	15328	15294
query12	159	105	105	105
query13	1555	422	414	414
query14	9608	7656	7653	7653
query15	280	179	182	179
query16	7822	528	482	482
query17	1674	599	593	593
query18	1626	308	315	308
query19	370	163	153	153
query20	124	114	118	114
query21	218	109	108	108
query22	4934	4701	4569	4569
query23	35017	34393	34098	34098
query24	11062	2783	2809	2783
query25	594	413	413	413
query26	948	160	164	160
query27	2086	291	294	291
query28	6440	2446	2451	2446
query29	716	437	420	420
query30	263	153	167	153
query31	1062	802	819	802
query32	95	55	58	55
query33	793	292	301	292
query34	938	513	510	510
query35	906	771	755	755
query36	1110	945	963	945
query37	144	83	83	83
query38	4016	3930	3893	3893
query39	1483	1427	1432	1427
query40	211	99	99	99
query41	48	46	46	46
query42	124	102	101	101
query43	538	500	496	496
query44	1276	821	809	809
query45	198	168	169	168
query46	1161	722	695	695
query47	1941	1839	1863	1839
query48	411	326	321	321
query49	910	437	425	425
query50	825	407	379	379
query51	7180	6924	6954	6924
query52	104	89	86	86
query53	252	179	175	175
query54	1254	417	416	416
query55	78	79	78	78
query56	272	250	258	250
query57	1281	1207	1158	1158
query58	226	226	236	226
query59	3440	3126	3138	3126
query60	285	259	245	245
query61	97	98	99	98
query62	885	662	688	662
query63	223	183	183	183
query64	3625	626	634	626
query65	3298	3249	3224	3224
query66	836	311	307	307
query67	16110	15755	15713	15713
query68	3619	572	560	560
query69	606	297	292	292
query70	1205	1138	1141	1138
query71	426	275	280	275
query72	7773	4039	3751	3751
query73	774	359	360	359
query74	10267	9095	8972	8972
query75	4286	2665	2686	2665
query76	3187	937	926	926
query77	746	303	298	298
query78	10563	9648	9643	9643
query79	1701	590	587	587
query80	2713	440	451	440
query81	594	242	240	240
query82	730	145	141	141
query83	325	140	134	134
query84	286	74	72	72
query85	1550	286	276	276
query86	437	304	291	291
query87	4395	4428	4352	4352
query88	3185	2234	2187	2187
query89	414	290	287	287
query90	2180	189	180	180
query91	137	102	102	102
query92	67	48	47	47
query93	1288	525	532	525
query94	1201	286	292	286
query95	353	242	250	242
query96	624	273	282	273
query97	3284	3144	3142	3142
query98	217	205	197	197
query99	1700	1302	1307	1302
Total cold run time: 300328 ms
Total hot run time: 193054 ms

doris-robot · 2024-10-17T18:23:32Z

ClickBench: Total hot run time: 32.93 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 875adb5dc6bcd2db55442430a6c9628004a7b059, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.02
query3	0.22	0.06	0.06
query4	1.65	0.10	0.10
query5	0.51	0.52	0.52
query6	1.14	0.72	0.73
query7	0.02	0.02	0.02
query8	0.03	0.03	0.03
query9	0.56	0.51	0.53
query10	0.56	0.55	0.55
query11	0.13	0.11	0.10
query12	0.13	0.12	0.11
query13	0.62	0.59	0.60
query14	2.73	2.86	2.79
query15	0.91	0.83	0.84
query16	0.39	0.38	0.38
query17	1.01	1.01	0.98
query18	0.20	0.20	0.19
query19	1.98	1.82	1.93
query20	0.01	0.01	0.02
query21	15.38	0.60	0.58
query22	2.62	1.72	2.15
query23	17.01	1.02	0.73
query24	3.30	1.91	1.39
query25	0.29	0.12	0.08
query26	0.42	0.14	0.14
query27	0.04	0.04	0.04
query28	9.66	1.10	1.06
query29	12.59	3.22	3.21
query30	0.25	0.06	0.07
query31	2.86	0.39	0.39
query32	3.26	0.48	0.46
query33	3.03	3.04	3.01
query34	16.92	4.45	4.46
query35	4.51	4.52	4.48
query36	0.68	0.49	0.48
query37	0.08	0.05	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.16 s
Total hot run time: 32.93 s

github-actions · 2024-10-22T10:07:59Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-22T10:08:01Z

PR approved by anyone and no changes requested.

wangbo

LGTM

xinyiZzz · 2024-10-22T11:37:54Z

run buildall

doris-robot · 2024-10-22T12:57:11Z

TPC-H: Total hot run time: 41645 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01cce5acb604bf8079ffb616ba2e32811c730ef0, data reload: false

------ Round 1 ----------------------------------
q1	17733	7526	7294	7294
q2	2029	288	273	273
q3	11736	1104	1171	1104
q4	10583	865	863	863
q5	7770	3160	3134	3134
q6	241	156	157	156
q7	1043	632	606	606
q8	9369	1980	1941	1941
q9	6677	6496	6495	6495
q10	7037	2420	2448	2420
q11	454	244	253	244
q12	416	224	219	219
q13	17777	3028	3003	3003
q14	240	213	213	213
q15	569	516	522	516
q16	662	593	611	593
q17	989	643	549	549
q18	7339	6645	6637	6637
q19	1350	939	1143	939
q20	484	184	185	184
q21	4022	3250	3371	3250
q22	1110	1012	1026	1012
Total cold run time: 109630 ms
Total hot run time: 41645 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7333	7302	7290	7290
q2	333	244	250	244
q3	3116	2960	2939	2939
q4	2114	1874	1846	1846
q5	5781	5780	5803	5780
q6	236	145	149	145
q7	3013	1867	1782	1782
q8	3419	3561	3503	3503
q9	8973	9026	8876	8876
q10	3643	3565	3596	3565
q11	604	505	493	493
q12	848	639	610	610
q13	9915	3193	3202	3193
q14	307	275	290	275
q15	592	522	509	509
q16	701	654	625	625
q17	1867	1643	1626	1626
q18	8445	7809	7494	7494
q19	1730	1462	1549	1462
q20	2148	1891	1895	1891
q21	5711	5558	5514	5514
q22	1141	1095	1076	1076
Total cold run time: 71970 ms
Total hot run time: 60738 ms

doris-robot · 2024-10-22T13:09:19Z

TPC-DS: Total hot run time: 192151 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01cce5acb604bf8079ffb616ba2e32811c730ef0, data reload: false

query1	931	388	402	388
query2	6251	2163	2108	2108
query3	8683	207	216	207
query4	34459	23622	23604	23604
query5	3539	472	468	468
query6	264	164	161	161
query7	4192	290	283	283
query8	287	234	243	234
query9	9446	2645	2639	2639
query10	480	286	272	272
query11	17885	15574	15275	15275
query12	148	108	102	102
query13	1562	433	417	417
query14	9435	7558	7216	7216
query15	251	168	180	168
query16	7887	475	455	455
query17	1657	615	597	597
query18	2016	309	330	309
query19	375	162	154	154
query20	118	114	121	114
query21	207	108	109	108
query22	4881	4816	4526	4526
query23	36998	34423	34193	34193
query24	11042	2774	2747	2747
query25	529	401	405	401
query26	724	157	177	157
query27	2101	286	282	282
query28	7043	2429	2401	2401
query29	668	429	414	414
query30	280	153	153	153
query31	1068	784	777	777
query32	95	56	58	56
query33	781	303	321	303
query34	942	501	531	501
query35	909	794	745	745
query36	1127	908	956	908
query37	152	89	87	87
query38	4043	3962	3970	3962
query39	1482	1405	1432	1405
query40	220	102	103	102
query41	49	48	51	48
query42	126	101	104	101
query43	515	498	476	476
query44	1221	796	820	796
query45	202	168	168	168
query46	1131	698	675	675
query47	1960	1854	1888	1854
query48	422	320	317	317
query49	970	439	427	427
query50	829	386	386	386
query51	7180	6937	6952	6937
query52	99	89	94	89
query53	257	182	178	178
query54	1193	437	424	424
query55	79	78	82	78
query56	278	288	242	242
query57	1286	1147	1146	1146
query58	226	235	242	235
query59	3197	3097	2882	2882
query60	304	259	261	259
query61	107	101	103	101
query62	860	664	666	664
query63	215	189	181	181
query64	3614	646	615	615
query65	3394	3219	3215	3215
query66	825	309	301	301
query67	16172	15841	15670	15670
query68	4451	604	558	558
query69	521	296	289	289
query70	1194	1095	1092	1092
query71	356	281	285	281
query72	7315	3995	3979	3979
query73	777	350	358	350
query74	10308	9064	8996	8996
query75	4094	2628	2707	2628
query76	3181	902	925	902
query77	641	297	311	297
query78	10621	9666	9629	9629
query79	2679	595	607	595
query80	2646	434	445	434
query81	600	242	235	235
query82	702	135	136	135
query83	306	136	132	132
query84	282	71	69	69
query85	1460	287	284	284
query86	382	296	279	279
query87	4441	4385	4360	4360
query88	3901	2191	2160	2160
query89	414	284	284	284
query90	1933	183	182	182
query91	150	101	104	101
query92	62	47	48	47
query93	2127	555	546	546
query94	986	295	296	295
query95	350	249	251	249
query96	623	285	277	277
query97	3341	3124	3133	3124
query98	231	198	189	189
query99	1575	1337	1309	1309
Total cold run time: 304519 ms
Total hot run time: 192151 ms

doris-robot · 2024-10-22T13:14:49Z

ClickBench: Total hot run time: 32.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01cce5acb604bf8079ffb616ba2e32811c730ef0, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.65	0.10	0.10
query5	0.52	0.50	0.52
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.57	0.49	0.50
query10	0.56	0.55	0.55
query11	0.16	0.11	0.10
query12	0.14	0.12	0.11
query13	0.60	0.60	0.60
query14	2.74	2.74	2.72
query15	0.88	0.82	0.82
query16	0.39	0.38	0.39
query17	1.05	1.04	0.99
query18	0.20	0.20	0.20
query19	1.91	1.89	2.00
query20	0.01	0.01	0.01
query21	15.36	0.62	0.59
query22	2.76	2.20	2.37
query23	16.80	0.92	0.77
query24	3.22	0.65	1.29
query25	0.35	0.07	0.10
query26	0.37	0.14	0.13
query27	0.04	0.05	0.03
query28	10.72	1.08	1.07
query29	12.58	3.33	3.29
query30	0.24	0.05	0.06
query31	2.87	0.37	0.38
query32	3.28	0.46	0.46
query33	3.03	3.03	3.11
query34	17.08	4.41	4.42
query35	4.58	4.52	4.51
query36	0.69	0.50	0.48
query37	0.09	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.36 s
Total hot run time: 32.75 s

xinyiZzz · 2024-10-24T11:56:45Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/runtime/memory/heap_profiler.cpp

doris-robot · 2024-10-24T13:22:33Z

TPC-H: Total hot run time: 41814 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba35897bb785bef3165786982d46430a679b1c60, data reload: false

------ Round 1 ----------------------------------
q1	17571	7651	7621	7621
q2	2021	295	322	295
q3	11986	1085	1224	1085
q4	10578	862	767	767
q5	7772	3149	3101	3101
q6	240	153	149	149
q7	1025	627	614	614
q8	9374	2002	1931	1931
q9	6997	6493	6489	6489
q10	7078	2411	2453	2411
q11	445	250	252	250
q12	413	219	222	219
q13	17801	2986	3018	2986
q14	250	215	218	215
q15	571	512	511	511
q16	662	585	585	585
q17	989	494	619	494
q18	7315	6728	6664	6664
q19	1351	1023	922	922
q20	479	197	189	189
q21	4113	3322	3362	3322
q22	1123	1014	994	994
Total cold run time: 110154 ms
Total hot run time: 41814 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7307	7257	7282	7257
q2	351	249	246	246
q3	3082	2970	3013	2970
q4	2110	1861	1835	1835
q5	5773	5796	5828	5796
q6	238	147	144	144
q7	2254	1834	1785	1785
q8	3420	3505	3540	3505
q9	8871	8939	8888	8888
q10	3601	3598	3550	3550
q11	574	491	520	491
q12	847	654	623	623
q13	11474	3179	3203	3179
q14	324	271	288	271
q15	583	537	519	519
q16	708	660	644	644
q17	1854	1609	1615	1609
q18	8434	7696	7628	7628
q19	1716	1439	1557	1439
q20	2153	1902	1890	1890
q21	5636	5364	5441	5364
q22	1103	1064	1069	1064
Total cold run time: 72413 ms
Total hot run time: 60697 ms

doris-robot · 2024-10-24T13:30:40Z

TeamCity be ut coverage result:
Function Coverage: 37.45% (9716/25942)
Line Coverage: 28.73% (80583/280471)
Region Coverage: 28.15% (41643/147938)
Branch Coverage: 24.72% (21158/85588)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ba35897bb785bef3165786982d46430a679b1c60_ba35897bb785bef3165786982d46430a679b1c60/report/index.html

doris-robot · 2024-10-24T13:34:40Z

TPC-DS: Total hot run time: 192925 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba35897bb785bef3165786982d46430a679b1c60, data reload: false

query1	906	407	425	407
query2	6234	2096	2063	2063
query3	8679	194	202	194
query4	35404	23793	23657	23657
query5	5150	466	488	466
query6	290	178	175	175
query7	4201	298	288	288
query8	304	233	237	233
query9	9542	2716	2708	2708
query10	499	274	265	265
query11	18057	15442	15326	15326
query12	156	110	103	103
query13	1579	437	415	415
query14	9751	6877	6944	6877
query15	243	168	172	168
query16	7969	451	484	451
query17	1584	624	601	601
query18	2155	323	307	307
query19	277	154	156	154
query20	122	117	111	111
query21	220	111	113	111
query22	4929	4776	4755	4755
query23	34936	34312	34044	34044
query24	10997	2801	2817	2801
query25	626	426	402	402
query26	1152	162	167	162
query27	2283	279	291	279
query28	7496	2445	2467	2445
query29	847	420	428	420
query30	262	159	164	159
query31	1083	808	800	800
query32	95	57	56	56
query33	761	301	302	301
query34	932	515	523	515
query35	1096	891	900	891
query36	1117	944	940	940
query37	154	88	88	88
query38	4377	4292	4281	4281
query39	1488	1416	1451	1416
query40	203	99	99	99
query41	50	46	50	46
query42	121	106	102	102
query43	528	515	505	505
query44	1252	817	804	804
query45	201	165	164	164
query46	1157	719	708	708
query47	1952	1849	1881	1849
query48	422	328	323	323
query49	926	434	423	423
query50	811	413	386	386
query51	7042	6896	7022	6896
query52	97	87	88	87
query53	255	180	178	178
query54	1036	419	419	419
query55	77	78	78	78
query56	272	251	255	251
query57	1311	1160	1166	1160
query58	236	241	228	228
query59	3217	3004	2986	2986
query60	286	283	259	259
query61	104	99	98	98
query62	861	666	686	666
query63	231	183	180	180
query64	4087	635	625	625
query65	3322	3200	3233	3200
query66	798	296	299	296
query67	16099	15750	15767	15750
query68	4452	568	551	551
query69	458	293	295	293
query70	1196	1155	1159	1155
query71	335	280	284	280
query72	6954	3987	3952	3952
query73	787	363	363	363
query74	10047	8938	8988	8938
query75	3514	2690	2700	2690
query76	2770	893	855	855
query77	633	292	287	287
query78	10662	9690	9642	9642
query79	1835	599	608	599
query80	762	444	467	444
query81	530	246	241	241
query82	753	144	137	137
query83	301	137	134	134
query84	260	71	71	71
query85	1328	294	285	285
query86	433	300	299	299
query87	4798	4717	4719	4717
query88	3619	2214	2198	2198
query89	407	303	287	287
query90	2065	184	182	182
query91	145	104	103	103
query92	70	48	49	48
query93	2183	541	544	541
query94	897	294	283	283
query95	340	249	245	245
query96	626	277	275	275
query97	2894	2740	2717	2717
query98	208	200	196	196
query99	1598	1316	1336	1316
Total cold run time: 302767 ms
Total hot run time: 192925 ms

doris-robot · 2024-10-24T13:40:12Z

ClickBench: Total hot run time: 33.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba35897bb785bef3165786982d46430a679b1c60, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.02
query3	0.22	0.07	0.07
query4	1.65	0.10	0.11
query5	0.41	0.42	0.40
query6	1.13	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.02	0.03
query9	0.57	0.51	0.51
query10	0.57	0.54	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.62	0.60	0.58
query14	2.72	2.86	2.84
query15	0.88	0.82	0.82
query16	0.39	0.38	0.39
query17	1.10	1.00	1.08
query18	0.20	0.19	0.19
query19	1.98	1.90	1.95
query20	0.01	0.00	0.01
query21	15.37	0.60	0.57
query22	2.85	2.69	2.14
query23	16.96	0.97	0.91
query24	3.50	1.34	1.44
query25	0.20	0.09	0.08
query26	0.46	0.14	0.13
query27	0.04	0.05	0.04
query28	9.91	1.09	1.07
query29	12.57	3.35	3.37
query30	0.25	0.07	0.06
query31	2.88	0.40	0.38
query32	3.25	0.46	0.46
query33	2.98	3.02	3.02
query34	17.08	4.42	4.44
query35	4.49	4.48	4.49
query36	0.67	0.48	0.46
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.13	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.81 s
Total hot run time: 33.49 s

xinyiZzz · 2024-10-25T02:58:58Z

run buildall

doris-robot · 2024-10-25T04:20:32Z

TPC-H: Total hot run time: 41684 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f87ba4b7313be8827a7a7ab41613d08472b7dfe, data reload: false

------ Round 1 ----------------------------------
q1	17576	7798	7443	7443
q2	2015	299	273	273
q3	12261	1061	1144	1061
q4	10558	862	954	862
q5	7766	3161	3152	3152
q6	242	147	143	143
q7	1022	604	611	604
q8	9371	2075	2024	2024
q9	6718	6520	6455	6455
q10	7098	2392	2437	2392
q11	440	251	242	242
q12	407	211	215	211
q13	17782	3035	3022	3022
q14	239	214	225	214
q15	565	521	527	521
q16	658	582	593	582
q17	988	613	518	518
q18	7503	6751	6609	6609
q19	1331	1041	1017	1017
q20	490	183	180	180
q21	3960	3257	3191	3191
q22	1113	1011	968	968
Total cold run time: 110103 ms
Total hot run time: 41684 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7339	7324	7343	7324
q2	348	247	239	239
q3	3073	2988	3148	2988
q4	2116	1836	1868	1836
q5	5780	5792	5819	5792
q6	226	140	144	140
q7	2278	1828	1871	1828
q8	3392	3434	3502	3434
q9	8958	9044	8972	8972
q10	3610	3603	3584	3584
q11	591	508	496	496
q12	829	618	621	618
q13	10073	3175	3181	3175
q14	320	288	276	276
q15	562	533	530	530
q16	686	656	651	651
q17	1854	1623	1669	1623
q18	8296	7721	7735	7721
q19	1741	1643	1669	1643
q20	2123	1871	1906	1871
q21	5591	5472	5506	5472
q22	1156	1056	1088	1056
Total cold run time: 70942 ms
Total hot run time: 61269 ms

doris-robot · 2024-10-25T04:23:49Z

TeamCity be ut coverage result:
Function Coverage: 37.44% (9715/25950)
Line Coverage: 28.72% (80582/280558)
Region Coverage: 28.13% (41641/148009)
Branch Coverage: 24.69% (21155/85666)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2f87ba4b7313be8827a7a7ab41613d08472b7dfe_2f87ba4b7313be8827a7a7ab41613d08472b7dfe/report/index.html

doris-robot · 2024-10-25T04:32:35Z

TPC-DS: Total hot run time: 192222 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f87ba4b7313be8827a7a7ab41613d08472b7dfe, data reload: false

query1	851	427	404	404
query2	6244	2149	2024	2024
query3	8673	205	199	199
query4	34954	23643	23602	23602
query5	4573	454	442	442
query6	286	167	167	167
query7	4204	302	286	286
query8	303	244	253	244
query9	9550	2752	2705	2705
query10	498	259	249	249
query11	18100	15296	15363	15296
query12	151	100	102	100
query13	1572	429	407	407
query14	9640	6914	6902	6902
query15	234	176	173	173
query16	7996	500	500	500
query17	1532	592	567	567
query18	2183	302	316	302
query19	230	158	160	158
query20	123	111	116	111
query21	211	114	117	114
query22	4799	4684	4788	4684
query23	34744	34420	34018	34018
query24	10947	2790	2788	2788
query25	638	410	395	395
query26	1127	162	162	162
query27	2335	287	284	284
query28	7576	2452	2478	2452
query29	845	426	430	426
query30	269	161	161	161
query31	1042	791	815	791
query32	98	53	55	53
query33	769	277	284	277
query34	921	505	516	505
query35	1058	912	883	883
query36	1121	972	946	946
query37	137	74	77	74
query38	4279	4243	4320	4243
query39	1468	1441	1429	1429
query40	203	101	100	100
query41	49	49	47	47
query42	121	103	103	103
query43	545	513	504	504
query44	1215	816	802	802
query45	181	169	169	169
query46	1158	702	700	700
query47	1938	1813	1872	1813
query48	427	327	327	327
query49	948	416	408	408
query50	805	398	396	396
query51	7106	6931	7008	6931
query52	125	86	88	86
query53	255	178	176	176
query54	1157	390	391	390
query55	78	78	76	76
query56	256	240	242	240
query57	1262	1180	1163	1163
query58	223	218	195	195
query59	3145	3163	3087	3087
query60	275	247	238	238
query61	103	114	110	110
query62	839	669	673	669
query63	219	190	184	184
query64	3960	629	628	628
query65	3290	3208	3228	3208
query66	821	299	318	299
query67	16199	15711	15669	15669
query68	4779	562	568	562
query69	436	262	259	259
query70	1226	1130	1153	1130
query71	327	276	262	262
query72	6215	3977	3922	3922
query73	769	354	366	354
query74	10246	9108	8956	8956
query75	3441	2623	2661	2623
query76	2837	864	1072	864
query77	390	264	264	264
query78	10686	9576	9616	9576
query79	1712	591	610	591
query80	1056	415	424	415
query81	564	238	245	238
query82	637	115	118	115
query83	226	136	140	136
query84	248	67	66	66
query85	1216	296	293	293
query86	367	307	300	300
query87	4729	4700	4696	4696
query88	3340	2213	2278	2213
query89	399	291	288	288
query90	1952	185	186	185
query91	136	104	99	99
query92	54	48	51	48
query93	1735	524	528	524
query94	887	299	297	297
query95	348	244	253	244
query96	618	282	291	282
query97	2874	2736	2684	2684
query98	219	197	191	191
query99	1544	1334	1304	1304
Total cold run time: 299397 ms
Total hot run time: 192222 ms

doris-robot · 2024-10-25T04:38:06Z

ClickBench: Total hot run time: 32.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f87ba4b7313be8827a7a7ab41613d08472b7dfe, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.43	0.40	0.42
query6	1.15	0.66	0.64
query7	0.02	0.01	0.03
query8	0.04	0.03	0.03
query9	0.57	0.50	0.53
query10	0.57	0.55	0.53
query11	0.14	0.10	0.11
query12	0.13	0.10	0.10
query13	0.60	0.59	0.60
query14	2.71	2.75	2.71
query15	0.89	0.82	0.83
query16	0.38	0.36	0.36
query17	1.05	0.98	1.08
query18	0.20	0.20	0.19
query19	1.94	1.85	1.96
query20	0.02	0.01	0.02
query21	15.36	0.61	0.61
query22	2.90	1.84	2.10
query23	16.78	1.37	0.78
query24	3.05	1.05	1.57
query25	0.13	0.22	0.26
query26	0.33	0.14	0.14
query27	0.04	0.04	0.04
query28	10.08	1.10	1.07
query29	12.57	3.31	3.30
query30	0.24	0.06	0.05
query31	2.85	0.38	0.37
query32	3.29	0.45	0.46
query33	2.99	3.00	3.04
query34	17.06	4.49	4.47
query35	4.51	4.58	4.51
query36	0.65	0.47	0.50
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.13	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.09 s
Total hot run time: 32.72 s

github-actions · 2024-10-28T11:25:20Z

PR approved by at least one committer and no changes requested.

wangbo

LGTM

…and automatically generate dot (apache#42059)" This reverts commit b3b04ab.

…matically generate dot (apache#42059)

### What problem does this PR solve? pick [opt](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc tcache #38185 [opt](memory) Support Jemalloc heap profile start at runtime and automatically generate dot #42059

github-actions bot reviewed Oct 17, 2024

View reviewed changes

be/src/runtime/memory/heap_profiler.cpp Show resolved Hide resolved

yiguolei previously approved these changes Oct 22, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 22, 2024

github-actions bot added the reviewed label Oct 22, 2024

wangbo previously approved these changes Oct 22, 2024

View reviewed changes

xinyiZzz added 2 commits October 24, 2024 19:31

1

9fa57ad

2

ba35897

xinyiZzz dismissed stale reviews from wangbo and yiguolei via ba35897 October 24, 2024 11:56

xinyiZzz force-pushed the 20241016_fix_memory branch from 01cce5a to ba35897 Compare October 24, 2024 11:56

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 24, 2024

github-actions bot reviewed Oct 24, 2024

View reviewed changes

be/src/runtime/memory/heap_profiler.cpp Show resolved Hide resolved

Merge branch 'master' into 20241016_fix_memory

2f87ba4

xinyiZzz added dev/2.1.x dev/3.0.x labels Oct 25, 2024

yiguolei approved these changes Oct 28, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 28, 2024

wangbo approved these changes Oct 29, 2024

View reviewed changes

wangbo merged commit b3b04ab into apache:master Oct 29, 2024
24 of 27 checks passed

github-actions bot added the cherry-pick-conflict-in-3.0 label Oct 29, 2024

hubgeter added a commit to hubgeter/doris that referenced this pull request Nov 11, 2024

Revert "[opt](memory) Support Jemalloc heap profile start at runtime …

975af55

…and automatically generate dot (apache#42059)" This reverts commit b3b04ab.

xinyiZzz mentioned this pull request Dec 3, 2024

[pick](branch-2.1) pick #42059 #44938

Merged

16 tasks

yiguolei pushed a commit that referenced this pull request Dec 4, 2024

[pick](branch-2.1) pick #42059 (#44938)

02fdf53

yiguolei added dev/2.1.8-merged and removed dev/2.1.x labels Dec 4, 2024

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Dec 6, 2024

[opt](memory) Support Jemalloc heap profile start at runtime and auto…

2bb3c1d

…matically generate dot (apache#42059)

xinyiZzz mentioned this pull request Dec 6, 2024

[pick](branch-3.0) pick #38185 #42059 #45092

Merged

16 tasks

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Dec 16, 2024

[opt](memory) Support Jemalloc heap profile start at runtime and auto…

904596a

…matically generate dot (apache#42059)

yiguolei added dev/3.0.4-merged and removed dev/3.0.x cherry-pick-conflict-in-3.0 labels Dec 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[opt](memory) Support Jemalloc heap profile start at runtime and automatically generate dot #42059

[opt](memory) Support Jemalloc heap profile start at runtime and automatically generate dot #42059

xinyiZzz commented Oct 17, 2024 •

edited

Loading

doris-robot commented Oct 17, 2024

xinyiZzz commented Oct 17, 2024

github-actions bot left a comment

doris-robot commented Oct 17, 2024

doris-robot commented Oct 17, 2024

doris-robot commented Oct 17, 2024

github-actions bot commented Oct 22, 2024

github-actions bot commented Oct 22, 2024

wangbo left a comment

xinyiZzz commented Oct 22, 2024

doris-robot commented Oct 22, 2024

doris-robot commented Oct 22, 2024

doris-robot commented Oct 22, 2024

xinyiZzz commented Oct 24, 2024

github-actions bot left a comment

doris-robot commented Oct 24, 2024

doris-robot commented Oct 24, 2024

doris-robot commented Oct 24, 2024

doris-robot commented Oct 24, 2024

xinyiZzz commented Oct 25, 2024

doris-robot commented Oct 25, 2024

doris-robot commented Oct 25, 2024

doris-robot commented Oct 25, 2024

doris-robot commented Oct 25, 2024

github-actions bot commented Oct 28, 2024

wangbo left a comment

[opt](memory) Support Jemalloc heap profile start at runtime and automatically generate dot #42059

[opt](memory) Support Jemalloc heap profile start at runtime and automatically generate dot #42059

Conversation

xinyiZzz commented Oct 17, 2024 • edited Loading

doris-robot commented Oct 17, 2024

xinyiZzz commented Oct 17, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

doris-robot commented Oct 17, 2024

doris-robot commented Oct 17, 2024

doris-robot commented Oct 17, 2024

github-actions bot commented Oct 22, 2024

github-actions bot commented Oct 22, 2024

wangbo left a comment

Choose a reason for hiding this comment

xinyiZzz commented Oct 22, 2024

doris-robot commented Oct 22, 2024

doris-robot commented Oct 22, 2024

doris-robot commented Oct 22, 2024

xinyiZzz commented Oct 24, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

doris-robot commented Oct 24, 2024

doris-robot commented Oct 24, 2024

doris-robot commented Oct 24, 2024

doris-robot commented Oct 24, 2024

xinyiZzz commented Oct 25, 2024

doris-robot commented Oct 25, 2024

doris-robot commented Oct 25, 2024

doris-robot commented Oct 25, 2024

doris-robot commented Oct 25, 2024

github-actions bot commented Oct 28, 2024

wangbo left a comment

Choose a reason for hiding this comment

xinyiZzz commented Oct 17, 2024 •

edited

Loading