Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](profile) Operator of same pipeline should have same indent level #47921

Merged
merged 3 commits into from
Feb 18, 2025

Conversation

zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Feb 14, 2025

What problem does this PR solve?

Operator of same pipeline should have same indent level

Master

MergedProfile
     Fragments:
       Fragment 0:
         Pipeline : 0(instance_num=16):
            - WaitWorkerTime: avg 29.266us, max 40.353us, min 22.437us
           RESULT_SINK_OPERATOR (id=0):
              - CloseTime: avg 126.883us, max 923.120us, min 9.218us
              - ExecTime: avg 391.625us, max 1.224ms, min 173.281us
              ...
              - WaitForDependency[RESULT_SINK_OPERATOR_DEPENDENCY]Time: avg 0ns, max 0ns, min 0ns
             LOCAL_EXCHANGE_OPERATOR (PASSTHROUGH) (id=-1):
                - BlocksProduced: sum 0, avg 0, max 0, min 0
                - CloseTime: avg 0ns, max 0ns, min 0ns
                - ExecTime: avg 29.423us, max 38.958us, min 20.25us
                ...
                - WaitForDependency[LOCAL_EXCHANGE_OPERATOR_DEPENDENCY]Time: avg 33.716ms, max 33.805ms, min 33.531ms
         Pipeline : 1(instance_num=1):
            - WaitWorkerTime: avg 26.452us, max 26.452us, min 26.452us
           LOCAL_EXCHANGE_SINK_OPERATOR (PASSTHROUGH) (id=-1):
              - CloseTime: avg 146.343us, max 146.343us, min 146.343us
              - ExecTime: avg 276.967us, max 276.967us, min 276.967us
              - InitTime: avg 117.846us, max 117.846us, min 117.846us
              - InputRows: sum 0, avg 0, max 0, min 0
              ...
              - WaitForDependency[LOCAL_EXCHANGE_SINK_DEPENDENCY]Time: avg 0ns, max 0ns, min 0ns
             OLAP_SCAN_OPERATOR (id=0. nereids_id=122. table name = test_bloom_filter_hit(test_bloom_filter_hit)):
                - BlocksProduced: sum 0, avg 0, max 0, min 0
                - CloseTime: avg 127.942us, max 127.942us, min 127.942us
                - ExecTime: avg 37.62ms, max 37.62ms, min 37.62ms
                - InitTime: avg 3.267ms, max 3.267ms, min 3.267ms
                ...
                - WaitForDependency[OLAP_SCAN_OPERATOR_DEPENDENCY]Time: avg 20.935ms, max 20.935ms, min 20.935ms

After opt

MergedProfile
     Fragments:
       Fragment 0:
         Pipeline : 0(instance_num=1):
            - WaitWorkerTime: avg 25.288us, max 25.288us, min 25.288us
           RESULT_SINK_OPERATOR (id=0):
              - CloseTime: avg 3.262us, max 3.262us, min 3.262us
              - ExecTime: avg 181.520us, max 181.520us, min 181.520us
              - InputRows: sum 100, avg 100, max 100, min 100
              ...
              - WaitForDependency[RESULT_SINK_OPERATOR_DEPENDENCY]Time: avg 0ns, max 0ns, min 0ns
           EXCHANGE_OPERATOR (id=17):
              - BlocksProduced: sum 1, avg 1, max 1, min 1
              - ExecTime: avg 149.70us, max 149.70us, min 149.70us
              ...
              - RowsProduced: sum 100, avg 100, max 100, min 100
              - WaitForDependencyTime: avg 0ns, max 0ns, min 0ns
                - WaitForData0: avg 5sec473ms, max 5sec473ms, min 5sec473ms
       Fragment 1:
         Pipeline : 0(instance_num=48):
            - WaitWorkerTime: avg 15.408us, max 46.623us, min 5.290us
           DATA_STREAM_SINK_OPERATOR (id=17,dst_id=17):
              - BlocksProduced: sum 48, avg 1, max 1, min 1
              - ExecTime: avg 23.363us, max 40.711us, min 13.443us
              - InputRows: sum 100, avg 2, max 100, min 0
              ...
              - WaitForDependencyTime: avg 0ns, max 0ns, min 0ns
                - WaitForRpcBufferQueue: avg 0ns, max 0ns, min 0ns
           LOCAL_EXCHANGE_OPERATOR (LOCAL_MERGE_SORT) (id=-4):
              - BlocksProduced: sum 1, avg 0, max 1, min 0
              - CloseTime: avg 0ns, max 0ns, min 0ns
              - ExecTime: avg 15.815us, max 699.196us, min 620ns
              ...
              - RowsProduced: sum 100, avg 2, max 100, min 0
              - WaitForDependencyTime: avg 5sec472ms, max 5sec472ms, min 5sec472

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

Copy link
Contributor

PR approved by anyone and no changes requested.

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31856 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d95ada3786998cc4614e9192732c28b199b0c1b5, data reload: false

------ Round 1 ----------------------------------
q1	17589	5284	5108	5108
q2	2055	305	173	173
q3	10409	1243	766	766
q4	10209	1031	564	564
q5	7504	2687	2297	2297
q6	183	165	137	137
q7	899	747	612	612
q8	9288	1288	1291	1288
q9	4870	4709	4628	4628
q10	6829	2318	1880	1880
q11	458	290	259	259
q12	352	358	223	223
q13	17765	3647	3099	3099
q14	229	224	206	206
q15	511	465	465	465
q16	620	622	578	578
q17	577	861	337	337
q18	6503	6117	6153	6117
q19	1466	955	555	555
q20	330	341	192	192
q21	2864	2157	2053	2053
q22	365	343	319	319
Total cold run time: 101875 ms
Total hot run time: 31856 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5166	5127	5084	5084
q2	235	324	232	232
q3	2187	2646	2338	2338
q4	1385	1802	1374	1374
q5	4246	4115	4123	4115
q6	208	162	128	128
q7	1880	1823	1654	1654
q8	2603	2581	2584	2581
q9	7124	7127	7103	7103
q10	3006	3241	2785	2785
q11	567	511	514	511
q12	679	808	661	661
q13	3548	3863	3304	3304
q14	295	293	274	274
q15	501	464	462	462
q16	636	692	616	616
q17	1151	1605	1345	1345
q18	7647	7426	7320	7320
q19	814	864	877	864
q20	1932	2029	1867	1867
q21	5478	4972	4770	4770
q22	579	571	528	528
Total cold run time: 51867 ms
Total hot run time: 49916 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190192 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d95ada3786998cc4614e9192732c28b199b0c1b5, data reload: false

query1	1323	945	919	919
query2	6233	1789	1804	1789
query3	10903	4407	4453	4407
query4	55262	25909	23476	23476
query5	5040	545	487	487
query6	329	211	200	200
query7	4892	517	296	296
query8	322	250	234	234
query9	5663	2528	2535	2528
query10	437	304	252	252
query11	15061	15044	14912	14912
query12	164	112	109	109
query13	1053	522	403	403
query14	10781	6812	6762	6762
query15	211	202	178	178
query16	7075	633	462	462
query17	1093	721	606	606
query18	1594	416	321	321
query19	203	218	166	166
query20	123	124	122	122
query21	252	125	103	103
query22	4347	4586	4335	4335
query23	34046	33290	33206	33206
query24	5771	2428	2405	2405
query25	466	461	395	395
query26	702	286	158	158
query27	1695	506	339	339
query28	2911	2432	2421	2421
query29	600	559	430	430
query30	210	195	161	161
query31	896	846	841	841
query32	82	60	66	60
query33	449	357	296	296
query34	797	854	518	518
query35	791	820	768	768
query36	982	1015	913	913
query37	128	99	83	83
query38	4211	4356	4229	4229
query39	1499	1463	1446	1446
query40	209	120	105	105
query41	50	49	53	49
query42	121	106	108	106
query43	487	514	471	471
query44	1355	798	799	798
query45	180	171	170	170
query46	868	1062	688	688
query47	1827	1878	1743	1743
query48	410	439	330	330
query49	686	540	446	446
query50	734	756	424	424
query51	4302	4244	4229	4229
query52	116	114	103	103
query53	246	283	204	204
query54	488	496	432	432
query55	91	82	81	81
query56	274	299	268	268
query57	1140	1180	1170	1170
query58	252	243	248	243
query59	2679	2910	2521	2521
query60	293	275	268	268
query61	121	148	124	124
query62	761	755	698	698
query63	250	209	196	196
query64	1857	1032	701	701
query65	3290	3187	3111	3111
query66	701	391	302	302
query67	15895	15508	15298	15298
query68	5303	775	517	517
query69	515	298	275	275
query70	1245	1116	1126	1116
query71	450	314	287	287
query72	6400	3730	3706	3706
query73	1333	767	344	344
query74	9069	9170	8880	8880
query75	3271	3177	2732	2732
query76	3745	1251	754	754
query77	562	372	280	280
query78	9851	9990	9274	9274
query79	2810	806	592	592
query80	730	557	437	437
query81	499	278	236	236
query82	658	123	94	94
query83	166	170	164	164
query84	286	89	78	78
query85	828	349	292	292
query86	394	306	283	283
query87	4380	4542	4422	4422
query88	3525	2167	2171	2167
query89	415	324	285	285
query90	1816	203	189	189
query91	136	138	110	110
query92	70	58	61	58
query93	1976	1019	567	567
query94	693	382	302	302
query95	357	265	257	257
query96	484	565	277	277
query97	2821	2826	2736	2736
query98	228	201	199	199
query99	1699	1370	1267	1267
Total cold run time: 294995 ms
Total hot run time: 190192 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.22 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d95ada3786998cc4614e9192732c28b199b0c1b5, data reload: false

query1	0.06	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.40	0.42	0.41
query6	1.15	0.68	0.66
query7	0.03	0.02	0.02
query8	0.03	0.03	0.03
query9	0.59	0.51	0.53
query10	0.57	0.59	0.56
query11	0.14	0.10	0.11
query12	0.13	0.11	0.12
query13	0.61	0.60	0.60
query14	2.72	2.70	2.73
query15	0.92	0.84	0.84
query16	0.38	0.37	0.40
query17	1.01	1.01	1.01
query18	0.21	0.19	0.19
query19	1.89	1.75	1.94
query20	0.01	0.02	0.01
query21	15.37	0.91	0.55
query22	0.75	1.20	0.63
query23	14.94	1.37	0.64
query24	9.93	1.45	0.66
query25	0.34	0.15	0.09
query26	1.00	0.18	0.13
query27	0.04	0.05	0.05
query28	6.81	0.88	0.43
query29	12.54	4.03	3.30
query30	0.26	0.09	0.06
query31	2.82	0.58	0.39
query32	3.22	0.54	0.47
query33	2.98	2.99	3.10
query34	15.73	5.11	4.46
query35	4.51	4.49	4.49
query36	0.66	0.49	0.47
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.02	0.02	0.02
query40	0.17	0.13	0.14
query41	0.09	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.18 s
Total hot run time: 30.22 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31356 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 797b2e81ad610a1f3b9bfbc198e2d837f5491c3f, data reload: false

------ Round 1 ----------------------------------
q1	17595	5279	5052	5052
q2	2084	314	177	177
q3	10648	1298	752	752
q4	10290	999	560	560
q5	8061	2420	2264	2264
q6	197	168	136	136
q7	903	818	616	616
q8	9291	1291	1018	1018
q9	4805	4754	4761	4754
q10	6821	2299	1885	1885
q11	486	280	251	251
q12	345	343	214	214
q13	17794	3634	3004	3004
q14	221	225	203	203
q15	506	467	468	467
q16	622	610	577	577
q17	570	855	336	336
q18	6621	6247	6119	6119
q19	1808	951	544	544
q20	300	315	188	188
q21	2667	2120	1945	1945
q22	359	328	294	294
Total cold run time: 102994 ms
Total hot run time: 31356 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5182	5066	5302	5066
q2	238	331	229	229
q3	2112	2650	2307	2307
q4	1478	1826	1387	1387
q5	4214	4111	4164	4111
q6	208	162	123	123
q7	1840	1832	1700	1700
q8	2604	2600	2492	2492
q9	7193	7180	7138	7138
q10	3023	3245	2769	2769
q11	576	516	501	501
q12	661	761	623	623
q13	3419	3776	3306	3306
q14	293	307	280	280
q15	502	467	475	467
q16	639	689	643	643
q17	1109	1575	1380	1380
q18	7582	7462	7348	7348
q19	764	816	863	816
q20	1962	2019	1872	1872
q21	5426	5095	4730	4730
q22	634	589	532	532
Total cold run time: 51659 ms
Total hot run time: 49820 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189713 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 797b2e81ad610a1f3b9bfbc198e2d837f5491c3f, data reload: false

query1	1307	968	932	932
query2	6217	1819	1801	1801
query3	11114	4666	4534	4534
query4	53719	24419	23077	23077
query5	5265	549	470	470
query6	357	208	186	186
query7	5156	496	296	296
query8	326	243	230	230
query9	6383	2506	2522	2506
query10	387	312	249	249
query11	15177	15049	14856	14856
query12	156	109	108	108
query13	1152	536	373	373
query14	10277	6298	6861	6298
query15	201	204	181	181
query16	6976	665	510	510
query17	1084	722	592	592
query18	1521	414	354	354
query19	192	195	160	160
query20	122	124	119	119
query21	202	122	112	112
query22	4431	4728	4574	4574
query23	33985	33364	33568	33364
query24	5789	2422	2426	2422
query25	446	473	413	413
query26	774	279	158	158
query27	1994	487	332	332
query28	2799	2424	2391	2391
query29	579	533	411	411
query30	211	190	158	158
query31	874	878	808	808
query32	99	60	61	60
query33	443	363	293	293
query34	748	860	492	492
query35	801	839	754	754
query36	963	1015	903	903
query37	122	95	73	73
query38	4229	4358	4295	4295
query39	1520	1476	1456	1456
query40	208	114	103	103
query41	51	49	50	49
query42	119	110	112	110
query43	499	498	482	482
query44	1273	798	799	798
query45	180	171	165	165
query46	873	1092	660	660
query47	1833	1867	1761	1761
query48	374	409	304	304
query49	702	506	428	428
query50	699	777	424	424
query51	4295	4342	4244	4244
query52	108	107	93	93
query53	246	266	183	183
query54	478	480	403	403
query55	84	83	77	77
query56	289	283	265	265
query57	1119	1168	1145	1145
query58	246	244	241	241
query59	2841	2933	2680	2680
query60	301	305	264	264
query61	124	119	123	119
query62	735	760	689	689
query63	234	189	189	189
query64	2461	1026	663	663
query65	3221	3126	3098	3098
query66	763	393	295	295
query67	15848	15642	15716	15642
query68	5366	770	495	495
query69	515	302	261	261
query70	1197	1141	1082	1082
query71	433	290	256	256
query72	5876	3682	3825	3682
query73	1066	731	346	346
query74	8933	8976	8704	8704
query75	3190	3145	2694	2694
query76	3902	1169	742	742
query77	523	374	279	279
query78	9978	9964	9328	9328
query79	2363	797	578	578
query80	598	523	460	460
query81	589	273	236	236
query82	543	123	99	99
query83	176	162	150	150
query84	287	92	70	70
query85	756	346	295	295
query86	380	295	285	285
query87	4460	4501	4397	4397
query88	3685	2182	2145	2145
query89	403	317	289	289
query90	1745	189	185	185
query91	128	138	106	106
query92	70	59	55	55
query93	1929	1001	567	567
query94	729	456	308	308
query95	361	267	246	246
query96	466	569	266	266
query97	2811	2847	2736	2736
query98	231	218	208	208
query99	1569	1391	1288	1288
Total cold run time: 293560 ms
Total hot run time: 189713 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.18 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 797b2e81ad610a1f3b9bfbc198e2d837f5491c3f, data reload: false

query1	0.04	0.05	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.61	0.11	0.11
query5	0.42	0.41	0.39
query6	1.16	0.67	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.61	0.51	0.52
query10	0.59	0.59	0.57
query11	0.15	0.11	0.10
query12	0.16	0.11	0.11
query13	0.61	0.61	0.59
query14	2.70	2.73	2.72
query15	0.92	0.85	0.86
query16	0.38	0.39	0.37
query17	1.03	1.03	1.04
query18	0.22	0.20	0.19
query19	1.88	1.81	1.98
query20	0.01	0.02	0.01
query21	15.36	0.91	0.54
query22	0.76	1.18	0.64
query23	14.97	1.39	0.64
query24	11.90	0.90	0.39
query25	0.35	0.13	0.08
query26	1.00	0.18	0.15
query27	0.06	0.05	0.05
query28	6.09	0.77	0.44
query29	12.56	3.88	3.28
query30	0.26	0.09	0.07
query31	2.81	0.59	0.38
query32	3.22	0.55	0.47
query33	2.99	3.03	3.02
query34	15.76	5.22	4.51
query35	4.58	4.52	4.53
query36	0.66	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.69 s
Total hot run time: 30.18 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 43.38% (11330/26118)
Line Coverage: 33.37% (95120/285046)
Region Coverage: 32.15% (48730/151585)
Branch Coverage: 27.97% (24573/87866)
Coverage Report: http://coverage.selectdb-in.cc/coverage/797b2e81ad610a1f3b9bfbc198e2d837f5491c3f_797b2e81ad610a1f3b9bfbc198e2d837f5491c3f/report/index.html

@zhiqiang-hhhh
Copy link
Contributor Author

run external

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 18, 2025
@yiguolei yiguolei merged commit 9647eba into apache:master Feb 18, 2025
25 of 26 checks passed
@zhiqiang-hhhh zhiqiang-hhhh deleted the profile-indent branch February 18, 2025 01:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants