[enhance](nereids) add rule MultiDistinctSplit #45209

feiniaofeiafei · 2024-12-09T13:27:44Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This pr add a rewrite rule, which can do this 2 type of rewrite:

This rewrite can greatly improve the execution speed of multiple count(distinct) operations. When 3be, ndv=10000000, the performance can be improved by three to four times.

select count(distinct a),count(distinct b),count(distinct c) from t;
->
with tmp as (select * from t) 
select * from (select count(distinct a) from tmp) t1 cross join  (select count(distinct b) from tmp) t2 cross join  (select count(distinct c) from tmp) t3

2.Before this PR, the following SQL statement would fail to execute due to an error: "The query contains multi count distinct or sum distinct, each can't have multi columns". This PR rewrites this type of SQL statement as follows, making it executable without an error.

select count(distinct a,d),count(distinct b,c),count(distinct c) from t;
->
with tmp as (select * from t) 
select * from (select count(distinct a,d) from tmp) t1 cross join  (select count(distinct b,c) from tmp) t2 cross join  (select count(distinct c) from tmp) t3

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-12-09T13:27:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

feiniaofeiafei · 2024-12-09T13:28:03Z

run buildall

doris-robot · 2024-12-09T14:06:40Z

TPC-H: Total hot run time: 40494 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb681c96b4fe6a558241aac28ee847320831475d, data reload: false

------ Round 1 ----------------------------------
q1	17982	7520	7407	7407
q2	2046	175	178	175
q3	10915	1082	1181	1082
q4	10530	757	715	715
q5	7588	2704	2712	2704
q6	235	147	143	143
q7	1001	654	590	590
q8	9421	1900	1935	1900
q9	6721	6578	6556	6556
q10	7188	2287	2331	2287
q11	467	271	258	258
q12	489	233	233	233
q13	18018	3301	3293	3293
q14	263	219	228	219
q15	597	533	535	533
q16	677	597	594	594
q17	1006	612	578	578
q18	7471	6776	6725	6725
q19	2372	953	976	953
q20	1507	182	178	178
q21	5075	3316	3056	3056
q22	380	315	317	315
Total cold run time: 111949 ms
Total hot run time: 40494 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7269	7208	7217	7208
q2	332	237	234	234
q3	3072	2909	2992	2909
q4	2123	1786	1806	1786
q5	5612	5637	5671	5637
q6	217	142	139	139
q7	2250	1809	1763	1763
q8	3398	3539	3405	3405
q9	9093	9010	9073	9010
q10	3581	3530	3568	3530
q11	596	502	515	502
q12	866	604	613	604
q13	13363	3189	3247	3189
q14	299	278	273	273
q15	580	524	514	514
q16	685	619	632	619
q17	1853	1665	1610	1610
q18	8169	7652	7568	7568
q19	1687	1518	1472	1472
q20	2114	1874	1869	1869
q21	5634	5407	5478	5407
q22	665	549	583	549
Total cold run time: 73458 ms
Total hot run time: 59797 ms

doris-robot · 2024-12-09T14:19:19Z

TPC-DS: Total hot run time: 196435 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb681c96b4fe6a558241aac28ee847320831475d, data reload: false

query1	1222	762	772	762
query2	6241	2055	2028	2028
query3	10962	4541	4437	4437
query4	66683	29060	23501	23501
query5	4871	471	481	471
query6	410	177	175	175
query7	5535	306	292	292
query8	329	252	224	224
query9	8705	2715	2699	2699
query10	419	254	243	243
query11	17271	15396	15952	15396
query12	152	110	105	105
query13	1454	440	404	404
query14	10516	7297	6862	6862
query15	224	185	194	185
query16	7219	466	449	449
query17	1198	577	581	577
query18	1915	305	316	305
query19	205	163	161	161
query20	121	114	112	112
query21	208	105	102	102
query22	4905	4389	4717	4389
query23	35661	34881	34488	34488
query24	5466	2502	2455	2455
query25	485	391	394	391
query26	700	157	160	157
query27	1985	278	289	278
query28	4427	2525	2442	2442
query29	665	400	413	400
query30	209	151	149	149
query31	1032	853	909	853
query32	72	58	84	58
query33	444	286	286	286
query34	945	513	538	513
query35	885	771	755	755
query36	1079	946	924	924
query37	125	70	74	70
query38	4501	4504	4460	4460
query39	1523	1468	1452	1452
query40	206	101	95	95
query41	44	42	49	42
query42	113	100	99	99
query43	556	499	489	489
query44	1209	835	858	835
query45	194	164	175	164
query46	1184	712	725	712
query47	2049	1987	1926	1926
query48	414	329	319	319
query49	741	402	376	376
query50	880	391	400	391
query51	7389	6967	7091	6967
query52	99	84	83	83
query53	249	177	174	174
query54	509	388	398	388
query55	78	76	73	73
query56	245	225	222	222
query57	1298	1131	1143	1131
query58	207	206	207	206
query59	3232	3041	3228	3041
query60	268	242	242	242
query61	106	100	104	100
query62	792	710	684	684
query63	209	184	182	182
query64	1361	644	611	611
query65	3268	3217	3206	3206
query66	648	281	293	281
query67	16074	15619	15531	15531
query68	4092	564	572	564
query69	424	246	253	246
query70	1202	1119	1131	1119
query71	327	246	265	246
query72	6142	3998	4035	3998
query73	776	343	350	343
query74	10112	8996	9129	8996
query75	3467	2652	2644	2644
query76	2005	1089	1134	1089
query77	493	268	261	261
query78	10575	9511	9401	9401
query79	1328	584	588	584
query80	860	422	433	422
query81	486	230	231	230
query82	1292	154	115	115
query83	161	139	152	139
query84	280	67	77	67
query85	848	287	297	287
query86	338	277	300	277
query87	4677	4526	4528	4526
query88	3604	2199	2182	2182
query89	413	297	294	294
query90	1984	185	182	182
query91	137	100	101	100
query92	62	52	49	49
query93	1811	544	531	531
query94	722	280	279	279
query95	351	241	240	240
query96	626	286	283	283
query97	2874	2669	2673	2669
query98	220	195	194	194
query99	1579	1332	1298	1298
Total cold run time: 318859 ms
Total hot run time: 196435 ms

doris-robot · 2024-12-09T14:24:48Z

ClickBench: Total hot run time: 32.27 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fb681c96b4fe6a558241aac28ee847320831475d, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.08
query4	1.62	0.11	0.10
query5	0.43	0.41	0.41
query6	1.16	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.51
query10	0.55	0.55	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	2.84	2.75	2.68
query15	0.88	0.82	0.82
query16	0.38	0.39	0.37
query17	1.07	1.08	1.06
query18	0.23	0.20	0.20
query19	1.88	1.89	2.05
query20	0.01	0.01	0.01
query21	15.36	0.59	0.57
query22	2.69	1.65	2.37
query23	17.07	1.11	0.76
query24	2.82	1.02	2.02
query25	0.23	0.14	0.08
query26	0.49	0.15	0.13
query27	0.04	0.05	0.03
query28	9.98	1.11	1.07
query29	12.52	3.17	3.18
query30	0.24	0.06	0.06
query31	2.87	0.39	0.36
query32	3.29	0.46	0.45
query33	2.97	3.00	3.04
query34	17.21	4.49	4.48
query35	4.56	4.43	4.56
query36	0.66	0.48	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.37 s
Total hot run time: 32.27 s

feiniaofeiafei · 2024-12-10T12:42:51Z

run buildall

feiniaofeiafei · 2024-12-10T13:42:04Z

run buildall

doris-robot · 2024-12-10T14:16:06Z

TPC-H: Total hot run time: 39674 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32f3621cf7b3bc651f9e30212cc6589578131989, data reload: false

------ Round 1 ----------------------------------
q1	17570	7466	7222	7222
q2	2043	175	170	170
q3	10688	1112	1137	1112
q4	10575	728	715	715
q5	7591	2717	2718	2717
q6	239	147	143	143
q7	990	634	608	608
q8	9258	1747	1897	1747
q9	6662	6460	6437	6437
q10	6990	2301	2319	2301
q11	465	250	256	250
q12	416	223	220	220
q13	17766	3065	3037	3037
q14	242	211	210	210
q15	580	539	532	532
q16	659	577	581	577
q17	966	617	511	511
q18	7263	6667	6708	6667
q19	1363	995	1034	995
q20	471	180	189	180
q21	3954	3350	3021	3021
q22	380	317	302	302
Total cold run time: 107131 ms
Total hot run time: 39674 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7256	7242	7188	7188
q2	323	227	222	222
q3	2873	2767	2880	2767
q4	2068	1780	1800	1780
q5	5628	5695	5607	5607
q6	221	145	147	145
q7	2248	1793	1845	1793
q8	3383	3545	3490	3490
q9	8927	8955	9044	8955
q10	3612	3507	3539	3507
q11	603	522	504	504
q12	835	632	626	626
q13	12144	3170	3213	3170
q14	303	267	273	267
q15	580	537	545	537
q16	691	633	636	633
q17	1828	1649	1615	1615
q18	8460	7677	7790	7677
q19	1762	1504	1511	1504
q20	2116	1910	1870	1870
q21	5650	5427	5527	5427
q22	642	592	588	588
Total cold run time: 72153 ms
Total hot run time: 59872 ms

doris-robot · 2024-12-10T14:28:46Z

TPC-DS: Total hot run time: 197910 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32f3621cf7b3bc651f9e30212cc6589578131989, data reload: false

query1	1250	964	911	911
query2	6230	2090	2001	2001
query3	10981	4380	4379	4379
query4	66596	29649	23571	23571
query5	5035	466	450	450
query6	434	185	188	185
query7	5611	297	293	293
query8	320	233	236	233
query9	9227	2712	2676	2676
query10	441	242	252	242
query11	17618	15277	15903	15277
query12	142	97	101	97
query13	1517	427	404	404
query14	10403	7245	7757	7245
query15	229	195	201	195
query16	7139	479	462	462
query17	1272	591	618	591
query18	1676	297	297	297
query19	202	157	150	150
query20	118	112	108	108
query21	212	108	103	103
query22	4909	4516	4508	4508
query23	34793	34669	35621	34669
query24	5509	2461	2476	2461
query25	475	391	404	391
query26	640	152	150	150
query27	1766	275	277	275
query28	4250	2456	2449	2449
query29	675	409	424	409
query30	207	153	158	153
query31	1028	830	849	830
query32	69	57	58	57
query33	426	302	315	302
query34	920	547	523	523
query35	891	786	765	765
query36	1089	963	995	963
query37	125	79	73	73
query38	4500	4501	4500	4500
query39	1499	1476	1471	1471
query40	197	103	98	98
query41	47	40	41	40
query42	104	97	102	97
query43	566	490	514	490
query44	1192	822	825	822
query45	193	163	167	163
query46	1175	728	741	728
query47	2035	1944	1982	1944
query48	409	340	307	307
query49	739	394	379	379
query50	838	385	395	385
query51	7411	7169	7062	7062
query52	95	86	85	85
query53	254	188	181	181
query54	502	384	403	384
query55	74	74	76	74
query56	255	239	228	228
query57	1274	1115	1132	1115
query58	231	208	228	208
query59	3162	3062	3045	3045
query60	275	248	266	248
query61	137	127	124	124
query62	789	694	685	685
query63	218	195	187	187
query64	1415	654	634	634
query65	3287	3183	3204	3183
query66	696	299	298	298
query67	15944	15789	15685	15685
query68	4025	560	544	544
query69	424	250	256	250
query70	1226	1152	1128	1128
query71	345	248	255	248
query72	6087	4098	4102	4098
query73	766	356	362	356
query74	9372	9000	9168	9000
query75	3407	2695	2662	2662
query76	1908	1088	1068	1068
query77	495	273	268	268
query78	10548	9470	9485	9470
query79	1456	595	596	595
query80	850	423	425	423
query81	480	235	233	233
query82	1367	120	117	117
query83	162	145	142	142
query84	280	71	71	71
query85	857	301	294	294
query86	343	297	285	285
query87	4725	4629	4658	4629
query88	3475	2218	2177	2177
query89	423	298	287	287
query90	1985	187	187	187
query91	142	108	104	104
query92	65	50	53	50
query93	1928	530	526	526
query94	764	287	334	287
query95	346	244	242	242
query96	612	281	278	278
query97	2851	2729	2681	2681
query98	223	190	196	190
query99	1603	1322	1324	1322
Total cold run time: 317478 ms
Total hot run time: 197910 ms

doris-robot · 2024-12-10T14:34:16Z

ClickBench: Total hot run time: 32.7 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 32f3621cf7b3bc651f9e30212cc6589578131989, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.06
query4	1.63	0.10	0.10
query5	0.42	0.41	0.39
query6	1.16	0.67	0.66
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.53	0.49
query10	0.58	0.57	0.55
query11	0.13	0.10	0.09
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	2.75	2.71	2.89
query15	0.91	0.82	0.82
query16	0.37	0.39	0.36
query17	1.05	1.05	1.06
query18	0.23	0.22	0.20
query19	1.96	1.84	2.02
query20	0.02	0.00	0.01
query21	15.36	0.57	0.58
query22	2.97	2.23	1.89
query23	17.06	0.90	0.88
query24	3.44	1.09	1.00
query25	0.30	0.10	0.10
query26	0.43	0.14	0.13
query27	0.04	0.04	0.07
query28	10.60	1.11	1.07
query29	12.53	3.28	3.26
query30	0.24	0.06	0.06
query31	2.87	0.39	0.37
query32	3.30	0.46	0.46
query33	2.98	2.99	3.05
query34	16.98	4.45	4.47
query35	4.52	4.52	4.56
query36	0.67	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.71 s
Total hot run time: 32.7 s

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAnalysis.java

fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/DistinctSplit.java

feiniaofeiafei · 2024-12-11T13:44:39Z

run buildall

feiniaofeiafei · 2024-12-11T13:45:04Z

run buildall

feiniaofeiafei · 2024-12-11T13:45:19Z

run buildall

doris-robot · 2024-12-11T14:20:08Z

TPC-H: Total hot run time: 39791 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 93ee3423bd7944d0a680a6a41f47103fcfeb8a95, data reload: false

------ Round 1 ----------------------------------
q1	17596	7460	7296	7296
q2	2050	172	162	162
q3	10602	1117	1148	1117
q4	10568	747	674	674
q5	7582	2674	2675	2674
q6	234	145	141	141
q7	993	611	613	611
q8	9227	1860	1919	1860
q9	6607	6400	6439	6400
q10	6987	2302	2287	2287
q11	461	255	250	250
q12	421	223	222	222
q13	17804	3041	3015	3015
q14	252	212	203	203
q15	580	526	530	526
q16	651	591	576	576
q17	983	615	533	533
q18	7195	6612	6712	6612
q19	1347	976	984	976
q20	462	175	180	175
q21	3980	3304	3166	3166
q22	380	320	315	315
Total cold run time: 106962 ms
Total hot run time: 39791 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7285	7228	7246	7228
q2	332	229	237	229
q3	2913	2800	2962	2800
q4	2130	1871	1897	1871
q5	5658	5668	5634	5634
q6	229	139	144	139
q7	2249	1838	1828	1828
q8	3411	3531	3516	3516
q9	8934	9106	9023	9023
q10	3594	3532	3544	3532
q11	619	511	515	511
q12	835	619	649	619
q13	12450	3236	3243	3236
q14	301	276	275	275
q15	582	527	538	527
q16	677	651	637	637
q17	1848	1629	1600	1600
q18	8322	7729	7699	7699
q19	1784	1517	1520	1517
q20	2102	1862	1861	1861
q21	5618	5511	5567	5511
q22	622	551	601	551
Total cold run time: 72495 ms
Total hot run time: 60344 ms

doris-robot · 2024-12-11T14:32:47Z

TPC-DS: Total hot run time: 196228 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 93ee3423bd7944d0a680a6a41f47103fcfeb8a95, data reload: false

query1	1244	935	978	935
query2	6251	2129	2071	2071
query3	10903	4468	4418	4418
query4	66831	29416	23549	23549
query5	4924	472	436	436
query6	405	186	174	174
query7	5541	300	297	297
query8	327	227	247	227
query9	9240	2698	2706	2698
query10	460	236	236	236
query11	17066	15178	15993	15178
query12	152	101	105	101
query13	1500	449	426	426
query14	9704	6702	7399	6702
query15	216	191	213	191
query16	7109	496	481	481
query17	1254	600	596	596
query18	1883	320	324	320
query19	209	172	154	154
query20	137	121	115	115
query21	218	105	108	105
query22	4721	4499	4484	4484
query23	34475	33978	33910	33910
query24	5673	2524	2548	2524
query25	481	411	399	399
query26	645	154	147	147
query27	1840	284	289	284
query28	4621	2483	2503	2483
query29	669	423	432	423
query30	212	158	205	158
query31	974	817	811	811
query32	71	55	57	55
query33	446	295	297	295
query34	936	542	515	515
query35	881	776	790	776
query36	1118	980	955	955
query37	124	73	75	73
query38	4437	4370	4380	4370
query39	1527	1451	1462	1451
query40	195	101	96	96
query41	47	41	41	41
query42	117	97	99	97
query43	547	506	497	497
query44	1210	858	852	852
query45	188	171	164	164
query46	1182	737	748	737
query47	2061	1900	1971	1900
query48	458	322	322	322
query49	736	409	402	402
query50	838	411	385	385
query51	7508	7295	7159	7159
query52	103	88	88	88
query53	251	177	176	176
query54	529	391	396	391
query55	81	86	75	75
query56	250	238	233	233
query57	1255	1114	1112	1112
query58	217	208	215	208
query59	3117	2943	2836	2836
query60	280	247	246	246
query61	155	108	105	105
query62	767	673	654	654
query63	205	177	188	177
query64	1362	674	706	674
query65	3259	3165	3165	3165
query66	706	302	304	302
query67	15841	15705	15617	15617
query68	4146	571	582	571
query69	419	247	251	247
query70	1144	1101	1127	1101
query71	367	255	239	239
query72	6422	4080	4066	4066
query73	787	358	358	358
query74	10136	9010	8967	8967
query75	3401	2668	2681	2668
query76	1834	1031	1078	1031
query77	521	273	271	271
query78	10408	9516	9407	9407
query79	2094	614	597	597
query80	1408	425	427	425
query81	518	226	221	221
query82	1258	124	121	121
query83	179	146	147	146
query84	286	76	69	69
query85	1070	315	312	312
query86	416	291	286	286
query87	4658	4658	4556	4556
query88	3867	2238	2194	2194
query89	420	293	287	287
query90	2006	185	186	185
query91	141	105	103	103
query92	67	48	50	48
query93	2959	556	546	546
query94	922	289	296	289
query95	351	248	248	248
query96	631	283	274	274
query97	2824	2662	2669	2662
query98	213	189	196	189
query99	1590	1310	1310	1310
Total cold run time: 320353 ms
Total hot run time: 196228 ms

doris-robot · 2024-12-11T14:38:14Z

ClickBench: Total hot run time: 32.23 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 93ee3423bd7944d0a680a6a41f47103fcfeb8a95, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.04
query3	0.23	0.08	0.06
query4	1.63	0.11	0.11
query5	0.43	0.42	0.41
query6	1.20	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.52	0.51
query10	0.55	0.58	0.54
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.61
query14	2.69	2.68	2.73
query15	0.88	0.82	0.82
query16	0.38	0.38	0.37
query17	1.02	1.07	1.03
query18	0.23	0.21	0.22
query19	1.91	1.83	2.01
query20	0.02	0.01	0.02
query21	15.36	0.60	0.57
query22	2.54	1.73	1.70
query23	17.06	0.81	0.98
query24	2.81	1.16	0.75
query25	0.20	0.10	0.07
query26	0.46	0.14	0.14
query27	0.04	0.04	0.03
query28	10.95	1.11	1.08
query29	12.55	3.33	3.33
query30	0.25	0.07	0.07
query31	2.84	0.38	0.37
query32	3.31	0.46	0.46
query33	3.01	3.02	3.10
query34	17.02	4.46	4.43
query35	4.49	4.45	4.47
query36	0.65	0.47	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.81 s
Total hot run time: 32.23 s

feiniaofeiafei · 2024-12-12T04:04:26Z

run buildall

doris-robot · 2024-12-12T04:23:30Z

TPC-H: Total hot run time: 40034 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fbdbae126e036a2cee3bdf56f805bae4d9454160, data reload: false

------ Round 1 ----------------------------------
q1	17660	7457	7252	7252
q2	2050	184	168	168
q3	10542	1143	1131	1131
q4	10221	801	716	716
q5	7603	2733	2726	2726
q6	235	146	147	146
q7	1001	619	602	602
q8	9239	1830	1859	1830
q9	6628	6482	6478	6478
q10	7059	2298	2319	2298
q11	476	249	259	249
q12	407	224	229	224
q13	17781	3021	2968	2968
q14	258	216	211	211
q15	570	518	519	518
q16	658	594	579	579
q17	979	577	528	528
q18	7357	6673	6697	6673
q19	1358	1016	914	914
q20	467	187	180	180
q21	3979	3328	3422	3328
q22	374	315	317	315
Total cold run time: 106902 ms
Total hot run time: 40034 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7282	7223	7204	7204
q2	334	230	235	230
q3	2869	2779	3057	2779
q4	2106	1853	1802	1802
q5	5530	5666	5614	5614
q6	225	143	141	141
q7	2208	1778	1775	1775
q8	3340	3528	3475	3475
q9	8916	9019	9000	9000
q10	3628	3539	3510	3510
q11	599	534	495	495
q12	816	607	620	607
q13	10897	3171	3203	3171
q14	311	264	266	264
q15	571	518	510	510
q16	695	654	655	654
q17	1843	1611	1570	1570
q18	7800	7513	7486	7486
q19	1693	1465	1537	1465
q20	2072	1790	1799	1790
q21	5387	5220	5137	5137
q22	623	586	540	540
Total cold run time: 69745 ms
Total hot run time: 59219 ms

doris-robot · 2024-12-12T04:35:31Z

TPC-DS: Total hot run time: 189796 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fbdbae126e036a2cee3bdf56f805bae4d9454160, data reload: false

query1	970	390	369	369
query2	6514	2129	2040	2040
query3	6709	229	218	218
query4	33900	23669	23424	23424
query5	4382	465	453	453
query6	301	179	185	179
query7	4622	306	317	306
query8	305	250	238	238
query9	9607	2704	2708	2704
query10	459	262	267	262
query11	18077	15128	15187	15128
query12	155	103	105	103
query13	1689	437	436	436
query14	9323	6451	7483	6451
query15	253	168	181	168
query16	8152	482	463	463
query17	1602	556	551	551
query18	2125	295	312	295
query19	317	148	141	141
query20	118	110	111	110
query21	210	100	98	98
query22	4628	4468	4425	4425
query23	35281	33754	33732	33732
query24	11258	2454	2389	2389
query25	535	376	384	376
query26	1112	152	151	151
query27	3181	278	279	278
query28	8081	2435	2425	2425
query29	649	403	400	400
query30	302	153	152	152
query31	1016	831	802	802
query32	110	61	61	61
query33	766	287	282	282
query34	970	504	524	504
query35	891	777	753	753
query36	1100	943	966	943
query37	200	74	77	74
query38	4422	4122	4213	4122
query39	1459	1411	1413	1411
query40	293	102	98	98
query41	49	43	44	43
query42	114	100	95	95
query43	530	479	489	479
query44	1225	813	806	806
query45	183	163	165	163
query46	1169	724	685	685
query47	1959	1814	1850	1814
query48	413	321	333	321
query49	1280	390	400	390
query50	797	392	383	383
query51	7092	7034	7129	7034
query52	103	91	86	86
query53	250	184	185	184
query54	1056	407	407	407
query55	78	78	81	78
query56	259	243	230	230
query57	1235	1108	1085	1085
query58	224	202	218	202
query59	3426	3085	3035	3035
query60	283	265	254	254
query61	145	136	133	133
query62	889	690	678	678
query63	219	191	188	188
query64	5050	731	627	627
query65	3248	3188	3219	3188
query66	1298	315	304	304
query67	16181	15617	15839	15617
query68	4958	566	554	554
query69	432	249	256	249
query70	1202	1130	1083	1083
query71	338	278	250	250
query72	6039	4070	4035	4035
query73	759	368	353	353
query74	10305	8907	8856	8856
query75	3415	2649	2623	2623
query76	2894	1254	1118	1118
query77	491	276	266	266
query78	10269	9486	9388	9388
query79	1296	606	582	582
query80	1071	432	460	432
query81	512	246	226	226
query82	1018	122	120	120
query83	236	149	149	149
query84	238	70	73	70
query85	1344	315	320	315
query86	354	307	291	291
query87	4716	4601	4451	4451
query88	3447	2209	2190	2190
query89	413	292	293	292
query90	1983	187	183	183
query91	139	107	107	107
query92	60	56	50	50
query93	1089	551	547	547
query94	976	298	294	294
query95	354	249	251	249
query96	627	287	276	276
query97	2831	2695	2659	2659
query98	229	197	197	197
query99	1564	1317	1331	1317
Total cold run time: 300398 ms
Total hot run time: 189796 ms

doris-robot · 2024-12-12T04:41:01Z

ClickBench: Total hot run time: 32.85 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fbdbae126e036a2cee3bdf56f805bae4d9454160, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.08	0.08
query4	1.60	0.11	0.10
query5	0.42	0.40	0.41
query6	1.15	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.02
query9	0.57	0.51	0.50
query10	0.55	0.58	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.72	2.76	2.75
query15	0.89	0.83	0.81
query16	0.39	0.37	0.38
query17	1.06	1.02	0.99
query18	0.22	0.21	0.21
query19	1.95	1.83	1.95
query20	0.01	0.01	0.01
query21	15.37	0.61	0.56
query22	2.68	2.08	2.39
query23	16.96	1.06	0.86
query24	3.40	1.06	2.13
query25	0.33	0.08	0.11
query26	0.53	0.14	0.13
query27	0.05	0.05	0.05
query28	9.47	1.11	1.08
query29	12.55	3.20	3.18
query30	0.25	0.06	0.06
query31	2.87	0.39	0.37
query32	3.26	0.47	0.46
query33	2.96	3.00	3.05
query34	17.04	4.48	4.46
query35	4.51	4.47	4.47
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.15	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.17 s
Total hot run time: 32.85 s

feiniaofeiafei · 2024-12-12T06:31:36Z

run buildall

feiniaofeiafei · 2024-12-24T14:29:14Z

run buildall

add rule count distinct split add rule count distinct split add regression test add regression fix code style change by comment change to custom rewrite change to custom rewrite fix regression fix regression

feiniaofeiafei · 2024-12-25T02:55:41Z

run buildall

doris-robot · 2024-12-25T03:53:40Z

TPC-H: Total hot run time: 32459 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a6d2c1cdd8bdd574c14008adc0f44f3a1ac4ec60, data reload: false

------ Round 1 ----------------------------------
q1	17952	6112	6071	6071
q2	2052	331	171	171
q3	10613	1276	711	711
q4	10236	849	432	432
q5	8111	2234	1963	1963
q6	212	192	149	149
q7	892	747	625	625
q8	9244	1331	1163	1163
q9	5224	4887	4925	4887
q10	6751	2297	1878	1878
q11	472	279	262	262
q12	363	358	213	213
q13	17771	3580	3022	3022
q14	248	227	230	227
q15	563	508	502	502
q16	631	617	586	586
q17	568	844	320	320
q18	7115	6546	6246	6246
q19	2402	975	532	532
q20	291	316	187	187
q21	2777	2159	2003	2003
q22	360	321	309	309
Total cold run time: 104848 ms
Total hot run time: 32459 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6350	6202	6202	6202
q2	239	320	224	224
q3	2267	2605	2270	2270
q4	1360	1823	1365	1365
q5	4381	4801	4885	4801
q6	177	180	142	142
q7	2048	1921	1782	1782
q8	2550	2841	2673	2673
q9	7268	7235	7256	7235
q10	3012	3326	2840	2840
q11	585	517	488	488
q12	682	751	619	619
q13	3358	3633	3097	3097
q14	277	331	295	295
q15	569	499	503	499
q16	660	700	657	657
q17	1220	1721	1255	1255
q18	7600	7443	7064	7064
q19	787	1105	1038	1038
q20	1925	1979	1865	1865
q21	5645	5155	4882	4882
q22	600	618	580	580
Total cold run time: 53560 ms
Total hot run time: 51873 ms

doris-robot · 2024-12-25T04:05:42Z

TPC-DS: Total hot run time: 189957 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a6d2c1cdd8bdd574c14008adc0f44f3a1ac4ec60, data reload: false

query1	975	381	383	381
query2	6521	2471	2315	2315
query3	6704	212	207	207
query4	33736	23645	23951	23645
query5	4332	637	458	458
query6	300	203	191	191
query7	4617	503	316	316
query8	306	250	240	240
query9	9681	2740	2725	2725
query10	453	315	260	260
query11	18304	15372	15600	15372
query12	159	110	106	106
query13	1673	531	401	401
query14	11068	7343	7198	7198
query15	226	189	178	178
query16	8195	598	419	419
query17	1602	751	582	582
query18	2105	437	299	299
query19	227	180	153	153
query20	117	117	114	114
query21	210	124	102	102
query22	4333	4160	4339	4160
query23	34315	33271	33442	33271
query24	6345	2297	2246	2246
query25	479	448	366	366
query26	775	268	153	153
query27	1999	454	320	320
query28	5314	2476	2430	2430
query29	630	533	411	411
query30	225	183	151	151
query31	981	903	833	833
query32	75	62	56	56
query33	494	340	288	288
query34	761	843	517	517
query35	799	816	757	757
query36	997	1034	933	933
query37	116	97	74	74
query38	4289	4065	4067	4065
query39	1510	1454	1396	1396
query40	199	116	102	102
query41	49	43	44	43
query42	122	101	102	101
query43	512	532	501	501
query44	1318	794	818	794
query45	175	172	167	167
query46	880	1038	632	632
query47	1919	1946	1881	1881
query48	375	397	329	329
query49	745	468	368	368
query50	636	661	379	379
query51	7145	7118	7048	7048
query52	104	106	91	91
query53	220	254	195	195
query54	471	471	402	402
query55	80	80	80	80
query56	288	259	234	234
query57	1198	1186	1124	1124
query58	236	219	239	219
query59	3029	3210	2901	2901
query60	264	284	253	253
query61	110	111	111	111
query62	903	803	750	750
query63	232	187	192	187
query64	3516	982	671	671
query65	3247	3181	3199	3181
query66	873	406	314	314
query67	15964	15882	15449	15449
query68	9541	746	500	500
query69	484	287	260	260
query70	1185	1077	1135	1077
query71	448	288	254	254
query72	5936	3803	3846	3803
query73	813	752	359	359
query74	10137	9102	8894	8894
query75	4649	3173	2654	2654
query76	5628	1188	753	753
query77	1007	365	273	273
query78	10077	10206	9418	9418
query79	2567	897	600	600
query80	726	568	439	439
query81	473	268	231	231
query82	324	155	125	125
query83	192	164	157	157
query84	281	86	76	76
query85	738	372	300	300
query86	360	324	284	284
query87	4451	4589	4303	4303
query88	3213	2312	2229	2229
query89	413	349	300	300
query90	2066	188	188	188
query91	143	130	116	116
query92	67	58	49	49
query93	939	852	522	522
query94	653	405	267	267
query95	332	259	247	247
query96	487	615	288	288
query97	2703	2859	2733	2733
query98	220	202	203	202
query99	1638	1555	1460	1460
Total cold run time: 293649 ms
Total hot run time: 189957 ms

doris-robot · 2024-12-25T04:11:02Z

ClickBench: Total hot run time: 31.28 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a6d2c1cdd8bdd574c14008adc0f44f3a1ac4ec60, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.08
query4	1.61	0.11	0.10
query5	0.42	0.41	0.42
query6	1.16	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.50	0.53
query10	0.54	0.59	0.56
query11	0.14	0.11	0.10
query12	0.15	0.11	0.11
query13	0.61	0.61	0.60
query14	2.80	2.77	2.81
query15	0.90	0.82	0.82
query16	0.38	0.40	0.38
query17	1.02	1.06	1.04
query18	0.23	0.21	0.20
query19	1.85	1.78	2.02
query20	0.02	0.01	0.01
query21	15.36	0.93	0.60
query22	0.75	0.76	0.66
query23	15.29	1.45	0.61
query24	2.61	0.82	1.72
query25	0.16	0.06	0.05
query26	0.24	0.15	0.15
query27	0.04	0.07	0.05
query28	14.26	1.48	1.04
query29	12.56	3.95	3.28
query30	0.25	0.09	0.06
query31	2.82	0.59	0.38
query32	3.23	0.54	0.46
query33	3.11	3.11	3.13
query34	16.51	5.14	4.51
query35	4.56	4.50	4.51
query36	0.64	0.48	0.49
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.7 s
Total hot run time: 31.28 s

feiniaofeiafei · 2024-12-25T07:53:01Z

run p0

github-actions · 2024-12-26T02:39:22Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-26T02:39:24Z

PR approved by anyone and no changes requested.

### What problem does this PR solve? Problem Summary: This pr add a rewrite rule, which can do this 2 type of rewrite: 1. This rewrite can greatly improve the execution speed of multiple count(distinct) operations. When 3be, ndv=10000000, the performance can be improved by three to four times. select count(distinct a),count(distinct b),count(distinct c) from t; -> with tmp as (select * from t) select * from (select count(distinct a) from tmp) t1 cross join (select count(distinct b) from tmp) t2 cross join (select count(distinct c) from tmp) t3 2.Before this PR, the following SQL statement would fail to execute due to an error: "The query contains multi count distinct or sum distinct, each can't have multi columns". This PR rewrites this type of SQL statement as follows, making it executable without an error. select count(distinct a,d),count(distinct b,c),count(distinct c) from t; -> with tmp as (select * from t) select * from (select count(distinct a,d) from tmp) t1 cross join (select count(distinct b,c) from tmp) t2 cross join (select count(distinct c) from tmp) t3 ### Release note Support multi count distinct with different parameters

### What problem does this PR solve? related pr: #45209 If split cout distinct and group by are used, inner join will be used to link the results, but if the group by column has a null value, inner join will not be linked, so we use `NullSafeEqual` as the link condition --------- Co-authored-by: garenshi <[email protected]>

feiniaofeiafei force-pushed the count_distinct_rewrite branch from fb681c9 to 99b8ad5 Compare December 10, 2024 12:41

feiniaofeiafei force-pushed the count_distinct_rewrite branch from 99b8ad5 to bcd9fcc Compare December 10, 2024 13:37

morrySnow reviewed Dec 11, 2024

View reviewed changes

feiniaofeiafei force-pushed the count_distinct_rewrite branch from d420066 to 93ee342 Compare December 11, 2024 13:45

feiniaofeiafei force-pushed the count_distinct_rewrite branch 3 times, most recently from 5aa17ad to fbdbae1 Compare December 12, 2024 04:04

feiniaofeiafei force-pushed the count_distinct_rewrite branch 2 times, most recently from e798aac to a6f812c Compare December 12, 2024 06:31

feiniaofeiafei force-pushed the count_distinct_rewrite branch from 791a242 to 92ebe5c Compare December 24, 2024 14:29

feiniaofeiafei force-pushed the count_distinct_rewrite branch from 92ebe5c to 6603b90 Compare December 25, 2024 02:53

feiniaofeiafei added 5 commits December 25, 2024 10:54

add rule count distinct split

13c384e

add rule count distinct split add rule count distinct split add regression test add regression fix code style change by comment change to custom rewrite change to custom rewrite fix regression fix regression

split other agg function into seperate aggregate

a5e68e3

fix regression

51e7f52

fix comment

609ee04

add regression

a6d2c1c

feiniaofeiafei force-pushed the count_distinct_rewrite branch from 6603b90 to a6d2c1c Compare December 25, 2024 02:54

feiniaofeiafei changed the title ~~[enhance](nereids) add rule count distinct split~~ [enhance](nereids) add rule MultiDistinctSplit Dec 25, 2024

morrySnow approved these changes Dec 26, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 26, 2024

github-actions bot added the reviewed label Dec 26, 2024

924060929 approved these changes Jan 2, 2025

View reviewed changes

morrySnow removed the dev/3.0.x label Jan 3, 2025

morrySnow merged commit c28c00a into apache:master Jan 3, 2025
29 of 30 checks passed

morrySnow added dev/3.0.x dev/3.1.x and removed dev/3.0.x labels Jan 7, 2025

github-actions bot mentioned this pull request Jan 7, 2025

branch-3.0: [enhance](nereids) add rule MultiDistinctSplit #45209 #46540

Open

qzsee mentioned this pull request Jan 7, 2025

[fix][nereids] fix split count distinct for null can't join #46563

Merged

16 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[enhance](nereids) add rule MultiDistinctSplit #45209

[enhance](nereids) add rule MultiDistinctSplit #45209

feiniaofeiafei commented Dec 9, 2024 •

edited

Loading

doris-robot commented Dec 9, 2024

feiniaofeiafei commented Dec 9, 2024

doris-robot commented Dec 9, 2024

doris-robot commented Dec 9, 2024

doris-robot commented Dec 9, 2024

feiniaofeiafei commented Dec 10, 2024

feiniaofeiafei commented Dec 10, 2024

doris-robot commented Dec 10, 2024

doris-robot commented Dec 10, 2024

doris-robot commented Dec 10, 2024

feiniaofeiafei commented Dec 11, 2024

feiniaofeiafei commented Dec 11, 2024

feiniaofeiafei commented Dec 11, 2024

doris-robot commented Dec 11, 2024

doris-robot commented Dec 11, 2024

doris-robot commented Dec 11, 2024

feiniaofeiafei commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

feiniaofeiafei commented Dec 12, 2024

feiniaofeiafei commented Dec 24, 2024

feiniaofeiafei commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

feiniaofeiafei commented Dec 25, 2024

github-actions bot commented Dec 26, 2024

github-actions bot commented Dec 26, 2024

[enhance](nereids) add rule MultiDistinctSplit #45209

[enhance](nereids) add rule MultiDistinctSplit #45209

Conversation

feiniaofeiafei commented Dec 9, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented Dec 9, 2024

feiniaofeiafei commented Dec 9, 2024

doris-robot commented Dec 9, 2024

doris-robot commented Dec 9, 2024

doris-robot commented Dec 9, 2024

feiniaofeiafei commented Dec 10, 2024

feiniaofeiafei commented Dec 10, 2024

doris-robot commented Dec 10, 2024

doris-robot commented Dec 10, 2024

doris-robot commented Dec 10, 2024

feiniaofeiafei commented Dec 11, 2024

feiniaofeiafei commented Dec 11, 2024

feiniaofeiafei commented Dec 11, 2024

doris-robot commented Dec 11, 2024

doris-robot commented Dec 11, 2024

doris-robot commented Dec 11, 2024

feiniaofeiafei commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

feiniaofeiafei commented Dec 12, 2024

feiniaofeiafei commented Dec 24, 2024

feiniaofeiafei commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

feiniaofeiafei commented Dec 25, 2024

github-actions bot commented Dec 26, 2024

github-actions bot commented Dec 26, 2024

feiniaofeiafei commented Dec 9, 2024 •

edited

Loading