Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [fix](hudi)Add hudi catalog read partition table partition prune #44669 #45111

Merged
merged 1 commit into from
Dec 6, 2024

Conversation

github-actions[bot]
Copy link
Contributor

@github-actions github-actions bot commented Dec 6, 2024

Cherry-picked from #44669

)

### What problem does this PR solve?
Problem Summary:
In the previous PR #41218, some partition pruning logic was changed,
which caused the hudi partition pruning to fail. This PR is to fix this
problem.

### Release note

[fix](hudi) fix hudi partition prune issue
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Dec 6, 2024
@dataroaring dataroaring reopened this Dec 6, 2024
@doris-robot
Copy link

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40293 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 764aa1951e9daa06e462c3bff81e898684101aa1, data reload: false

------ Round 1 ----------------------------------
q1	17571	7314	7303	7303
q2	2068	165	169	165
q3	10696	1034	1155	1034
q4	10543	736	777	736
q5	7718	2800	2690	2690
q6	231	145	146	145
q7	951	598	590	590
q8	9588	1926	1942	1926
q9	7047	6395	6429	6395
q10	6945	2310	2295	2295
q11	450	261	255	255
q12	398	209	202	202
q13	17769	2990	2965	2965
q14	237	221	207	207
q15	551	501	523	501
q16	666	606	639	606
q17	966	588	504	504
q18	7125	6668	6485	6485
q19	4471	1070	1108	1070
q20	478	201	197	197
q21	3892	3061	3220	3061
q22	1093	961	981	961
Total cold run time: 111454 ms
Total hot run time: 40293 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7415	7229	7274	7229
q2	326	225	232	225
q3	2992	2826	2848	2826
q4	2086	1734	1742	1734
q5	5641	5706	5719	5706
q6	219	139	141	139
q7	2372	1753	1764	1753
q8	3347	3441	3381	3381
q9	8900	8772	8741	8741
q10	3575	3506	3539	3506
q11	587	482	491	482
q12	816	620	603	603
q13	16464	3193	3168	3168
q14	320	272	268	268
q15	566	525	518	518
q16	725	672	661	661
q17	1817	1629	1595	1595
q18	8250	7844	7636	7636
q19	1719	1549	1517	1517
q20	2092	1851	1837	1837
q21	5427	5405	5230	5230
q22	1115	1064	1012	1012
Total cold run time: 76771 ms
Total hot run time: 59767 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195221 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 764aa1951e9daa06e462c3bff81e898684101aa1, data reload: false

query1	1243	926	906	906
query2	6230	2119	2030	2030
query3	10939	4254	4261	4254
query4	67373	28949	23408	23408
query5	5256	432	439	432
query6	438	169	161	161
query7	5627	314	302	302
query8	325	236	234	234
query9	9451	2702	2656	2656
query10	491	258	251	251
query11	17683	15221	15662	15221
query12	160	102	110	102
query13	1551	421	419	419
query14	10555	6563	7448	6563
query15	209	173	174	173
query16	7358	533	493	493
query17	1120	612	594	594
query18	1878	345	316	316
query19	239	166	162	162
query20	120	116	108	108
query21	216	106	105	105
query22	4609	4448	4536	4448
query23	34715	34256	34183	34183
query24	6142	2917	2874	2874
query25	551	420	427	420
query26	673	170	168	168
query27	1717	299	306	299
query28	4557	2512	2476	2476
query29	682	462	451	451
query30	251	168	186	168
query31	996	802	807	802
query32	63	55	63	55
query33	449	282	272	272
query34	904	493	501	493
query35	853	727	741	727
query36	1067	943	957	943
query37	118	79	77	77
query38	4092	4042	4209	4042
query39	1509	1478	1444	1444
query40	209	101	97	97
query41	49	45	45	45
query42	110	97	98	97
query43	547	502	488	488
query44	1189	784	811	784
query45	181	172	167	167
query46	1150	705	726	705
query47	1996	1935	1894	1894
query48	470	361	366	361
query49	721	391	400	391
query50	821	424	412	412
query51	7308	7144	6901	6901
query52	102	85	90	85
query53	260	182	180	180
query54	546	433	435	433
query55	79	76	73	73
query56	255	235	250	235
query57	1168	1142	1094	1094
query58	205	192	196	192
query59	3100	3008	2909	2909
query60	277	239	246	239
query61	108	108	106	106
query62	776	681	662	662
query63	206	176	187	176
query64	1621	647	613	613
query65	3260	3215	3186	3186
query66	715	306	295	295
query67	15930	15472	15392	15392
query68	4517	533	529	529
query69	413	245	276	245
query70	1159	1135	1092	1092
query71	353	260	246	246
query72	6380	3980	3873	3873
query73	760	333	327	327
query74	10352	8903	9014	8903
query75	3286	2641	2588	2588
query76	1834	1020	1081	1020
query77	509	264	259	259
query78	10869	9561	9413	9413
query79	11007	585	592	585
query80	2452	415	410	410
query81	548	241	240	240
query82	1681	111	112	111
query83	268	145	140	140
query84	286	90	79	79
query85	1269	307	283	283
query86	474	301	296	296
query87	4463	4233	4291	4233
query88	6038	2363	2393	2363
query89	530	293	287	287
query90	2117	182	186	182
query91	179	142	144	142
query92	73	46	47	46
query93	7417	527	519	519
query94	895	268	292	268
query95	333	246	251	246
query96	647	285	273	273
query97	3291	3198	3102	3102
query98	224	201	198	198
query99	1887	1279	1319	1279
Total cold run time: 342280 ms
Total hot run time: 195221 ms

@morningman morningman merged commit 7df5c10 into branch-3.0 Dec 6, 2024
19 of 21 checks passed
@github-actions github-actions bot deleted the auto-pick-44669-branch-3.0 branch December 6, 2024 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants